[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=144534=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144534
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 15/Sep/18 02:56
Start Date: 15/Sep/18 02:56
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #6316: [BEAM-4461] Add Unnest 
transform.
URL: https://github.com/apache/beam/pull/6316#issuecomment-421525759
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144534)
Time Spent: 9.5h  (was: 9h 20m)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 9.5h
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3286) Go SDK support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3286?focusedWorklogId=144526=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144526
 ]

ASF GitHub Bot logged work on BEAM-3286:


Author: ASF GitHub Bot
Created on: 15/Sep/18 01:33
Start Date: 15/Sep/18 01:33
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #6402: [BEAM-3286] Fix Go 
index parsing in the face of runner-generated keys
URL: https://github.com/apache/beam/pull/6402#issuecomment-421521171
 
 
   R: @lostluck 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144526)
Time Spent: 2h 50m  (was: 2h 40m)

> Go SDK support for portable side input
> --
>
> Key: BEAM-3286
> URL: https://issues.apache.org/jira/browse/BEAM-3286
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=144525=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144525
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 15/Sep/18 01:25
Start Date: 15/Sep/18 01:25
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6316: [BEAM-4461] Add 
Unnest transform.
URL: https://github.com/apache/beam/pull/6316#issuecomment-421520688
 
 
   @akedin All comments addressed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144525)
Time Spent: 9h 20m  (was: 9h 10m)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 9h 20m
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3286) Go SDK support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3286?focusedWorklogId=144524=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144524
 ]

ASF GitHub Bot logged work on BEAM-3286:


Author: ASF GitHub Bot
Created on: 15/Sep/18 01:22
Start Date: 15/Sep/18 01:22
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #6402: [BEAM-3286] Fix Go 
index parsing in the face of runner-generated keys
URL: https://github.com/apache/beam/pull/6402#issuecomment-421520488
 
 
   Run Go PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144524)
Time Spent: 2h 40m  (was: 2.5h)

> Go SDK support for portable side input
> --
>
> Key: BEAM-3286
> URL: https://issues.apache.org/jira/browse/BEAM-3286
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3286) Go SDK support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3286?focusedWorklogId=144523=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144523
 ]

ASF GitHub Bot logged work on BEAM-3286:


Author: ASF GitHub Bot
Created on: 15/Sep/18 01:21
Start Date: 15/Sep/18 01:21
Worklog Time Spent: 10m 
  Work Description: herohde opened a new pull request #6402: [BEAM-3286] 
Fix Go index parsing in the face of runner-generated keys
URL: https://github.com/apache/beam/pull/6402
 
 
   It turns out that Dataflow sometimes replaces the main input key with, say, 
"-23", when it mucks with the graph.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144523)
Time Spent: 2.5h  (was: 2h 20m)

> Go SDK support for portable side input
> --
>
> Key: BEAM-3286
> URL: https://issues.apache.org/jira/browse/BEAM-3286
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>

[jira] [Work logged] (BEAM-5317) Finish Python 3 porting for options module

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5317?focusedWorklogId=144522=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144522
 ]

ASF GitHub Bot logged work on BEAM-5317:


Author: ASF GitHub Bot
Created on: 15/Sep/18 01:17
Start Date: 15/Sep/18 01:17
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #6397: 
[BEAM-5317] Finish Python3 porting for options module
URL: https://github.com/apache/beam/pull/6397#discussion_r217871523
 
 

 ##
 File path: sdks/python/apache_beam/options/pipeline_options_validator_test.py
 ##
 @@ -311,7 +311,7 @@ def get_validator(matcher):
  '--staging_location=gs://foo/bar',
  '--temp_location=gs://foo/bar',]
   if matcher:
-options.append('--on_success_matcher=' + matcher)
+options.append('%s=%s' % ('--on_success_matcher=', matcher.decode()))
 
 Review comment:
   After `--on_success_matcher`. Also please squash the previous commits. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144522)
Time Spent: 3.5h  (was: 3h 20m)

> Finish Python 3 porting for options module
> --
>
> Key: BEAM-5317
> URL: https://issues.apache.org/jira/browse/BEAM-5317
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Manu Zhang
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5317) Finish Python 3 porting for options module

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5317?focusedWorklogId=144521=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144521
 ]

ASF GitHub Bot logged work on BEAM-5317:


Author: ASF GitHub Bot
Created on: 15/Sep/18 01:14
Start Date: 15/Sep/18 01:14
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #6397: 
[BEAM-5317] Finish Python3 porting for options module
URL: https://github.com/apache/beam/pull/6397#discussion_r217871368
 
 

 ##
 File path: sdks/python/apache_beam/options/pipeline_options_validator_test.py
 ##
 @@ -311,7 +311,7 @@ def get_validator(matcher):
  '--staging_location=gs://foo/bar',
  '--temp_location=gs://foo/bar',]
   if matcher:
-options.append('--on_success_matcher=' + matcher)
+options.append('%s=%s' % ('--on_success_matcher=', matcher.decode()))
 
 Review comment:
   extra `=`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144521)
Time Spent: 3h 20m  (was: 3h 10m)

> Finish Python 3 porting for options module
> --
>
> Key: BEAM-5317
> URL: https://issues.apache.org/jira/browse/BEAM-5317
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Manu Zhang
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=144520=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144520
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 15/Sep/18 01:06
Start Date: 15/Sep/18 01:06
Worklog Time Spent: 10m 
  Work Description: reuvenlax closed pull request #6317: [BEAM-4461]  Add 
mapping between FieldType and Java types.
URL: https://github.com/apache/beam/pull/6317
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldTypeDescriptors.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldTypeDescriptors.java
new file mode 100644
index 000..252ea5e1831
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldTypeDescriptors.java
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.BiMap;
+import com.google.common.collect.ImmutableBiMap;
+import java.lang.reflect.ParameterizedType;
+import java.util.Collection;
+import java.util.Map;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.joda.time.Instant;
+/**
+ * Utilities for converting between {@link Schema} field types and {@link 
TypeDescriptor}s that
+ * define Java objects which can represent these field types.
+ */
+public class FieldTypeDescriptors {
+  private static final BiMap PRIMITIVE_MAPPING =
+  ImmutableBiMap.builder()
+  .put(TypeName.BYTE, TypeDescriptors.bytes())
+  .put(TypeName.INT16, TypeDescriptors.shorts())
+  .put(TypeName.INT32, TypeDescriptors.integers())
+  .put(TypeName.INT64, TypeDescriptors.longs())
+  .put(TypeName.DECIMAL, TypeDescriptors.bigdecimals())
+  .put(TypeName.FLOAT, TypeDescriptors.floats())
+  .put(TypeName.DOUBLE, TypeDescriptors.doubles())
+  .put(TypeName.STRING, TypeDescriptors.strings())
+  .put(TypeName.DATETIME, TypeDescriptor.of(Instant.class))
+  .put(TypeName.BOOLEAN, TypeDescriptors.booleans())
+  .put(TypeName.BYTES, TypeDescriptor.of(byte[].class))
+  .build();
+  /** Get a {@link TypeDescriptor} from a {@link FieldType}. */
+  public static TypeDescriptor javaTypeForFieldType(FieldType fieldType) {
+switch (fieldType.getTypeName()) {
+  case ARRAY:
+return 
TypeDescriptors.lists(javaTypeForFieldType(fieldType.getCollectionElementType()));
+  case MAP:
+return TypeDescriptors.maps(
+javaTypeForFieldType(fieldType.getMapKeyType()),
+javaTypeForFieldType(fieldType.getMapValueType()));
+  case ROW:
+return TypeDescriptors.rows();
+  default:
+return PRIMITIVE_MAPPING.get(fieldType.getTypeName());
+}
+  }
+  /** Get a {@link FieldType} from a {@link TypeDescriptor}. */
+  public static FieldType fieldTypeForJavaType(TypeDescriptor typeDescriptor) {
+if (typeDescriptor.isArray()
+|| typeDescriptor.isSubtypeOf(TypeDescriptor.of(Collection.class))) {
+  return getArrayFieldType(typeDescriptor);
+} else if (typeDescriptor.isSubtypeOf(TypeDescriptor.of(Map.class))) {
+  return getMapFieldType(typeDescriptor);
+} else if (typeDescriptor.isSubtypeOf(TypeDescriptor.of(Row.class))) {
+  throw new IllegalArgumentException(
+  "Cannot automatically determine a field type from a Row class"
+  + " as we cannot determine the schema. You should set a field 
type explicitly.");
+} else {
+  TypeName typeName = 

[beam] branch master updated (f568578 -> b4d1ef3)

[reuvenlax]

This is an automated email from the ASF dual-hosted git repository.

reuvenlax pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f568578  Merge pull request #6393 from 
yifanzou/BEAM-5383/migrate_bigquery_io_read_it_python
 add be5600b  Add mapping between FieldType and Java types.
 add e7aa13c  Apply spotless.
 add b4d1ef3  Merge pull request #6317: [BEAM-4461]  Add mapping between 
FieldType and Java types.

No new revisions were added by this update.

Summary of changes:
 .../beam/sdk/schemas/FieldTypeDescriptors.java | 119 +++
 .../apache/beam/sdk/values/TypeDescriptors.java|  10 ++
 .../beam/sdk/schemas/FieldTypeDescriptorsTest.java | 160 +
 3 files changed, 289 insertions(+)
 create mode 100644 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldTypeDescriptors.java
 create mode 100644 
sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/FieldTypeDescriptorsTest.java

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=144519=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144519
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 15/Sep/18 00:57
Start Date: 15/Sep/18 00:57
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6341: [BEAM-4130] Fixing 
bug in flink job server creation
URL: https://github.com/apache/beam/pull/6341#issuecomment-421518808
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144519)
Time Spent: 9h  (was: 8h 50m)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 2.7.0
>
>  Time Spent: 9h
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5317) Finish Python 3 porting for options module

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5317?focusedWorklogId=144518=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144518
 ]

ASF GitHub Bot logged work on BEAM-5317:


Author: ASF GitHub Bot
Created on: 15/Sep/18 00:56
Start Date: 15/Sep/18 00:56
Worklog Time Spent: 10m 
  Work Description: manuzhang commented on a change in pull request #6397: 
[BEAM-5317] Finish Python3 porting for options module
URL: https://github.com/apache/beam/pull/6397#discussion_r217870574
 
 

 ##
 File path: sdks/python/apache_beam/options/pipeline_options_validator_test.py
 ##
 @@ -329,7 +329,11 @@ def get_validator(matcher):
 ]
 
 for case in test_case:
-  errors = get_validator(case['on_success_matcher']).validate()
+  matcher = case['on_success_matcher']
+  if matcher and type(matcher) is bytes:
+errors = get_validator(matcher.decode('utf-8')).validate()
 
 Review comment:
   Thanks, that looks better and more consistent


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144518)
Time Spent: 3h 10m  (was: 3h)

> Finish Python 3 porting for options module
> --
>
> Key: BEAM-5317
> URL: https://issues.apache.org/jira/browse/BEAM-5317
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Manu Zhang
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=144517=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144517
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 15/Sep/18 00:56
Start Date: 15/Sep/18 00:56
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #6341: 
[BEAM-4130] Fixing bug in flink job server creation
URL: https://github.com/apache/beam/pull/6341#discussion_r217869549
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
 ##
 @@ -130,7 +137,7 @@ private FlinkJobServerDriver(
 this.configuration = configuration;
 this.executor = executor;
 this.jobServerFactory = jobServerFactory;
-this.artifactServerFactory = jobServerFactory;
+this.artifactServerFactory = artifactServerFactory;
 
 Review comment:
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144517)
Time Spent: 8h 50m  (was: 8h 40m)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 2.7.0
>
>  Time Spent: 8h 50m
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=144516=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144516
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 15/Sep/18 00:56
Start Date: 15/Sep/18 00:56
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #6341: 
[BEAM-4130] Fixing bug in flink job server creation
URL: https://github.com/apache/beam/pull/6341#discussion_r217870561
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
 ##
 @@ -107,12 +107,19 @@ public static FlinkJobServerDriver 
fromConfig(ServerConfiguration configuration)
 new 
ThreadFactoryBuilder().setNameFormat("flink-runner-job-server").setDaemon(true).build();
 ListeningExecutorService executor =
 
MoreExecutors.listeningDecorator(Executors.newCachedThreadPool(threadFactory));
-ServerFactory jobServerFactory = ServerFactory.createWithPortSupplier(() 
-> configuration.port);
+ServerFactory jobServerFactory =
+ServerFactory.createWithPortSupplier(() -> getPort(configuration.host, 
configuration.port));
 ServerFactory artifactServerFactory =
-ServerFactory.createWithPortSupplier(() -> configuration.artifactPort);
+ServerFactory.createWithPortSupplier(
+() -> getPort(configuration.host, configuration.artifactPort));
 return create(configuration, executor, jobServerFactory, 
artifactServerFactory);
   }
 
+  private static int getPort(String host, int port) {
+// If host is empty then use dynamic port
+return Strings.isNullOrEmpty(host) ? 0 : port;
 
 Review comment:
   We need to be able to use dynamic port to create server in case of test.
   Port supplier is used when using allocateAndCreate which is expected to use 
a dynamic port. As port was fixed in portSupplier, this functionality was 
broken.
   I am using host as the marker to create dynamic port or static as we do in 
[L181 
](https://github.com/apache/beam/pull/6341/files#diff-e52a25378b181af8cecbf3d274946fd0L181)
 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144516)
Time Spent: 8h 50m  (was: 8h 40m)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 2.7.0
>
>  Time Spent: 8h 50m
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5389) [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]

[yifan zou (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615532#comment-16615532
 ] 

yifan zou commented on BEAM-5389:
-

PostCommit fixed. 
https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1468/

> [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]
>  
> -
>
> Key: BEAM-5389
> URL: https://issues.apache.org/jira/browse/BEAM-5389
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1465/]
> Relevant log:
> 11:10:52.145 [LIFECYCLE] 
> [org.gradle.internal.operations.DefaultBuildOperationExecutor] > Task 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> org.apache.beam.examples.complete.TfIdfIT > testE2ETfIdf FAILED
> 11:10:57.266 [DEBUG] [TestEventLogger] java.lang.AssertionError: 
> 11:10:57.266 [DEBUG] [TestEventLogger] Expected: Expected checksum is 
> (f072786dde51dc09fc74bf38ffbfc27dcfdf0b96)
> 11:10:57.266 [DEBUG] [TestEventLogger] but: was 
> (0549d1dc8821976121771aefcb0e2297177bdb88)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:138)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdf.runTfIdf(TfIdf.java:427)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdfIT.testE2ETfIdf(TfIdfIT.java:72)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> java.lang.reflect.Method.invoke(Method.java:498)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 11:10:57.269 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 11:10:57.269 [DEBUG] [TestEventLogger] at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> 11:10:57.269 [DEBUG] [TestEventLogger] at 
> 

[jira] [Closed] (BEAM-5365) Migrate integration tests for bigquery_tornadoes

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-5365.
---
   Resolution: Fixed
Fix Version/s: 2.8.0

> Migrate integration tests for bigquery_tornadoes
> 
>
> Key: BEAM-5365
> URL: https://issues.apache.org/jira/browse/BEAM-5365
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5389) [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-5389.
---
   Resolution: Fixed
Fix Version/s: 2.8.0

> [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]
>  
> -
>
> Key: BEAM-5389
> URL: https://issues.apache.org/jira/browse/BEAM-5389
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1465/]
> Relevant log:
> 11:10:52.145 [LIFECYCLE] 
> [org.gradle.internal.operations.DefaultBuildOperationExecutor] > Task 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> org.apache.beam.examples.complete.TfIdfIT > testE2ETfIdf FAILED
> 11:10:57.266 [DEBUG] [TestEventLogger] java.lang.AssertionError: 
> 11:10:57.266 [DEBUG] [TestEventLogger] Expected: Expected checksum is 
> (f072786dde51dc09fc74bf38ffbfc27dcfdf0b96)
> 11:10:57.266 [DEBUG] [TestEventLogger] but: was 
> (0549d1dc8821976121771aefcb0e2297177bdb88)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:138)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdf.runTfIdf(TfIdf.java:427)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdfIT.testE2ETfIdf(TfIdfIT.java:72)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> java.lang.reflect.Method.invoke(Method.java:498)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 11:10:57.269 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 11:10:57.269 [DEBUG] [TestEventLogger] at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> 11:10:57.269 [DEBUG] [TestEventLogger] at 
> 

[jira] [Resolved] (BEAM-5383) Migrate integration tests for python bigquery io read

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou resolved BEAM-5383.
-
   Resolution: Fixed
Fix Version/s: 2.8.0

> Migrate integration tests for python  bigquery io read 
> ---
>
> Key: BEAM-5383
> URL: https://issues.apache.org/jira/browse/BEAM-5383
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1468

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #1438

[Apache Jenkins Server]

See 


Changes:

[pablo] Adding a custom coder test

[pablo] Adding TopWikipediaSessions Integration test

[github] Adding license

[pablo] Fixing issues

[thw] [BEAM-5385] Flink job server to use REST port from --flink-master-url

[yifanzou] [BEAM-5389] fix the checksum assertion in the TfIdf IT

[aaltay] [BEAM-5375] KafkaIO : Handle runtime exceptions while fetching from

[yifanzou] [BEAM-5383] migrate bigquer_io_read_it_test to beam

[github] Adding TODO

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam15 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision f568578ca443a48c4780231e996b66f9649d77c1 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f568578ca443a48c4780231e996b66f9649d77c1
Commit message: "Merge pull request #6393 from 
yifanzou/BEAM-5383/migrate_bigquery_io_read_it_python"
 > git rev-list --no-walk 5d07d9fcf6ef15894cf8491d9c83ace9e622b26b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1854558439531201588.sh
+ rm -rf 

[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2845293400310477198.sh
+ rm -rf 

[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7490632426920485340.sh
+ virtualenv 

New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins509162784042394879.sh
+ 

 install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./env/.perfkit_env/lib/python2.7/site-packages (40.2.0)
Requirement already up-to-date: pip in 
./env/.perfkit_env/lib/python2.7/site-packages (18.0)
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7564712264800027797.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git 

Cloning into 
'
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8278004772484170565.sh
+ 

 install -r 

Collecting absl-py (from -r 

 (line 14))
Collecting jinja2>=2.7 (from -r 

 (line 15))
  Using cached 
https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
Requirement already satisfied: setuptools in 
./env/.perfkit_env/lib/python2.7/site-packages (from -r 

 (line 16)) (40.2.0)
Collecting colorlog[windows]==2.6.0 (from -r 

 (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r 

[jira] [Reopened] (BEAM-4788) Test Auto JIRA subtask456

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou reopened BEAM-4788:
-

> Test Auto JIRA subtask456
> -
>
> Key: BEAM-4788
> URL: https://issues.apache.org/jira/browse/BEAM-4788
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>
> updated couple feilds of this issue by using jira api



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4788) Test Auto JIRA subtask456

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-4788.
---
Resolution: Won't Fix

> Test Auto JIRA subtask456
> -
>
> Key: BEAM-4788
> URL: https://issues.apache.org/jira/browse/BEAM-4788
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>
> updated couple feilds of this issue by using jira api



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Website_Cron #60

[Apache Jenkins Server]

See 


Changes:

[pablo] Adding a custom coder test

[pablo] Adding TopWikipediaSessions Integration test

[github] Adding license

[pablo] Fixing issues

[thw] [BEAM-5385] Flink job server to use REST port from --flink-master-url

[yifanzou] [BEAM-5389] fix the checksum assertion in the TfIdf IT

[aaltay] [BEAM-5375] KafkaIO : Handle runtime exceptions while fetching from

[yifanzou] [BEAM-5383] migrate bigquer_io_read_it_test to beam

[github] Adding TODO

--
[...truncated 8.17 KB...]
:assemble (Thread[Task worker for ':buildSrc' Thread 4,5,main]) completed. Took 
0.0 secs.
:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 4,5,main]) started.

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 1.397 secs.
:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.0 secs.
:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.029 secs.
:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) completed. Took 0.0 secs.
:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 4,5,main]) started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 4,5,main]) completed. 
Took 0.0 secs.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 4,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.003 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.003 secs.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.001 secs.
:testClasses (Thread[Task worker for ':buildSrc' Thread 4,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc' Thread 4,5,main]) completed. 
Took 0.0 secs.
:test (Thread[Task worker for ':buildSrc' Thread 4,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc' Thread 

[jira] [Assigned] (BEAM-5391) Use repeated fields for model for ordering of input/output/etc maps

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde reassigned BEAM-5391:
---

Assignee: (was: Kenneth Knowles)

> Use repeated fields for model for ordering of input/output/etc maps
> ---
>
> Key: BEAM-5391
> URL: https://issues.apache.org/jira/browse/BEAM-5391
> Project: Beam
>  Issue Type: Sub-task
>  Components: beam-model
>Reporter: Henning Rohde
>Priority: Major
>
> The Go SDK uses positional side input tagging and needs the ordering. It 
> cannot as easily encode it into a sequential tag ("i0", "i1", ..) because 
> runners much with the naming when doing certain optimizations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5391) Use repeated fields for model for ordering of input/output/etc maps

[Henning Rohde (JIRA)]

Henning Rohde created BEAM-5391:
---

 Summary: Use repeated fields for model for ordering of 
input/output/etc maps
 Key: BEAM-5391
 URL: https://issues.apache.org/jira/browse/BEAM-5391
 Project: Beam
  Issue Type: Sub-task
  Components: beam-model
Reporter: Henning Rohde
Assignee: Kenneth Knowles


The Go SDK uses positional side input tagging and needs the ordering. It cannot 
as easily encode it into a sequential tag ("i0", "i1", ..) because runners much 
with the naming when doing certain optimizations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Python_PVR_Flink_Gradle #19

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5383) Migrate integration tests for python bigquery io read

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?focusedWorklogId=144501=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144501
 ]

ASF GitHub Bot logged work on BEAM-5383:


Author: ASF GitHub Bot
Created on: 14/Sep/18 23:14
Start Date: 14/Sep/18 23:14
Worklog Time Spent: 10m 
  Work Description: pabloem closed pull request #6393: [BEAM-5383] migrate 
bigquer_io_read_it_test to beam
URL: https://github.com/apache/beam/pull/6393
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/io/gcp/bigquery_io_read_it_test.py 
b/sdks/python/apache_beam/io/gcp/bigquery_io_read_it_test.py
new file mode 100644
index 000..b9b3b41c324
--- /dev/null
+++ b/sdks/python/apache_beam/io/gcp/bigquery_io_read_it_test.py
@@ -0,0 +1,60 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+"""A Dataflow job that counts the number of rows in a BQ table.
+
+   Can be configured to simulate slow reading for a given number of rows.
+"""
+
+from __future__ import absolute_import
+
+import logging
+import unittest
+
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
+
+from apache_beam.io.gcp import bigquery_io_read_pipeline
+from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
+from apache_beam.testing.test_pipeline import TestPipeline
+
+
+class BigqueryIOReadIT(unittest.TestCase):
+
+  DEFAULT_DATASET = "big_query_import_export"
+  DEFAULT_TABLE_PREFIX = "export_"
+  NUM_RECORDS = {"1K": 1000,}
+
+  def run_bigquery_io_read_pipeline(self, input_size):
+test_pipeline = TestPipeline(is_integration_test=True)
+pipeline_verifiers = [PipelineStateMatcher(),]
+extra_opts = {'input_table': self.DEFAULT_DATASET + "." +
+ self.DEFAULT_TABLE_PREFIX + input_size,
+  'num_records': self.NUM_RECORDS[input_size],
+  'on_success_matcher': all_of(*pipeline_verifiers)}
+bigquery_io_read_pipeline.run(test_pipeline.get_full_options_as_args(
+**extra_opts))
+
+  @attr('IT')
+  def test_1K_table(self):
+self.run_bigquery_io_read_pipeline('1K')
+
+
+if __name__ == '__main__':
+  logging.getLogger().setLevel(logging.INFO)
+  unittest.main()
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_io_read_pipeline.py 
b/sdks/python/apache_beam/io/gcp/bigquery_io_read_pipeline.py
new file mode 100644
index 000..15bedf84748
--- /dev/null
+++ b/sdks/python/apache_beam/io/gcp/bigquery_io_read_pipeline.py
@@ -0,0 +1,93 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+# Copyright 2016 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under 

[beam] branch master updated (859b8d5 -> f568578)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 859b8d5  Merge pull request #6311 from pabloem/twsit
 add 59925c7  [BEAM-5383] migrate bigquer_io_read_it_test to beam
 new f568578  Merge pull request #6393 from 
yifanzou/BEAM-5383/migrate_bigquery_io_read_it_python

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache_beam/io/gcp/bigquery_io_read_it_test.py | 60 ++
 .../io/gcp/bigquery_io_read_pipeline.py| 93 ++
 2 files changed, 153 insertions(+)
 create mode 100644 sdks/python/apache_beam/io/gcp/bigquery_io_read_it_test.py
 create mode 100644 sdks/python/apache_beam/io/gcp/bigquery_io_read_pipeline.py

[beam] 01/01: Merge pull request #6393 from yifanzou/BEAM-5383/migrate_bigquery_io_read_it_python

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f568578ca443a48c4780231e996b66f9649d77c1
Merge: 859b8d5 59925c7
Author: Pablo 
AuthorDate: Fri Sep 14 16:14:03 2018 -0700

Merge pull request #6393 from 
yifanzou/BEAM-5383/migrate_bigquery_io_read_it_python

[BEAM-5383] migrate bigquer_io_read_it_test to beam

 .../apache_beam/io/gcp/bigquery_io_read_it_test.py | 60 ++
 .../io/gcp/bigquery_io_read_pipeline.py| 93 ++
 2 files changed, 153 insertions(+)

[jira] [Work logged] (BEAM-5383) Migrate integration tests for python bigquery io read

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?focusedWorklogId=144500=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144500
 ]

ASF GitHub Bot logged work on BEAM-5383:


Author: ASF GitHub Bot
Created on: 14/Sep/18 23:13
Start Date: 14/Sep/18 23:13
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #6393: [BEAM-5383] migrate 
bigquer_io_read_it_test to beam
URL: https://github.com/apache/beam/pull/6393#issuecomment-421507641
 
 
   Yifan this is excellent. Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144500)
Time Spent: 2h 50m  (was: 2h 40m)

> Migrate integration tests for python  bigquery io read 
> ---
>
> Key: BEAM-5383
> URL: https://issues.apache.org/jira/browse/BEAM-5383
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5383) Migrate integration tests for python bigquery io read

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?focusedWorklogId=144497=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144497
 ]

ASF GitHub Bot logged work on BEAM-5383:


Author: ASF GitHub Bot
Created on: 14/Sep/18 23:00
Start Date: 14/Sep/18 23:00
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6393: [BEAM-5383] migrate 
bigquer_io_read_it_test to beam
URL: https://github.com/apache/beam/pull/6393#issuecomment-421505549
 
 
   +R: @pabloem 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144497)
Time Spent: 2h 40m  (was: 2.5h)

> Migrate integration tests for python  bigquery io read 
> ---
>
> Key: BEAM-5383
> URL: https://issues.apache.org/jira/browse/BEAM-5383
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5317) Finish Python 3 porting for options module

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5317?focusedWorklogId=144494=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144494
 ]

ASF GitHub Bot logged work on BEAM-5317:


Author: ASF GitHub Bot
Created on: 14/Sep/18 22:56
Start Date: 14/Sep/18 22:56
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #6397: 
[BEAM-5317] Finish Python3 porting for options module
URL: https://github.com/apache/beam/pull/6397#discussion_r217860728
 
 

 ##
 File path: sdks/python/apache_beam/options/pipeline_options_validator_test.py
 ##
 @@ -329,7 +329,11 @@ def get_validator(matcher):
 ]
 
 for case in test_case:
-  errors = get_validator(case['on_success_matcher']).validate()
+  matcher = case['on_success_matcher']
+  if matcher and type(matcher) is bytes:
+errors = get_validator(matcher.decode('utf-8')).validate()
 
 Review comment:
   I think I would instead change line 315 to:
   ```
   options.append('%s=%s' % ('--on_success_matcher=' , matcher.decode()))
   ```
   line 155 in test_pipeline.py to: 
   ```
   options.append('--%s=%s' % (k, pickler.dumps(v).decode()))
   ```
   and change `abc` to `b'abc'` in one of the test cases.
   
   What do you all think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144494)
Time Spent: 3h  (was: 2h 50m)

> Finish Python 3 porting for options module
> --
>
> Key: BEAM-5317
> URL: https://issues.apache.org/jira/browse/BEAM-5317
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Manu Zhang
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4841) Test Auto JIRA Subtask 789

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou updated BEAM-4841:

Fix Version/s: (was: Not applicable)
   2.8.0

> Test Auto JIRA Subtask 789
> --
>
> Key: BEAM-4841
> URL: https://issues.apache.org/jira/browse/BEAM-4841
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: 2.8.0
>
>
> Test JIRA issue creation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5317) Finish Python 3 porting for options module

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5317?focusedWorklogId=144489=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144489
 ]

ASF GitHub Bot logged work on BEAM-5317:


Author: ASF GitHub Bot
Created on: 14/Sep/18 22:39
Start Date: 14/Sep/18 22:39
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #6397: 
[BEAM-5317] Finish Python3 porting for options module
URL: https://github.com/apache/beam/pull/6397#discussion_r217858460
 
 

 ##
 File path: sdks/python/apache_beam/options/pipeline_options_validator_test.py
 ##
 @@ -329,7 +329,11 @@ def get_validator(matcher):
 ]
 
 for case in test_case:
-  errors = get_validator(case['on_success_matcher']).validate()
+  matcher = case['on_success_matcher']
+  if matcher and type(matcher) is bytes:
+errors = get_validator(matcher.decode('utf-8')).validate()
 
 Review comment:
   @superbobry notice that we are are using a wrapper around pickle: 
https://github.com/apache/beam/blob/859b8d5c1aade54ea94b1778ea4b889a526e5fa5/sdks/python/apache_beam/internal/pickler.py#L23
 so I think decoding using 'utf-8' or even 'latin-1' should be ok. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144489)
Time Spent: 2h 50m  (was: 2h 40m)

> Finish Python 3 porting for options module
> --
>
> Key: BEAM-5317
> URL: https://issues.apache.org/jira/browse/BEAM-5317
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Manu Zhang
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1467

[Apache Jenkins Server]

See 


Changes:

[aaltay] [BEAM-5375] KafkaIO : Handle runtime exceptions while fetching from

--
[...truncated 23.16 MB...]
... 21 more
Caused by: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Value must 
not be NULL in table users.
at io.grpc.Status.asRuntimeException(Status.java:526)
at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:468)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
com.google.cloud.spanner.spi.v1.WatchdogInterceptor$MonitoredCall$1.onClose(WatchdogInterceptor.java:190)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
at 
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at 
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
... 3 more

Sep 14, 2018 10:28:56 PM 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn processElement
WARNING: Failed to submit the mutation group
com.google.cloud.spanner.SpannerException: FAILED_PRECONDITION: 
io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Value must not be NULL in 
table users.
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:80)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:456)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.commit(GrpcSpannerRpc.java:404)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:797)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:794)
at 
com.google.cloud.spanner.SpannerImpl.runWithRetries(SpannerImpl.java:227)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl.writeAtLeastOnce(SpannerImpl.java:793)
at 
com.google.cloud.spanner.SessionPool$PooledSession.writeAtLeastOnce(SessionPool.java:319)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:60)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn.processElement(SpannerIO.java:1103)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:275)
at 
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:240)
at 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #1482

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5383) Migrate integration tests for python bigquery io read

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?focusedWorklogId=144485=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144485
 ]

ASF GitHub Bot logged work on BEAM-5383:


Author: ASF GitHub Bot
Created on: 14/Sep/18 22:23
Start Date: 14/Sep/18 22:23
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6393: [BEAM-5383] migrate 
bigquer_io_read_it_test to beam
URL: https://github.com/apache/beam/pull/6393#issuecomment-421499651
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144485)
Time Spent: 2.5h  (was: 2h 20m)

> Migrate integration tests for python  bigquery io read 
> ---
>
> Key: BEAM-5383
> URL: https://issues.apache.org/jira/browse/BEAM-5383
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Py_ValCont #686

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #18

[Apache Jenkins Server]

See 


Changes:

[pablo] Adding TopWikipediaSessions Integration test

[pablo] Fixing issues

[github] Adding TODO

--
[...truncated 6.28 MB...]
[grpc-default-executor-2] INFO 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService - 
Beam Fn Control client connected with id 1
[grpc-default-executor-2] INFO sdk_worker.run - Got work 1
[grpc-default-executor-2] INFO sdk_worker.run - Got work 6
[grpc-default-executor-2] INFO sdk_worker.run - Got work 5
[grpc-default-executor-2] INFO sdk_worker.run - Got work 4
[grpc-default-executor-2] INFO sdk_worker.run - Got work 3
[grpc-default-executor-2] INFO sdk_worker.run - Got work 2
[grpc-default-executor-1] INFO sdk_worker.run - Got work 8
[grpc-default-executor-1] INFO sdk_worker.create_state_handler - Creating 
channel for localhost:38829
[grpc-default-executor-1] INFO sdk_worker.run - Got work 7
[grpc-default-executor-1] INFO data_plane.create_data_channel - Creating 
channel for localhost:42853
[grpc-default-executor-1] INFO 
org.apache.beam.runners.fnexecution.data.GrpcDataService - Beam Fn Data client 
connected.
[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (a0929a621acabd718983914a3c10ea6a) switched from 
RUNNING to FINISHED.
[Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (a0929a621acabd718983914a3c10ea6a).
[Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task Source: Collection Source 
-> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (a0929a621acabd718983914a3c10ea6a) [FINISHED]
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task Source: 
Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem a0929a621acabd718983914a3c10ea6a.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Collection 
Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (a0929a621acabd718983914a3c10ea6a) switched from 
RUNNING to FINISHED.
[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO sdk_worker.run - Got work 9
[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Source: Collection Source -> 

[beam] 01/01: Merge pull request #6311 from pabloem/twsit

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 859b8d5c1aade54ea94b1778ea4b889a526e5fa5
Merge: c7d134a cc6a57d
Author: Pablo 
AuthorDate: Fri Sep 14 15:08:17 2018 -0700

Merge pull request #6311 from pabloem/twsit

[BEAM-5377] Adding TopWikipediaSessions integration test

 .../examples/complete/TopWikipediaSessions.java| 20 +--
 .../examples/complete/TopWikipediaSessionsIT.java  | 69 ++
 2 files changed, 84 insertions(+), 5 deletions(-)

[beam] branch master updated (c7d134a -> 859b8d5)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c7d134a  Merge pull request #6297 from pabloem/custom-coder-pc
 add 092b629  Adding TopWikipediaSessions Integration test
 add 90075a2  Fixing issues
 add cc6a57d  Adding TODO
 new 859b8d5  Merge pull request #6311 from pabloem/twsit

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../examples/complete/TopWikipediaSessions.java| 20 +
 .../TopWikipediaSessionsIT.java}   | 35 ++
 2 files changed, 31 insertions(+), 24 deletions(-)
 copy examples/java/src/test/java/org/apache/beam/examples/{WordCountIT.java => 
complete/TopWikipediaSessionsIT.java} (66%)

[jira] [Work logged] (BEAM-5317) Finish Python 3 porting for options module

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5317?focusedWorklogId=144477=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144477
 ]

ASF GitHub Bot logged work on BEAM-5317:


Author: ASF GitHub Bot
Created on: 14/Sep/18 22:02
Start Date: 14/Sep/18 22:02
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #6397: 
[BEAM-5317] Finish Python3 porting for options module
URL: https://github.com/apache/beam/pull/6397#discussion_r217853282
 
 

 ##
 File path: sdks/python/apache_beam/options/pipeline_options_validator_test.py
 ##
 @@ -329,7 +329,11 @@ def get_validator(matcher):
 ]
 
 for case in test_case:
-  errors = get_validator(case['on_success_matcher']).validate()
+  matcher = case['on_success_matcher']
+  if matcher and type(matcher) is bytes:
+errors = get_validator(matcher.decode('utf-8')).validate()
 
 Review comment:
   I think we also need to fix the code that is tested by this test. It looks 
like we are using a pipeline option  
https://github.com/apache/beam/blob/c7d134a35dca488f0cbdf607013a6027599b96ca/sdks/python/apache_beam/options/pipeline_options.py#L671
 to pass a pickled piece of code that performs some validation logic in test 
pipelines. In Python 3 these pickled objects will be returned as bytes, which 
we will attempt to concat to a string: 
https://github.com/apache/beam/blob/c7d134a35dca488f0cbdf607013a6027599b96ca/sdks/python/apache_beam/testing/test_pipeline.py#L155
 
   This test is running into the same problem.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144477)
Time Spent: 2h 40m  (was: 2.5h)

> Finish Python 3 porting for options module
> --
>
> Key: BEAM-5317
> URL: https://issues.apache.org/jira/browse/BEAM-5317
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Manu Zhang
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #1481

[Apache Jenkins Server]

See 


Changes:

[pablo] Adding a custom coder test

[github] Adding license

--
[...truncated 28.47 MB...]
[dag-scheduler-event-loop] INFO org.apache.spark.SparkContext - Created 
broadcast 107 from broadcast at DAGScheduler.scala:1039
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - 
Submitting 4 missing tasks from ResultStage 547 (MapPartitionsRDD[2773] at map 
at TranslationUtils.java:129) (first 15 tasks are for partitions Vector(0, 1, 
2, 3))
[dag-scheduler-event-loop] INFO 
org.apache.spark.scheduler.TaskSchedulerImpl - Adding task set 547.0 with 4 
tasks
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager - 
Starting task 0.0 in stage 547.0 (TID 463, localhost, executor driver, 
partition 0, PROCESS_LOCAL, 8308 bytes)
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager - 
Starting task 1.0 in stage 547.0 (TID 464, localhost, executor driver, 
partition 1, PROCESS_LOCAL, 8308 bytes)
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager - 
Starting task 2.0 in stage 547.0 (TID 465, localhost, executor driver, 
partition 2, PROCESS_LOCAL, 8308 bytes)
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager - 
Starting task 3.0 in stage 547.0 (TID 466, localhost, executor driver, 
partition 3, PROCESS_LOCAL, 8308 bytes)
[Executor task launch worker for task 463] INFO 
org.apache.spark.executor.Executor - Running task 0.0 in stage 547.0 (TID 463)
[Executor task launch worker for task 464] INFO 
org.apache.spark.executor.Executor - Running task 1.0 in stage 547.0 (TID 464)
[Executor task launch worker for task 465] INFO 
org.apache.spark.executor.Executor - Running task 2.0 in stage 547.0 (TID 465)
[Executor task launch worker for task 466] INFO 
org.apache.spark.executor.Executor - Running task 3.0 in stage 547.0 (TID 466)
[Executor task launch worker for task 466] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty 
blocks out of 5 blocks
[Executor task launch worker for task 463] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty 
blocks out of 5 blocks
[Executor task launch worker for task 463] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches 
in 0 ms
[Executor task launch worker for task 466] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches 
in 0 ms
[Executor task launch worker for task 463] INFO 
org.apache.spark.storage.BlockManager - Found block rdd_2441_0 locally
[Executor task launch worker for task 466] INFO 
org.apache.spark.storage.BlockManager - Found block rdd_2441_3 locally
[Executor task launch worker for task 466] INFO 
org.apache.spark.storage.memory.MemoryStore - Block rdd_2756_3 stored as bytes 
in memory (estimated size 4.0 B, free 13.5 GB)
[Executor task launch worker for task 463] INFO 
org.apache.spark.storage.memory.MemoryStore - Block rdd_2756_0 stored as bytes 
in memory (estimated size 4.0 B, free 13.5 GB)
[Executor task launch worker for task 464] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty 
blocks out of 5 blocks
[Executor task launch worker for task 465] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty 
blocks out of 5 blocks
[Executor task launch worker for task 464] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches 
in 0 ms
[Executor task launch worker for task 465] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches 
in 0 ms
[dispatcher-event-loop-0] INFO org.apache.spark.storage.BlockManagerInfo - 
Added rdd_2756_0 in memory on localhost:36991 (size: 4.0 B, free: 13.5 GB)
[Executor task launch worker for task 465] INFO 
org.apache.spark.storage.BlockManager - Found block rdd_2441_2 locally
[Executor task launch worker for task 464] INFO 
org.apache.spark.storage.BlockManager - Found block rdd_2441_1 locally
[dispatcher-event-loop-0] INFO org.apache.spark.storage.BlockManagerInfo - 
Added rdd_2756_3 in memory on localhost:36991 (size: 4.0 B, free: 13.5 GB)
[Executor task launch worker for task 465] INFO 
org.apache.spark.storage.memory.MemoryStore - Block rdd_2756_2 stored as bytes 
in memory (estimated size 4.0 B, free 13.5 GB)
[Executor task launch worker for task 464] INFO 
org.apache.spark.storage.memory.MemoryStore - Block rdd_2756_1 stored as bytes 
in memory (estimated size 4.0 B, free 13.5 GB)
[dispatcher-event-loop-3] INFO org.apache.spark.storage.BlockManagerInfo - 
Added rdd_2756_2 in memory on localhost:36991 (size: 4.0 B, free: 13.5 GB)
[dispatcher-event-loop-3] INFO org.apache.spark.storage.BlockManagerInfo - 

Jenkins build is back to normal : beam_PostCommit_Python_PVR_Flink_Gradle #17

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Py_ValCont #685

[Apache Jenkins Server]

See 
<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/685/display/redirect?page=changes>

Changes:

[pablo] Adding a custom coder test

[github] Adding license

--
[...truncated 111.73 KB...]

> Task :beam-sdks-go:resolveBuildDependencies
Resolving golang.org/x/net: commit='2fb46b16b8dda405028c50f7c7f0f9dd1fa6bfb1', 
urls=[https://go.googlesource.com/net]
Resolving golang.org/x/oauth2: 
commit='a032972e28060ca4f5644acffae3dfc268cc09db', 
urls=[https://go.googlesource.com/oauth2]
Resolving golang.org/x/sync: commit='fd80eb99c8f653c847d294a001bdf2a3a6f768f5', 
urls=[https://go.googlesource.com/sync]
Resolving golang.org/x/sys: commit='37707fdb30a5b38865cfb95e5aab41707daec7fd', 
urls=[https://go.googlesource.com/sys]
Resolving cached github.com/coreos/etcd: 
commit='11214aa33bf5a47d3d9d8dafe0f6b97237dfe921', 
urls=[https://github.com/coreos/etcd.git, g...@github.com:coreos/etcd.git]
Resolving cached github.com/coreos/etcd: 
commit='11214aa33bf5a47d3d9d8dafe0f6b97237dfe921', 
urls=[https://github.com/coreos/etcd.git, g...@github.com:coreos/etcd.git]
Resolving google.golang.org/api: 
commit='386d4e5f4f92f86e6aec85985761bba4b938a2d5', 
urls=[https://code.googlesource.com/google-api-go-client]
Resolving google.golang.org/genproto: 
commit='2b5a72b8730b0b16380010cfe5286c42108d88e7', 
urls=[https://github.com/google/go-genproto]
Resolving google.golang.org/grpc: 
commit='7646b5360d049a7ca31e9133315db43456f39e2e', 
urls=[https://github.com/grpc/grpc-go]
Resolving cached github.com/coreos/etcd: 
commit='11214aa33bf5a47d3d9d8dafe0f6b97237dfe921', 
urls=[https://github.com/coreos/etcd.git, g...@github.com:coreos/etcd.git]
Resolving cached github.com/coreos/etcd: 
commit='11214aa33bf5a47d3d9d8dafe0f6b97237dfe921', 
urls=[https://github.com/coreos/etcd.git, g...@github.com:coreos/etcd.git]

> Task :beam-sdks-go:installDependencies
> Task :beam-sdks-go:buildLinuxAmd64
> Task :beam-sdks-go:build

> Task :beam-sdks-python-container:resolveBuildDependencies
Resolving 
./github.com/apache/beam/sdks/go@<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/go>

> Task :beam-sdks-python-container:installDependencies
> Task :beam-sdks-python-container:buildLinuxAmd64
> Task :beam-sdks-python-container:build
> Task :beam-sdks-python-container:dockerPrepare
> Task :beam-sdks-python-container:docker

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 5.0.
See 
https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 5m 25s
16 actionable tasks: 15 executed, 1 up-to-date

Publishing build scan...
https://gradle.com/s/bflxdupdqmwzq


# Verify it exists
docker images | grep $TAG
us.gcr.io/apache-beam-testing/jenkins/python   20180914-212709 e5f32cda505c 
   Less than a second ago   1.61GB

# Push the container
gcloud docker -- push $CONTAINER
The push refers to a repository [us.gcr.io/apache-beam-testing/jenkins/python]
75370dbfed35: Preparing
3eada468234f: Preparing
9a2eac1756cb: Preparing
a94417eddb78: Preparing
bb9c21da9d58: Preparing
263d03b62eba: Preparing
679645bb925b: Preparing
130f23df7a35: Preparing
0d184ba72737: Preparing
a19cb627cc73: Preparing
263d03b62eba: Waiting
ab016c9ea8f8: Preparing
2eb1c9bfc5ea: Preparing
679645bb925b: Waiting
0d184ba72737: Waiting
130f23df7a35: Waiting
a19cb627cc73: Waiting
ab016c9ea8f8: Waiting
0b703c74a09c: Preparing
2eb1c9bfc5ea: Waiting
b28ef0b6fef8: Preparing
b28ef0b6fef8: Waiting
3eada468234f: Layer already exists
a94417eddb78: Layer already exists
75370dbfed35: Layer already exists
9a2eac1756cb: Layer already exists
bb9c21da9d58: Layer already exists
263d03b62eba: Layer already exists
679645bb925b: Layer already exists
130f23df7a35: Layer already exists
0d184ba72737: Layer already exists
a19cb627cc73: Layer already exists
ab016c9ea8f8: Layer already exists
0b703c74a09c: Layer already exists
2eb1c9bfc5ea: Layer already exists
b28ef0b6fef8: Layer already exists
20180910-234438: digest: 
sha256:a72d812627cfc4527d11b2ddb8d6f584230ce76966b3fa43a61ffc33444b6e13 size: 
3278
7a7f18b001e6: Preparing
6839f54e7716: Preparing
270836ec6d83: Preparing
d5795008aed6: Preparing
90e8e3984e66: Preparing
263d03b62eba: Preparing
679645bb925b: Preparing
130f23df7a35: Preparing
0d184ba72737: Preparing
a19cb627cc73: Preparing
ab016c9ea8f8: Preparing
2eb1c9bfc5ea: Preparing
0b703c74a09c: Preparing
b28ef0b6fef8: Preparing
679645bb925b: Waiting
130f23df7a35: Waiting
0d184ba72737: Waiting
a19cb627cc73: Waiting
ab016c9ea8f8: Waiting
2eb1c9bfc5ea: Waiting
263d03b62eba: Waiting
0b703c74a09c: Waiting
b28ef0b6fef8: Waiting
90e8e3984e66: Pushed
130f23df7a35: Layer already exists
0d184ba72737: Layer already exists
a19cb627cc73: Layer already exists
ab016c9ea8f8: Layer already exists
2eb1c9bfc5ea: Layer already exists
263d03b62eba: Layer already exists
679645bb925b: Layer already exists
b28ef0b6fef8: Layer already

[beam] branch master updated (416275a -> c7d134a)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 416275a  Merge pull request #6399 from 
yifanzou/BEAM-5389/fix-java-example-integration-test
 add 799dfe5  Adding a custom coder test
 add 504aa28  Adding license
 new c7d134a  Merge pull request #6297 from pabloem/custom-coder-pc

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sdk/coders/PCollectionCustomCoderTest.java | 252 +
 1 file changed, 252 insertions(+)
 create mode 100644 
sdks/java/core/src/test/java/org/apache/beam/sdk/coders/PCollectionCustomCoderTest.java

[beam] 01/01: Merge pull request #6297 from pabloem/custom-coder-pc

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c7d134a35dca488f0cbdf607013a6027599b96ca
Merge: 416275a 504aa28
Author: Pablo 
AuthorDate: Fri Sep 14 14:26:47 2018 -0700

Merge pull request #6297 from pabloem/custom-coder-pc

[BEAM-5377] Adding a custom coder integration test for Beam

 .../sdk/coders/PCollectionCustomCoderTest.java | 252 +
 1 file changed, 252 insertions(+)

[jira] [Created] (BEAM-5390) Simplify special-casing of TopWikipediaSessions JSON deserializer

[Pablo Estrada (JIRA)]

Pablo Estrada created BEAM-5390:
---

 Summary: Simplify special-casing of TopWikipediaSessions JSON 
deserializer
 Key: BEAM-5390
 URL: https://issues.apache.org/jira/browse/BEAM-5390
 Project: Beam
  Issue Type: Bug
  Components: examples-java
Reporter: Pablo Estrada
Assignee: Pablo Estrada


Different runners may deserialize JSON numbers into different types (depending 
on configuration). This example is for now using a workaround to avoid trouble.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #16

[Apache Jenkins Server]

See 


Changes:

[yifanzou] [BEAM-5389] fix the checksum assertion in the TfIdf IT

--
[...truncated 6.27 MB...]
[grpc-default-executor-0] INFO sdk_worker_main.main - 
semi_persistent_directory: /tmp
[grpc-default-executor-0] WARN sdk_worker_main._load_main_session - No session 
file found: /tmp/staged/pickled_main_session. Functions defined in __main__ 
(interactive session) may fail. 
[grpc-default-executor-0] INFO sdk_worker_main.main - Python sdk harness 
started with pipeline_options: {'runner': u'None', 'streaming': True, 
'experiments': [u'beam_fn_api'], 'sdk_location': u'container', 'job_name': 
u'test_windowing_1536959355.34', 'job_endpoint': u'localhost:51979'}
[grpc-default-executor-0] INFO sdk_worker.__init__ - Creating insecure control 
channel.
[grpc-default-executor-0] INFO sdk_worker.__init__ - Control channel 
established.
[grpc-default-executor-0] INFO sdk_worker.__init__ - Initializing SDKHarness 
with 12 workers.
[grpc-default-executor-0] INFO 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService - 
Beam Fn Control client connected with id 1
[grpc-default-executor-0] INFO sdk_worker.run - Got work 2
[grpc-default-executor-1] INFO sdk_worker.run - Got work 5
[grpc-default-executor-1] INFO sdk_worker.run - Got work 6
[grpc-default-executor-1] INFO sdk_worker.run - Got work 4
[grpc-default-executor-1] INFO sdk_worker.run - Got work 1
[grpc-default-executor-0] INFO sdk_worker.run - Got work 3
[grpc-default-executor-0] INFO sdk_worker.run - Got work 7
[grpc-default-executor-0] INFO sdk_worker.create_state_handler - Creating 
channel for localhost:45493
[grpc-default-executor-0] INFO sdk_worker.run - Got work 8
[grpc-default-executor-0] INFO data_plane.create_data_channel - Creating 
channel for localhost:34541
[grpc-default-executor-0] INFO 
org.apache.beam.runners.fnexecution.data.GrpcDataService - Beam Fn Data client 
connected.
[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (230db8a0d3e3e2bdb96c5d38232e4dce) switched from 
RUNNING to FINISHED.
[Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (230db8a0d3e3e2bdb96c5d38232e4dce).
[Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task Source: Collection Source 
-> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (230db8a0d3e3e2bdb96c5d38232e4dce) [FINISHED]
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task Source: 
Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem 230db8a0d3e3e2bdb96c5d38232e4dce.
[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO 

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #15

[Apache Jenkins Server]

See 


Changes:

[thw] [BEAM-5385] Flink job server to use REST port from --flink-master-url

--
[...truncated 6.27 MB...]
[grpc-default-executor-0] INFO 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService - 
Beam Fn Control client connected with id 1
[grpc-default-executor-0] INFO sdk_worker.run - Got work 1
[grpc-default-executor-0] INFO sdk_worker.run - Got work 4
[grpc-default-executor-0] INFO sdk_worker.run - Got work 3
[grpc-default-executor-0] INFO sdk_worker.run - Got work 2
[grpc-default-executor-0] INFO sdk_worker.run - Got work 7
[grpc-default-executor-0] INFO sdk_worker.run - Got work 5
[grpc-default-executor-0] INFO sdk_worker.create_state_handler - Creating 
channel for localhost:39097
[grpc-default-executor-0] INFO sdk_worker.run - Got work 6
[grpc-default-executor-0] INFO sdk_worker.run - Got work 8
[grpc-default-executor-0] INFO 
org.apache.beam.runners.fnexecution.data.GrpcDataService - Beam Fn Data client 
connected.
[grpc-default-executor-0] INFO data_plane.create_data_channel - Creating 
channel for localhost:46285
[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (a8552008d3523c0cd6dd48b7561a0a68) switched from 
RUNNING to FINISHED.
[Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (a8552008d3523c0cd6dd48b7561a0a68).
[Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task Source: Collection Source 
-> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (a8552008d3523c0cd6dd48b7561a0a68) [FINISHED]
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task Source: 
Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem a8552008d3523c0cd6dd48b7561a0a68.
[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Collection 
Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (a8552008d3523c0cd6dd48b7561a0a68) switched from 
RUNNING to FINISHED.
[grpc-default-executor-0] INFO sdk_worker.run - Got work 9
[Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) 

[jira] [Work logged] (BEAM-5389) [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5389?focusedWorklogId=144465=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144465
 ]

ASF GitHub Bot logged work on BEAM-5389:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:51
Start Date: 14/Sep/18 20:51
Worklog Time Spent: 10m 
  Work Description: pabloem closed pull request #6399: [BEAM-5389] fix the 
checksum assertion in the TfIdf IT
URL: https://github.com/apache/beam/pull/6399
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/examples/java/src/test/java/org/apache/beam/examples/complete/TfIdfIT.java 
b/examples/java/src/test/java/org/apache/beam/examples/complete/TfIdfIT.java
index 21ec6b22cc9..e47e45cc4a0 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/complete/TfIdfIT.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/complete/TfIdfIT.java
@@ -37,7 +37,7 @@
 public class TfIdfIT {
 
   private static final String DEFAULT_INPUT = 
"gs://apache-beam-samples/shakespeare/";
-  private static final String EXPECTED_OUTPUT_CHECKSUM = 
"f072786dde51dc09fc74bf38ffbfc27dcfdf0b96";
+  private static final String EXPECTED_OUTPUT_CHECKSUM = 
"0549d1dc8821976121771aefcb0e2297177bdb88";
   private static final Pattern DEFAULT_SHARD_TEMPLATE =
   Pattern.compile("(?x) \\S* (? \\d+) -of- (? 
\\d+)\\.csv");
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144465)
Time Spent: 1.5h  (was: 1h 20m)

> [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]
>  
> -
>
> Key: BEAM-5389
> URL: https://issues.apache.org/jira/browse/BEAM-5389
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1465/]
> Relevant log:
> 11:10:52.145 [LIFECYCLE] 
> [org.gradle.internal.operations.DefaultBuildOperationExecutor] > Task 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> org.apache.beam.examples.complete.TfIdfIT > testE2ETfIdf FAILED
> 11:10:57.266 [DEBUG] [TestEventLogger] java.lang.AssertionError: 
> 11:10:57.266 [DEBUG] [TestEventLogger] Expected: Expected checksum is 
> (f072786dde51dc09fc74bf38ffbfc27dcfdf0b96)
> 11:10:57.266 [DEBUG] [TestEventLogger] but: was 
> (0549d1dc8821976121771aefcb0e2297177bdb88)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:138)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdf.runTfIdf(TfIdf.java:427)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdfIT.testE2ETfIdf(TfIdfIT.java:72)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> java.lang.reflect.Method.invoke(Method.java:498)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> 

[jira] [Work logged] (BEAM-5389) [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5389?focusedWorklogId=144464=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144464
 ]

ASF GitHub Bot logged work on BEAM-5389:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:51
Start Date: 14/Sep/18 20:51
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #6399: [BEAM-5389] fix the 
checksum assertion in the TfIdf IT
URL: https://github.com/apache/beam/pull/6399#issuecomment-421480809
 
 
   PostCommits have passed. Merging.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144464)
Time Spent: 1h 20m  (was: 1h 10m)

> [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]
>  
> -
>
> Key: BEAM-5389
> URL: https://issues.apache.org/jira/browse/BEAM-5389
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1465/]
> Relevant log:
> 11:10:52.145 [LIFECYCLE] 
> [org.gradle.internal.operations.DefaultBuildOperationExecutor] > Task 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> org.apache.beam.examples.complete.TfIdfIT > testE2ETfIdf FAILED
> 11:10:57.266 [DEBUG] [TestEventLogger] java.lang.AssertionError: 
> 11:10:57.266 [DEBUG] [TestEventLogger] Expected: Expected checksum is 
> (f072786dde51dc09fc74bf38ffbfc27dcfdf0b96)
> 11:10:57.266 [DEBUG] [TestEventLogger] but: was 
> (0549d1dc8821976121771aefcb0e2297177bdb88)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:138)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdf.runTfIdf(TfIdf.java:427)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdfIT.testE2ETfIdf(TfIdfIT.java:72)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> java.lang.reflect.Method.invoke(Method.java:498)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> 

[beam] 01/01: Merge pull request #6399 from yifanzou/BEAM-5389/fix-java-example-integration-test

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 416275a1c1762c54ea8bbaf07d551fdb7d161713
Merge: 41aaf77 8b77f9a
Author: Pablo 
AuthorDate: Fri Sep 14 13:51:11 2018 -0700

Merge pull request #6399 from 
yifanzou/BEAM-5389/fix-java-example-integration-test

[BEAM-5389] fix the checksum assertion in the TfIdf IT

 .../java/src/test/java/org/apache/beam/examples/complete/TfIdfIT.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[beam] branch master updated (41aaf77 -> 416275a)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 41aaf77  Merge pull request #6398: [BEAM-5385] Flink job server to use 
REST port from --flink-master-url instead of default.
 add 8b77f9a  [BEAM-5389] fix the checksum assertion in the TfIdf IT
 new 416275a  Merge pull request #6399 from 
yifanzou/BEAM-5389/fix-java-example-integration-test

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/src/test/java/org/apache/beam/examples/complete/TfIdfIT.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[jira] [Work logged] (BEAM-5385) Flink jobserver does not honor --flink-master-url

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5385?focusedWorklogId=144463=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144463
 ]

ASF GitHub Bot logged work on BEAM-5385:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:49
Start Date: 14/Sep/18 20:49
Worklog Time Spent: 10m 
  Work Description: tweise closed pull request #6398: [BEAM-5385] Flink job 
server to use REST port from --flink-master-url instead of default.
URL: https://github.com/apache/beam/pull/6398
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
index 1162a7ed95a..9d16f3f55b7 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
@@ -21,6 +21,8 @@
 import java.util.List;
 import org.apache.flink.api.java.CollectionEnvironment;
 import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.RestOptions;
 import org.apache.flink.runtime.state.AbstractStateBackend;
 import org.apache.flink.streaming.api.TimeCharacteristic;
 import 
org.apache.flink.streaming.api.environment.CheckpointConfig.ExternalizedCheckpointCleanup;
@@ -101,10 +103,13 @@ public static StreamExecutionEnvironment 
createStreamExecutionEnvironment(
   flinkStreamEnv = StreamExecutionEnvironment.getExecutionEnvironment();
 } else if (masterUrl.matches(".*:\\d*")) {
   List parts = Splitter.on(':').splitToList(masterUrl);
+  Configuration clientConfig = new Configuration();
+  clientConfig.setInteger(RestOptions.PORT, 
Integer.parseInt(parts.get(1)));
   flinkStreamEnv =
   StreamExecutionEnvironment.createRemoteEnvironment(
   parts.get(0),
   Integer.parseInt(parts.get(1)),
+  clientConfig,
   filesToStage.toArray(new String[filesToStage.size()]));
 } else {
   LOG.warn("Unrecognized Flink Master URL {}. Defaulting to [auto].", 
masterUrl);


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144463)
Time Spent: 40m  (was: 0.5h)

> Flink jobserver does not honor --flink-master-url
> -
>
> Key: BEAM-5385
> URL: https://issues.apache.org/jira/browse/BEAM-5385
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> It will use the external Flink cluster when specified, but only with default 
> port number 8081, because the actual port is not in propagated in 
> FlinkExecutionEnvironments (RestOptions.PORT setting).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (9b85316 -> 41aaf77)

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9b85316  [BEAM-5375] KafkaIO : Handle runtime exceptions while 
fetching from Kafka better.  (#6391)
 add 84a571e  [BEAM-5385] Flink job server to use REST port from 
--flink-master-url instead of default.
 new 41aaf77  Merge pull request #6398: [BEAM-5385] Flink job server to use 
REST port from --flink-master-url instead of default.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/runners/flink/FlinkExecutionEnvironments.java| 5 +
 1 file changed, 5 insertions(+)

[beam] 01/01: Merge pull request #6398: [BEAM-5385] Flink job server to use REST port from --flink-master-url instead of default.

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 41aaf778a78c68834be5c476555124210a721afe
Merge: 9b85316 84a571e
Author: Thomas Weise 
AuthorDate: Fri Sep 14 13:49:33 2018 -0700

Merge pull request #6398: [BEAM-5385] Flink job server to use REST port 
from --flink-master-url instead of default.

 .../org/apache/beam/runners/flink/FlinkExecutionEnvironments.java| 5 +
 1 file changed, 5 insertions(+)

diff --cc 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
index 853d4f3,9d16f3f..b5a1c8a
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
@@@ -19,9 -19,10 +19,11 @@@ package org.apache.beam.runners.flink
  
  import com.google.common.base.Splitter;
  import java.util.List;
 +import org.apache.flink.api.common.ExecutionConfig;
  import org.apache.flink.api.java.CollectionEnvironment;
  import org.apache.flink.api.java.ExecutionEnvironment;
+ import org.apache.flink.configuration.Configuration;
+ import org.apache.flink.configuration.RestOptions;
  import org.apache.flink.runtime.state.AbstractStateBackend;
  import org.apache.flink.streaming.api.TimeCharacteristic;
  import 
org.apache.flink.streaming.api.environment.CheckpointConfig.ExternalizedCheckpointCleanup;

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #14

[Apache Jenkins Server]

See 


Changes:

[aaltay] [BEAM-5375] KafkaIO : Handle runtime exceptions while fetching from

--
[...truncated 6.27 MB...]
[grpc-default-executor-0] INFO sdk_worker.__init__ - Creating insecure control 
channel.
[grpc-default-executor-0] INFO sdk_worker.__init__ - Control channel 
established.
[grpc-default-executor-0] INFO sdk_worker.__init__ - Initializing SDKHarness 
with 12 workers.
[grpc-default-executor-0] INFO 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService - 
Beam Fn Control client connected with id 1
[grpc-default-executor-0] INFO sdk_worker.run - Got work 1
[grpc-default-executor-0] INFO sdk_worker.run - Got work 3
[grpc-default-executor-1] INFO sdk_worker.run - Got work 2
[grpc-default-executor-1] INFO sdk_worker.run - Got work 6
[grpc-default-executor-1] INFO sdk_worker.run - Got work 4
[grpc-default-executor-1] INFO sdk_worker.run - Got work 5
[grpc-default-executor-1] INFO sdk_worker.run - Got work 7
[grpc-default-executor-1] INFO sdk_worker.create_state_handler - Creating 
channel for localhost:36657
[grpc-default-executor-1] INFO sdk_worker.run - Got work 8
[grpc-default-executor-0] INFO data_plane.create_data_channel - Creating 
channel for localhost:41801
[grpc-default-executor-0] INFO 
org.apache.beam.runners.fnexecution.data.GrpcDataService - Beam Fn Data client 
connected.
[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (90d1a6707bc8f2c32eae1b17e3aebf8c) switched from 
RUNNING to FINISHED.
[Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (90d1a6707bc8f2c32eae1b17e3aebf8c).
[Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task Source: Collection Source 
-> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (90d1a6707bc8f2c32eae1b17e3aebf8c) [FINISHED]
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task Source: 
Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem 90d1a6707bc8f2c32eae1b17e3aebf8c.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Collection 
Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (90d1a6707bc8f2c32eae1b17e3aebf8c) switched from 
RUNNING to FINISHED.
[grpc-default-executor-0] INFO sdk_worker.run - Got work 9
[Source: Collection Source -> 

[jira] [Work logged] (BEAM-5383) Migrate integration tests for python bigquery io read

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?focusedWorklogId=144461=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144461
 ]

ASF GitHub Bot logged work on BEAM-5383:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:41
Start Date: 14/Sep/18 20:41
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6393: [BEAM-5383] migrate 
bigquer_io_read_it_test to beam
URL: https://github.com/apache/beam/pull/6393#issuecomment-421478566
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144461)
Time Spent: 2h 10m  (was: 2h)

> Migrate integration tests for python  bigquery io read 
> ---
>
> Key: BEAM-5383
> URL: https://issues.apache.org/jira/browse/BEAM-5383
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5383) Migrate integration tests for python bigquery io read

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?focusedWorklogId=144462=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144462
 ]

ASF GitHub Bot logged work on BEAM-5383:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:41
Start Date: 14/Sep/18 20:41
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6393: [BEAM-5383] 
migrate bigquer_io_read_it_test to beam
URL: https://github.com/apache/beam/pull/6393#issuecomment-421475394
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144462)
Time Spent: 2h 20m  (was: 2h 10m)

> Migrate integration tests for python  bigquery io read 
> ---
>
> Key: BEAM-5383
> URL: https://issues.apache.org/jira/browse/BEAM-5383
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5375) KafkaIO reader should handle runtime exceptions kafka client

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5375?focusedWorklogId=144460=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144460
 ]

ASF GitHub Bot logged work on BEAM-5375:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:39
Start Date: 14/Sep/18 20:39
Worklog Time Spent: 10m 
  Work Description: rangadi commented on issue #6391: [BEAM-5375] KafkaIO : 
Handle runtime exceptions while fetching from Kafka better. 
URL: https://github.com/apache/beam/pull/6391#issuecomment-421478149
 
 
   Thanks @aaltay!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144460)
Time Spent: 1h 20m  (was: 1h 10m)

> KafkaIO reader should handle runtime exceptions kafka client
> 
>
> Key: BEAM-5375
> URL: https://issues.apache.org/jira/browse/BEAM-5375
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kafka
>Affects Versions: 2.7.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> KafkaIO reader might stop reading from Kafka without any explicit error 
> message if KafkaConsumer throws a runtime exception while polling for 
> messages. One of the Dataflow customers encountered this issue (see [user@ 
> thread|[https://lists.apache.org/thread.html/c0cf8f45f567a0623592e2d8340f5288e3e774b59bca985aec410a81@%3Cuser.beam.apache.org%3E])]
> 'consumerPollThread()' in KafkaIO deliberately avoided catching runtime 
> exceptions. It shoud handle it.. stuff happens at runtime. 
> It should result in 'IOException' from start()/advance(). The runners will 
> handle properly reporting and closing the readers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5375) KafkaIO reader should handle runtime exceptions kafka client

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5375?focusedWorklogId=144459=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144459
 ]

ASF GitHub Bot logged work on BEAM-5375:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:37
Start Date: 14/Sep/18 20:37
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #6391: [BEAM-5375] KafkaIO : 
Handle runtime exceptions while fetching from Kafka better. 
URL: https://github.com/apache/beam/pull/6391
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
index de241a87676..68ea8127753 100644
--- 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
+++ 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
@@ -172,7 +172,6 @@ public boolean start() throws IOException {
 offsetFetcherThread.scheduleAtFixedRate(
 this::updateLatestOffsets, 0, OFFSET_UPDATE_INTERVAL_SECONDS, 
TimeUnit.SECONDS);
 
-nextBatch();
 return advance();
   }
 
@@ -376,6 +375,7 @@ public long getSplitBacklogBytes() {
   // like 100 milliseconds does not work well. This along with large receive 
buffer for
   // consumer achieved best throughput in tests (see 
`defaultConsumerProperties`).
   private final ExecutorService consumerPollThread = 
Executors.newSingleThreadExecutor();
+  private AtomicReference consumerPollException = new 
AtomicReference<>();
   private final SynchronousQueue> 
availableRecordsQueue =
   new SynchronousQueue<>();
   private AtomicReference finalizedCheckpointMark = new 
AtomicReference<>();
@@ -391,7 +391,7 @@ public long getSplitBacklogBytes() {
   Executors.newSingleThreadScheduledExecutor();
   private static final int OFFSET_UPDATE_INTERVAL_SECONDS = 1;
 
-  static final long UNINITIALIZED_OFFSET = -1;
+  private static final long UNINITIALIZED_OFFSET = -1;
 
   //Add SpEL instance to cover the interface difference of Kafka client
   private transient ConsumerSpEL consumerSpEL;
@@ -570,28 +570,33 @@ Instant updateAndGetWatermark() {
   private void consumerPollLoop() {
 // Read in a loop and enqueue the batch of records, if any, to 
availableRecordsQueue.
 
-ConsumerRecords records = ConsumerRecords.empty();
-while (!closed.get()) {
-  try {
-if (records.isEmpty()) {
-  records = consumer.poll(KAFKA_POLL_TIMEOUT.getMillis());
-} else if (availableRecordsQueue.offer(
-records, RECORDS_ENQUEUE_POLL_TIMEOUT.getMillis(), 
TimeUnit.MILLISECONDS)) {
-  records = ConsumerRecords.empty();
-}
-KafkaCheckpointMark checkpointMark = 
finalizedCheckpointMark.getAndSet(null);
-if (checkpointMark != null) {
-  commitCheckpointMark(checkpointMark);
+try {
+  ConsumerRecords records = ConsumerRecords.empty();
+  while (!closed.get()) {
+try {
+  if (records.isEmpty()) {
+records = consumer.poll(KAFKA_POLL_TIMEOUT.getMillis());
+  } else if (availableRecordsQueue.offer(
+  records, RECORDS_ENQUEUE_POLL_TIMEOUT.getMillis(), 
TimeUnit.MILLISECONDS)) {
+records = ConsumerRecords.empty();
+  }
+  KafkaCheckpointMark checkpointMark = 
finalizedCheckpointMark.getAndSet(null);
+  if (checkpointMark != null) {
+commitCheckpointMark(checkpointMark);
+  }
+} catch (InterruptedException e) {
+  LOG.warn("{}: consumer thread is interrupted", this, e); // not 
expected
+  break;
+} catch (WakeupException e) {
+  break;
 }
-  } catch (InterruptedException e) {
-LOG.warn("{}: consumer thread is interrupted", this, e); // not 
expected
-break;
-  } catch (WakeupException e) {
-break;
   }
+  LOG.info("{}: Returning from consumer pool loop", this);
+} catch (Exception e) { // mostly an unrecoverable KafkaException.
+  LOG.error("{}: Exception while reading from Kafka", this, e);
+  consumerPollException.set(e);
+  throw e;
 }
-
-LOG.info("{}: Returning from consumer pool loop", this);
   }
 
   private void commitCheckpointMark(KafkaCheckpointMark checkpointMark) {
@@ -622,7 +627,7 @@ void finalizeCheckpointMarkAsync(KafkaCheckpointMark 
checkpointMark) {
 checkpointMarkCommitsEnqueued.inc();
   }
 
-  private void nextBatch() {
+  private void nextBatch() throws IOException {
 curBatch = Collections.emptyIterator();
 
 

[beam] branch master updated (5d07d9f -> 9b85316)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5d07d9f  Merge pull request #6276: [BEAM-5219] Make OutgoingMessage 
public
 add 9b85316  [BEAM-5375] KafkaIO : Handle runtime exceptions while 
fetching from Kafka better.  (#6391)

No new revisions were added by this update.

Summary of changes:
 .../beam/sdk/io/kafka/KafkaUnboundedReader.java| 55 +-
 .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java  | 40 
 2 files changed, 72 insertions(+), 23 deletions(-)

[jira] [Work logged] (BEAM-5389) [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5389?focusedWorklogId=144458=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144458
 ]

ASF GitHub Bot logged work on BEAM-5389:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:31
Start Date: 14/Sep/18 20:31
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6399: [BEAM-5389] 
fix the checksum assertion in the TfIdf IT
URL: https://github.com/apache/beam/pull/6399#issuecomment-421451763
 
 
   Run Java PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144458)
Time Spent: 1h 10m  (was: 1h)

> [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]
>  
> -
>
> Key: BEAM-5389
> URL: https://issues.apache.org/jira/browse/BEAM-5389
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1465/]
> Relevant log:
> 11:10:52.145 [LIFECYCLE] 
> [org.gradle.internal.operations.DefaultBuildOperationExecutor] > Task 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> org.apache.beam.examples.complete.TfIdfIT > testE2ETfIdf FAILED
> 11:10:57.266 [DEBUG] [TestEventLogger] java.lang.AssertionError: 
> 11:10:57.266 [DEBUG] [TestEventLogger] Expected: Expected checksum is 
> (f072786dde51dc09fc74bf38ffbfc27dcfdf0b96)
> 11:10:57.266 [DEBUG] [TestEventLogger] but: was 
> (0549d1dc8821976121771aefcb0e2297177bdb88)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:138)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdf.runTfIdf(TfIdf.java:427)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdfIT.testE2ETfIdf(TfIdfIT.java:72)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> java.lang.reflect.Method.invoke(Method.java:498)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> 

[jira] [Work logged] (BEAM-5389) [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5389?focusedWorklogId=144457=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144457
 ]

ASF GitHub Bot logged work on BEAM-5389:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:31
Start Date: 14/Sep/18 20:31
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6399: [BEAM-5389] fix the 
checksum assertion in the TfIdf IT
URL: https://github.com/apache/beam/pull/6399#issuecomment-421476033
 
 
   Run Java PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144457)
Time Spent: 1h  (was: 50m)

> [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]
>  
> -
>
> Key: BEAM-5389
> URL: https://issues.apache.org/jira/browse/BEAM-5389
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1465/]
> Relevant log:
> 11:10:52.145 [LIFECYCLE] 
> [org.gradle.internal.operations.DefaultBuildOperationExecutor] > Task 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> org.apache.beam.examples.complete.TfIdfIT > testE2ETfIdf FAILED
> 11:10:57.266 [DEBUG] [TestEventLogger] java.lang.AssertionError: 
> 11:10:57.266 [DEBUG] [TestEventLogger] Expected: Expected checksum is 
> (f072786dde51dc09fc74bf38ffbfc27dcfdf0b96)
> 11:10:57.266 [DEBUG] [TestEventLogger] but: was 
> (0549d1dc8821976121771aefcb0e2297177bdb88)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:138)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdf.runTfIdf(TfIdf.java:427)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdfIT.testE2ETfIdf(TfIdfIT.java:72)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> java.lang.reflect.Method.invoke(Method.java:498)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> 

[jira] [Work logged] (BEAM-5383) Migrate integration tests for python bigquery io read

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?focusedWorklogId=144450=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144450
 ]

ASF GitHub Bot logged work on BEAM-5383:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:28
Start Date: 14/Sep/18 20:28
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6393: [BEAM-5383] 
migrate bigquer_io_read_it_test to beam
URL: https://github.com/apache/beam/pull/6393#issuecomment-421182760
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144450)
Time Spent: 1.5h  (was: 1h 20m)

> Migrate integration tests for python  bigquery io read 
> ---
>
> Key: BEAM-5383
> URL: https://issues.apache.org/jira/browse/BEAM-5383
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5383) Migrate integration tests for python bigquery io read

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?focusedWorklogId=144452=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144452
 ]

ASF GitHub Bot logged work on BEAM-5383:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:28
Start Date: 14/Sep/18 20:28
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6393: [BEAM-5383] 
migrate bigquer_io_read_it_test to beam
URL: https://github.com/apache/beam/pull/6393#issuecomment-421461253
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144452)
Time Spent: 1h 50m  (was: 1h 40m)

> Migrate integration tests for python  bigquery io read 
> ---
>
> Key: BEAM-5383
> URL: https://issues.apache.org/jira/browse/BEAM-5383
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5383) Migrate integration tests for python bigquery io read

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?focusedWorklogId=19=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-19
 ]

ASF GitHub Bot logged work on BEAM-5383:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:28
Start Date: 14/Sep/18 20:28
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6393: [BEAM-5383] 
migrate bigquer_io_read_it_test to beam
URL: https://github.com/apache/beam/pull/6393#issuecomment-421182225
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 19)
Time Spent: 1h 20m  (was: 1h 10m)

> Migrate integration tests for python  bigquery io read 
> ---
>
> Key: BEAM-5383
> URL: https://issues.apache.org/jira/browse/BEAM-5383
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5383) Migrate integration tests for python bigquery io read

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?focusedWorklogId=144451=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144451
 ]

ASF GitHub Bot logged work on BEAM-5383:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:28
Start Date: 14/Sep/18 20:28
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6393: [BEAM-5383] 
migrate bigquer_io_read_it_test to beam
URL: https://github.com/apache/beam/pull/6393#issuecomment-421195395
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144451)
Time Spent: 1h 40m  (was: 1.5h)

> Migrate integration tests for python  bigquery io read 
> ---
>
> Key: BEAM-5383
> URL: https://issues.apache.org/jira/browse/BEAM-5383
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5383) Migrate integration tests for python bigquery io read

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?focusedWorklogId=144453=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144453
 ]

ASF GitHub Bot logged work on BEAM-5383:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:28
Start Date: 14/Sep/18 20:28
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6393: [BEAM-5383] 
migrate bigquer_io_read_it_test to beam
URL: https://github.com/apache/beam/pull/6393#issuecomment-421471693
 
 
   Run Python PostCommit
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144453)
Time Spent: 2h  (was: 1h 50m)

> Migrate integration tests for python  bigquery io read 
> ---
>
> Key: BEAM-5383
> URL: https://issues.apache.org/jira/browse/BEAM-5383
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=144454=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144454
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:28
Start Date: 14/Sep/18 20:28
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #6328: [BEAM-4176] 
enumerate primitive transforms in portable construction
URL: https://github.com/apache/beam/pull/6328#issuecomment-421475492
 
 
   Based upon community feedback, I opened up 
https://github.com/ryan-williams/beam/pull/2 against your branch. Please merge 
or advise on different course of action.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144454)
Time Spent: 19h 10m  (was: 19h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot 
> 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 19h 10m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5383) Migrate integration tests for python bigquery io read

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?focusedWorklogId=18=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-18
 ]

ASF GitHub Bot logged work on BEAM-5383:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:28
Start Date: 14/Sep/18 20:28
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6393: [BEAM-5383] migrate 
bigquer_io_read_it_test to beam
URL: https://github.com/apache/beam/pull/6393#issuecomment-421475394
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 18)
Time Spent: 1h 10m  (was: 1h)

> Migrate integration tests for python  bigquery io read 
> ---
>
> Key: BEAM-5383
> URL: https://issues.apache.org/jira/browse/BEAM-5383
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5288) Modify Environment to support non-dockerized SDK harness deployments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5288?focusedWorklogId=15=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-15
 ]

ASF GitHub Bot logged work on BEAM-5288:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:19
Start Date: 14/Sep/18 20:19
Worklog Time Spent: 10m 
  Work Description: angoenka edited a comment on issue #6373: [BEAM-5288] 
[NOT READY FOR MERGE]Enhance Environment proto to support different types of 
environments
URL: https://github.com/apache/beam/pull/6373#issuecomment-421473275
 
 
   @tweise The dataflow runner already use the field, of course we can make the 
breaking change and fix it in dataflow by tagging but to do a smooth 
transition, I hope to keep the URL field and remove in a subsequent PR.
   Absolutely. Once its done,I will squash the PR appropriately. 
   
   @mxm Absolutely. I will mark URL as deprecated and get rid of it ASAP.
   Yes, I think I missed the GO code. I wonder why it did not break. Probably 
we need to regenerate the proto code as generated code is also checked in.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 15)
Time Spent: 2h 50m  (was: 2h 40m)

> Modify Environment to support non-dockerized SDK harness deployments 
> -
>
> Key: BEAM-5288
> URL: https://issues.apache.org/jira/browse/BEAM-5288
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Maximilian Michels
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> As of mailing discussions and BEAM-5187, it has become clear that we need to 
> extend the Environment information. In addition to the Docker environment, 
> the extended environment holds deployment options for 1) a process-based 
> environment, 2) an externally managed environment.
> The proto definition, as of now, looks as follows:
> {noformat}
>  message Environment {
>// (Required) The URN of the payload
>string urn = 1;
>// (Optional) The data specifying any parameters to the URN. If
>// the URN does not require any arguments, this may be omitted.
>bytes payload = 2;
>  }
>  message StandardEnvironments {
>enum Environments {
>  DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
>  PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
>  EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
>}
>  }
>  // The payload of a Docker image
>  message DockerPayload {
>string container_image = 1;  // implicitly linux_amd64.
>  }
>  message ProcessPayload {
>string os = 1;  // "linux", "darwin", ..
>string arch = 2;  // "amd64", ..
>string command = 3; // process to execute
>map env = 4; // environment variables
>  }
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5288) Modify Environment to support non-dockerized SDK harness deployments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5288?focusedWorklogId=14=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-14
 ]

ASF GitHub Bot logged work on BEAM-5288:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:18
Start Date: 14/Sep/18 20:18
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6373: [BEAM-5288] [NOT 
READY FOR MERGE]Enhance Environment proto to support different types of 
environments
URL: https://github.com/apache/beam/pull/6373#issuecomment-421473275
 
 
   @tweise The dataflow runner already use the field, of course we can make the 
breaking change and fix it in dataflow by tagging but to do a smooth 
transition, I hope to keep the URL field and remove in a subsequent PR.
   
   @mxm Absolutely. I will mark URL as deprecated and get rid of it ASAP.
   Yes, I think I missed the GO code. I wonder why it did not break. Probably 
we need to regenerate the proto code as generated code is also checked in.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 14)
Time Spent: 2h 40m  (was: 2.5h)

> Modify Environment to support non-dockerized SDK harness deployments 
> -
>
> Key: BEAM-5288
> URL: https://issues.apache.org/jira/browse/BEAM-5288
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Maximilian Michels
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> As of mailing discussions and BEAM-5187, it has become clear that we need to 
> extend the Environment information. In addition to the Docker environment, 
> the extended environment holds deployment options for 1) a process-based 
> environment, 2) an externally managed environment.
> The proto definition, as of now, looks as follows:
> {noformat}
>  message Environment {
>// (Required) The URN of the payload
>string urn = 1;
>// (Optional) The data specifying any parameters to the URN. If
>// the URN does not require any arguments, this may be omitted.
>bytes payload = 2;
>  }
>  message StandardEnvironments {
>enum Environments {
>  DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
>  PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
>  EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
>}
>  }
>  // The payload of a Docker image
>  message DockerPayload {
>string container_image = 1;  // implicitly linux_amd64.
>  }
>  message ProcessPayload {
>string os = 1;  // "linux", "darwin", ..
>string arch = 2;  // "amd64", ..
>string command = 3; // process to execute
>map env = 4; // environment variables
>  }
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5385) Flink jobserver does not honor --flink-master-url

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5385?focusedWorklogId=13=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-13
 ]

ASF GitHub Bot logged work on BEAM-5385:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:14
Start Date: 14/Sep/18 20:14
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6398: 
[BEAM-5385] Flink job server to use REST port from --flink-master-url instead 
of default.
URL: https://github.com/apache/beam/pull/6398#discussion_r217829993
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
 ##
 @@ -101,10 +103,13 @@ public static StreamExecutionEnvironment 
createStreamExecutionEnvironment(
   flinkStreamEnv = StreamExecutionEnvironment.getExecutionEnvironment();
 } else if (masterUrl.matches(".*:\\d*")) {
   List parts = Splitter.on(':').splitToList(masterUrl);
 
 Review comment:
   Indeed.. BTW this bug only surfaces in the streaming execution environment - 
batch was working. The joy of having different API for batch and streaming.. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 13)
Time Spent: 0.5h  (was: 20m)

> Flink jobserver does not honor --flink-master-url
> -
>
> Key: BEAM-5385
> URL: https://issues.apache.org/jira/browse/BEAM-5385
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> It will use the external Flink cluster when specified, but only with default 
> port number 8081, because the actual port is not in propagated in 
> FlinkExecutionEnvironments (RestOptions.PORT setting).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5389) [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5389?focusedWorklogId=12=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-12
 ]

ASF GitHub Bot logged work on BEAM-5389:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:13
Start Date: 14/Sep/18 20:13
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6399: [BEAM-5389] fix the 
checksum assertion in the TfIdf IT
URL: https://github.com/apache/beam/pull/6399#issuecomment-421471871
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 12)
Time Spent: 50m  (was: 40m)

> [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]
>  
> -
>
> Key: BEAM-5389
> URL: https://issues.apache.org/jira/browse/BEAM-5389
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1465/]
> Relevant log:
> 11:10:52.145 [LIFECYCLE] 
> [org.gradle.internal.operations.DefaultBuildOperationExecutor] > Task 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> org.apache.beam.examples.complete.TfIdfIT > testE2ETfIdf FAILED
> 11:10:57.266 [DEBUG] [TestEventLogger] java.lang.AssertionError: 
> 11:10:57.266 [DEBUG] [TestEventLogger] Expected: Expected checksum is 
> (f072786dde51dc09fc74bf38ffbfc27dcfdf0b96)
> 11:10:57.266 [DEBUG] [TestEventLogger] but: was 
> (0549d1dc8821976121771aefcb0e2297177bdb88)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:138)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdf.runTfIdf(TfIdf.java:427)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdfIT.testE2ETfIdf(TfIdfIT.java:72)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> java.lang.reflect.Method.invoke(Method.java:498)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> 

[jira] [Work logged] (BEAM-5383) Migrate integration tests for python bigquery io read

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?focusedWorklogId=10=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-10
 ]

ASF GitHub Bot logged work on BEAM-5383:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:12
Start Date: 14/Sep/18 20:12
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6393: [BEAM-5383] migrate 
bigquer_io_read_it_test to beam
URL: https://github.com/apache/beam/pull/6393#issuecomment-421471693
 
 
   Run Python PostCommit
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 10)
Time Spent: 1h  (was: 50m)

> Migrate integration tests for python  bigquery io read 
> ---
>
> Key: BEAM-5383
> URL: https://issues.apache.org/jira/browse/BEAM-5383
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5288) Modify Environment to support non-dockerized SDK harness deployments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5288?focusedWorklogId=11=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-11
 ]

ASF GitHub Bot logged work on BEAM-5288:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:12
Start Date: 14/Sep/18 20:12
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6373: 
[BEAM-5288] [NOT READY FOR MERGE]Enhance Environment proto to support different 
types of environments
URL: https://github.com/apache/beam/pull/6373#discussion_r217829613
 
 

 ##
 File path: model/pipeline/src/main/proto/beam_runner_api.proto
 ##
 @@ -963,16 +963,39 @@ message SideInput {
   SdkFunctionSpec window_mapping_fn = 3;
 }
 
-// An environment for executing UDFs. Generally an SDK container URL, but
-// there can be many for a single SDK, for example to provide dependency
-// isolation.
+// An environment for executing UDFs. By default, an SDK container URL, but
+// can also be a process forked by a command, or an externally managed process.
 message Environment {
 
-  // (Required) The URL of a container
-  //
-  // TODO: reconcile with Fn API's DockerContainer structure by
-  // adding adequate metadata to know how to interpret the container
-  string url = 1;
+  // (Required) The URN of the payload
+  string urn = 1;
+
+  // (Optional) The data specifying any parameters to the URN. If
+  // the URN does not require any arguments, this may be omitted.
+  bytes payload = 2;
+}
+
+message StandardEnvironments {
+  enum Environments {
+DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
+
+PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
+
+EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
+  }
 
 Review comment:
   Correct, so perhaps we should also get rid of `EXTERNAL` then?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 11)
Time Spent: 2.5h  (was: 2h 20m)

> Modify Environment to support non-dockerized SDK harness deployments 
> -
>
> Key: BEAM-5288
> URL: https://issues.apache.org/jira/browse/BEAM-5288
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Maximilian Michels
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> As of mailing discussions and BEAM-5187, it has become clear that we need to 
> extend the Environment information. In addition to the Docker environment, 
> the extended environment holds deployment options for 1) a process-based 
> environment, 2) an externally managed environment.
> The proto definition, as of now, looks as follows:
> {noformat}
>  message Environment {
>// (Required) The URN of the payload
>string urn = 1;
>// (Optional) The data specifying any parameters to the URN. If
>// the URN does not require any arguments, this may be omitted.
>bytes payload = 2;
>  }
>  message StandardEnvironments {
>enum Environments {
>  DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
>  PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
>  EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
>}
>  }
>  // The payload of a Docker image
>  message DockerPayload {
>string container_image = 1;  // implicitly linux_amd64.
>  }
>  message ProcessPayload {
>string os = 1;  // "linux", "darwin", ..
>string arch = 2;  // "amd64", ..
>string command = 3; // process to execute
>map env = 4; // environment variables
>  }
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5288) Modify Environment to support non-dockerized SDK harness deployments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5288?focusedWorklogId=144437=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144437
 ]

ASF GitHub Bot logged work on BEAM-5288:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:06
Start Date: 14/Sep/18 20:06
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6373: 
[BEAM-5288] [NOT READY FOR MERGE]Enhance Environment proto to support different 
types of environments
URL: https://github.com/apache/beam/pull/6373#discussion_r217828345
 
 

 ##
 File path: model/pipeline/src/main/proto/beam_runner_api.proto
 ##
 @@ -963,16 +963,39 @@ message SideInput {
   SdkFunctionSpec window_mapping_fn = 3;
 }
 
-// An environment for executing UDFs. Generally an SDK container URL, but
-// there can be many for a single SDK, for example to provide dependency
-// isolation.
+// An environment for executing UDFs. By default, an SDK container URL, but
+// can also be a process forked by a command, or an externally managed process.
 message Environment {
 
-  // (Required) The URL of a container
-  //
-  // TODO: reconcile with Fn API's DockerContainer structure by
-  // adding adequate metadata to know how to interpret the container
-  string url = 1;
+  // (Required) The URN of the payload
+  string urn = 1;
+
+  // (Optional) The data specifying any parameters to the URN. If
+  // the URN does not require any arguments, this may be omitted.
+  bytes payload = 2;
+}
+
+message StandardEnvironments {
+  enum Environments {
+DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
+
+PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
+
+EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
+  }
 
 Review comment:
   Do you think we should add Embedded yet? It seems the least fleshed out and 
will only support Java pipelines.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144437)
Time Spent: 2h 20m  (was: 2h 10m)

> Modify Environment to support non-dockerized SDK harness deployments 
> -
>
> Key: BEAM-5288
> URL: https://issues.apache.org/jira/browse/BEAM-5288
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Maximilian Michels
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> As of mailing discussions and BEAM-5187, it has become clear that we need to 
> extend the Environment information. In addition to the Docker environment, 
> the extended environment holds deployment options for 1) a process-based 
> environment, 2) an externally managed environment.
> The proto definition, as of now, looks as follows:
> {noformat}
>  message Environment {
>// (Required) The URN of the payload
>string urn = 1;
>// (Optional) The data specifying any parameters to the URN. If
>// the URN does not require any arguments, this may be omitted.
>bytes payload = 2;
>  }
>  message StandardEnvironments {
>enum Environments {
>  DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
>  PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
>  EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
>}
>  }
>  // The payload of a Docker image
>  message DockerPayload {
>string container_image = 1;  // implicitly linux_amd64.
>  }
>  message ProcessPayload {
>string os = 1;  // "linux", "darwin", ..
>string arch = 2;  // "amd64", ..
>string command = 3; // process to execute
>map env = 4; // environment variables
>  }
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5288) Modify Environment to support non-dockerized SDK harness deployments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5288?focusedWorklogId=144435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144435
 ]

ASF GitHub Bot logged work on BEAM-5288:


Author: ASF GitHub Bot
Created on: 14/Sep/18 20:01
Start Date: 14/Sep/18 20:01
Worklog Time Spent: 10m 
  Work Description: mxm commented on issue #6373: [BEAM-5288] [NOT READY 
FOR MERGE]Enhance Environment proto to support different types of environments
URL: https://github.com/apache/beam/pull/6373#issuecomment-421469136
 
 
   @angoenka You want to keep the URL field in Environment? Ok from my side if 
the rest of the changes remain. We should annotate the field as deprecated and 
remove it as quickly as possible.
   
   I think we also need to adopt Go code before we remove the URL field.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144435)
Time Spent: 2h 10m  (was: 2h)

> Modify Environment to support non-dockerized SDK harness deployments 
> -
>
> Key: BEAM-5288
> URL: https://issues.apache.org/jira/browse/BEAM-5288
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Maximilian Michels
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> As of mailing discussions and BEAM-5187, it has become clear that we need to 
> extend the Environment information. In addition to the Docker environment, 
> the extended environment holds deployment options for 1) a process-based 
> environment, 2) an externally managed environment.
> The proto definition, as of now, looks as follows:
> {noformat}
>  message Environment {
>// (Required) The URN of the payload
>string urn = 1;
>// (Optional) The data specifying any parameters to the URN. If
>// the URN does not require any arguments, this may be omitted.
>bytes payload = 2;
>  }
>  message StandardEnvironments {
>enum Environments {
>  DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
>  PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
>  EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
>}
>  }
>  // The payload of a Docker image
>  message DockerPayload {
>string container_image = 1;  // implicitly linux_amd64.
>  }
>  message ProcessPayload {
>string os = 1;  // "linux", "darwin", ..
>string arch = 2;  // "amd64", ..
>string command = 3; // process to execute
>map env = 4; // environment variables
>  }
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1466

[Apache Jenkins Server]

See 


--
[...truncated 23.42 MB...]
... 21 more
Caused by: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Value must 
not be NULL in table users.
at io.grpc.Status.asRuntimeException(Status.java:526)
at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:468)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
com.google.cloud.spanner.spi.v1.WatchdogInterceptor$MonitoredCall$1.onClose(WatchdogInterceptor.java:190)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
at 
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at 
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
... 3 more

Sep 14, 2018 7:49:36 PM 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn processElement
WARNING: Failed to submit the mutation group
com.google.cloud.spanner.SpannerException: FAILED_PRECONDITION: 
io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Value must not be NULL in 
table users.
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:80)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:456)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.commit(GrpcSpannerRpc.java:404)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:797)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:794)
at 
com.google.cloud.spanner.SpannerImpl.runWithRetries(SpannerImpl.java:227)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl.writeAtLeastOnce(SpannerImpl.java:793)
at 
com.google.cloud.spanner.SessionPool$PooledSession.writeAtLeastOnce(SessionPool.java:319)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:60)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn.processElement(SpannerIO.java:1103)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:275)
at 
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:240)
at 

[jira] [Work logged] (BEAM-5317) Finish Python 3 porting for options module

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5317?focusedWorklogId=144431=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144431
 ]

ASF GitHub Bot logged work on BEAM-5317:


Author: ASF GitHub Bot
Created on: 14/Sep/18 19:46
Start Date: 14/Sep/18 19:46
Worklog Time Spent: 10m 
  Work Description: superbobry commented on a change in pull request #6397: 
[BEAM-5317] Finish Python3 porting for options module
URL: https://github.com/apache/beam/pull/6397#discussion_r217822040
 
 

 ##
 File path: sdks/python/apache_beam/options/pipeline_options_validator_test.py
 ##
 @@ -329,7 +329,11 @@ def get_validator(matcher):
 ]
 
 for case in test_case:
-  errors = get_validator(case['on_success_matcher']).validate()
+  matcher = case['on_success_matcher']
+  if matcher and type(matcher) is bytes:
 
 Review comment:
   Is there a reason for not using `isinstance` here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144431)
Time Spent: 2.5h  (was: 2h 20m)

> Finish Python 3 porting for options module
> --
>
> Key: BEAM-5317
> URL: https://issues.apache.org/jira/browse/BEAM-5317
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Manu Zhang
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5317) Finish Python 3 porting for options module

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5317?focusedWorklogId=144430=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144430
 ]

ASF GitHub Bot logged work on BEAM-5317:


Author: ASF GitHub Bot
Created on: 14/Sep/18 19:46
Start Date: 14/Sep/18 19:46
Worklog Time Spent: 10m 
  Work Description: superbobry commented on a change in pull request #6397: 
[BEAM-5317] Finish Python3 porting for options module
URL: https://github.com/apache/beam/pull/6397#discussion_r217822880
 
 

 ##
 File path: sdks/python/apache_beam/options/pipeline_options_validator_test.py
 ##
 @@ -329,7 +329,11 @@ def get_validator(matcher):
 ]
 
 for case in test_case:
-  errors = get_validator(case['on_success_matcher']).validate()
+  matcher = case['on_success_matcher']
+  if matcher and type(matcher) is bytes:
+errors = get_validator(matcher.decode('utf-8')).validate()
 
 Review comment:
   Are you sure you want to decode the bytes here? Looking at `test_case`, if 
the matcher type is bytes, then the content is almost surely not decodeable 
using UTF-8:
   
   ```python
   >>> import pickle
   >>> matcher = pickle.dumps(object)
   >>> matcher
   b'\x80\x03cbuiltins\nobject\nq\x00.'
   >>> matcher.decode("utf-8")
   Traceback (most recent call last):
 File "", line 1, in 
   UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: 
invalid start byte
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144430)
Time Spent: 2h 20m  (was: 2h 10m)

> Finish Python 3 porting for options module
> --
>
> Key: BEAM-5317
> URL: https://issues.apache.org/jira/browse/BEAM-5317
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Manu Zhang
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5385) Flink jobserver does not honor --flink-master-url

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5385?focusedWorklogId=144427=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144427
 ]

ASF GitHub Bot logged work on BEAM-5385:


Author: ASF GitHub Bot
Created on: 14/Sep/18 19:40
Start Date: 14/Sep/18 19:40
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6398: 
[BEAM-5385] Flink job server to use REST port from --flink-master-url instead 
of default.
URL: https://github.com/apache/beam/pull/6398#discussion_r217808715
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
 ##
 @@ -101,10 +103,13 @@ public static StreamExecutionEnvironment 
createStreamExecutionEnvironment(
   flinkStreamEnv = StreamExecutionEnvironment.getExecutionEnvironment();
 } else if (masterUrl.matches(".*:\\d*")) {
   List parts = Splitter.on(':').splitToList(masterUrl);
 
 Review comment:
   Wonder how such splitter would work for IPv6 addresses :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144427)
Time Spent: 20m  (was: 10m)

> Flink jobserver does not honor --flink-master-url
> -
>
> Key: BEAM-5385
> URL: https://issues.apache.org/jira/browse/BEAM-5385
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> It will use the external Flink cluster when specified, but only with default 
> port number 8081, because the actual port is not in propagated in 
> FlinkExecutionEnvironments (RestOptions.PORT setting).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5383) Migrate integration tests for python bigquery io read

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5383?focusedWorklogId=144426=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144426
 ]

ASF GitHub Bot logged work on BEAM-5383:


Author: ASF GitHub Bot
Created on: 14/Sep/18 19:29
Start Date: 14/Sep/18 19:29
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6393: [BEAM-5383] migrate 
bigquer_io_read_it_test to beam
URL: https://github.com/apache/beam/pull/6393#issuecomment-421461253
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144426)
Time Spent: 50m  (was: 40m)

> Migrate integration tests for python  bigquery io read 
> ---
>
> Key: BEAM-5383
> URL: https://issues.apache.org/jira/browse/BEAM-5383
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5323) Finish Python 3 porting for utils module

[Robbe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robbe resolved BEAM-5323.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Finish Python 3 porting for utils module
> 
>
> Key: BEAM-5323
> URL: https://issues.apache.org/jira/browse/BEAM-5323
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5323) Finish Python 3 porting for utils module

[Robbe (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615263#comment-16615263
 ] 

Robbe commented on BEAM-5323:
-

Yes, I'll close this.

> Finish Python 3 porting for utils module
> 
>
> Key: BEAM-5323
> URL: https://issues.apache.org/jira/browse/BEAM-5323
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5378) Ensure all Go SDK examples run successfully

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5378?focusedWorklogId=144424=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144424
 ]

ASF GitHub Bot logged work on BEAM-5378:


Author: ASF GitHub Bot
Created on: 14/Sep/18 19:02
Start Date: 14/Sep/18 19:02
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #6395: [BEAM-5378] Update 
go wordcap example to work on Dataflow runner
URL: https://github.com/apache/beam/pull/6395#issuecomment-421454652
 
 
   LGTM
   
   On Fri, Sep 14, 2018, 11:21 AM Ahmet Altay  wrote:
   
   > *@aaltay* commented on this pull request.
   > --
   >
   > In sdks/go/examples/wordcap/wordcap.go
   > :
   >
   > > @@ -31,7 +30,7 @@ import (
   >  )
   >
   >  var (
   > -  input = flag.String("input", 
os.ExpandEnv("$GOPATH/src/github.com/apache/beam/sdks/go/data/haiku/old_pond.txt"),
 "Files to read.")
   > +  input = flag.String("input", 
"gs://apache-beam-samples/shakespeare/kinglear.txt", "File(s) to read.")
   >
   > Yes, I tried that it does. (This is similar to the wordcount example.) I
   > believe this is a public file in GCS and does not require any auth to read
   > it.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > , or mute
   > the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144424)
Time Spent: 50m  (was: 40m)

> Ensure all Go SDK examples run successfully
> ---
>
> Key: BEAM-5378
> URL: https://issues.apache.org/jira/browse/BEAM-5378
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Tomas Roos
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> I've been spending a day or so running through the example available for the 
> Go SDK in order to see what works and on what runner (direct, dataflow), and 
> what doesn't and here's the results.
> All available examples for the go sdk. For me as a new developer on apache 
> beam and dataflow it would be a tremendous value to have all examples running 
> because many of them have legitimate use-cases behind them. 
> {code:java}
> ├── complete
> │   └── autocomplete
> │   └── autocomplete.go
> ├── contains
> │   └── contains.go
> ├── cookbook
> │   ├── combine
> │   │   └── combine.go
> │   ├── filter
> │   │   └── filter.go
> │   ├── join
> │   │   └── join.go
> │   ├── max
> │   │   └── max.go
> │   └── tornadoes
> │   └── tornadoes.go
> ├── debugging_wordcount
> │   └── debugging_wordcount.go
> ├── forest
> │   └── forest.go
> ├── grades
> │   └── grades.go
> ├── minimal_wordcount
> │   └── minimal_wordcount.go
> ├── multiout
> │   └── multiout.go
> ├── pingpong
> │   └── pingpong.go
> ├── streaming_wordcap
> │   └── wordcap.go
> ├── windowed_wordcount
> │   └── windowed_wordcount.go
> ├── wordcap
> │   └── wordcap.go
> ├── wordcount
> │   └── wordcount.go
> └── yatzy
> └── yatzy.go
> {code}
> All examples that are supposed to be runnable by the direct driver (not 
> depending on gcp platform services) are runnable.
> On the otherhand these are the tests that needs to be updated because its not 
> runnable on the dataflow platform for various reasons.
> I tried to figure them out and all I can do is to pin point at least where it 
> fails since my knowledge so far in the beam / dataflow internals is limited.
> .
> ├── complete
> │   └── autocomplete
> │   └── autocomplete.go
> Runs successfully if swapping the input to one of the shakespear data files 
> from gs://
> But when running this it yields a error from the top.Largest func (discussed 
> in another issue that top.Largest needs to have a serializeable combinator / 
> accumulator)
> ➜  autocomplete git:(master) ✗ ./autocomplete --project fair-app-213019 
> --runner dataflow --staging_location=gs://fair-app-213019/staging-test2 
> --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
>  
> 2018/09/11 15:35:26 Running autocomplete
> Unable to encode combiner for lifting: failed to encode custom coder: bad 
> underlying type: bad field type: bad element: unencodable type: 

[jira] [Work logged] (BEAM-5389) [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5389?focusedWorklogId=144416=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144416
 ]

ASF GitHub Bot logged work on BEAM-5389:


Author: ASF GitHub Bot
Created on: 14/Sep/18 18:53
Start Date: 14/Sep/18 18:53
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #6399: [BEAM-5389] fix the 
checksum assertion in the TfIdf IT
URL: https://github.com/apache/beam/pull/6399#issuecomment-421452286
 
 
   LGTM. I'll wait for tests to run.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144416)
Time Spent: 40m  (was: 0.5h)

> [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]
>  
> -
>
> Key: BEAM-5389
> URL: https://issues.apache.org/jira/browse/BEAM-5389
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1465/]
> Relevant log:
> 11:10:52.145 [LIFECYCLE] 
> [org.gradle.internal.operations.DefaultBuildOperationExecutor] > Task 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> org.apache.beam.examples.complete.TfIdfIT > testE2ETfIdf FAILED
> 11:10:57.266 [DEBUG] [TestEventLogger] java.lang.AssertionError: 
> 11:10:57.266 [DEBUG] [TestEventLogger] Expected: Expected checksum is 
> (f072786dde51dc09fc74bf38ffbfc27dcfdf0b96)
> 11:10:57.266 [DEBUG] [TestEventLogger] but: was 
> (0549d1dc8821976121771aefcb0e2297177bdb88)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:138)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdf.runTfIdf(TfIdf.java:427)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdfIT.testE2ETfIdf(TfIdfIT.java:72)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> java.lang.reflect.Method.invoke(Method.java:498)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> 

[jira] [Assigned] (BEAM-5389) [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou reassigned BEAM-5389:
---

Assignee: yifan zou

> [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]
>  
> -
>
> Key: BEAM-5389
> URL: https://issues.apache.org/jira/browse/BEAM-5389
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1465/]
> Relevant log:
> 11:10:52.145 [LIFECYCLE] 
> [org.gradle.internal.operations.DefaultBuildOperationExecutor] > Task 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> org.apache.beam.examples.complete.TfIdfIT > testE2ETfIdf FAILED
> 11:10:57.266 [DEBUG] [TestEventLogger] java.lang.AssertionError: 
> 11:10:57.266 [DEBUG] [TestEventLogger] Expected: Expected checksum is 
> (f072786dde51dc09fc74bf38ffbfc27dcfdf0b96)
> 11:10:57.266 [DEBUG] [TestEventLogger] but: was 
> (0549d1dc8821976121771aefcb0e2297177bdb88)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:138)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdf.runTfIdf(TfIdf.java:427)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdfIT.testE2ETfIdf(TfIdfIT.java:72)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> java.lang.reflect.Method.invoke(Method.java:498)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 11:10:57.269 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 11:10:57.269 [DEBUG] [TestEventLogger] at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> 11:10:57.269 [DEBUG] [TestEventLogger] at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> 11:10:57.269 [DEBUG] 

[jira] [Work logged] (BEAM-5389) [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5389?focusedWorklogId=144412=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144412
 ]

ASF GitHub Bot logged work on BEAM-5389:


Author: ASF GitHub Bot
Created on: 14/Sep/18 18:51
Start Date: 14/Sep/18 18:51
Worklog Time Spent: 10m 
  Work Description: yifanzou opened a new pull request #6399: [BEAM-5389] 
fix the checksum assertion in the TfIdf IT
URL: https://github.com/apache/beam/pull/6399
 
 
   **Please** add a meaningful description for your change here
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144412)
Time Spent: 10m
Remaining Estimate: 0h

> [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]
>  
> -
>
> Key: BEAM-5389
> URL: https://issues.apache.org/jira/browse/BEAM-5389
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Priority: Major
>  Time Spent: 10m

[jira] [Work logged] (BEAM-5389) [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5389?focusedWorklogId=144414=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144414
 ]

ASF GitHub Bot logged work on BEAM-5389:


Author: ASF GitHub Bot
Created on: 14/Sep/18 18:51
Start Date: 14/Sep/18 18:51
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6399: [BEAM-5389] fix the 
checksum assertion in the TfIdf IT
URL: https://github.com/apache/beam/pull/6399#issuecomment-421451763
 
 
   Run Java PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144414)
Time Spent: 20m  (was: 10m)

> [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]
>  
> -
>
> Key: BEAM-5389
> URL: https://issues.apache.org/jira/browse/BEAM-5389
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1465/]
> Relevant log:
> 11:10:52.145 [LIFECYCLE] 
> [org.gradle.internal.operations.DefaultBuildOperationExecutor] > Task 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> org.apache.beam.examples.complete.TfIdfIT > testE2ETfIdf FAILED
> 11:10:57.266 [DEBUG] [TestEventLogger] java.lang.AssertionError: 
> 11:10:57.266 [DEBUG] [TestEventLogger] Expected: Expected checksum is 
> (f072786dde51dc09fc74bf38ffbfc27dcfdf0b96)
> 11:10:57.266 [DEBUG] [TestEventLogger] but: was 
> (0549d1dc8821976121771aefcb0e2297177bdb88)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:138)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdf.runTfIdf(TfIdf.java:427)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdfIT.testE2ETfIdf(TfIdfIT.java:72)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> java.lang.reflect.Method.invoke(Method.java:498)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 

[jira] [Work logged] (BEAM-5389) [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5389?focusedWorklogId=144415=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144415
 ]

ASF GitHub Bot logged work on BEAM-5389:


Author: ASF GitHub Bot
Created on: 14/Sep/18 18:51
Start Date: 14/Sep/18 18:51
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6399: [BEAM-5389] fix the 
checksum assertion in the TfIdf IT
URL: https://github.com/apache/beam/pull/6399#issuecomment-421451844
 
 
   +R: @pabloem 
   A quick fix on the failing postcommit.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144415)
Time Spent: 0.5h  (was: 20m)

> [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]
>  
> -
>
> Key: BEAM-5389
> URL: https://issues.apache.org/jira/browse/BEAM-5389
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1465/]
> Relevant log:
> 11:10:52.145 [LIFECYCLE] 
> [org.gradle.internal.operations.DefaultBuildOperationExecutor] > Task 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> org.apache.beam.examples.complete.TfIdfIT > testE2ETfIdf FAILED
> 11:10:57.266 [DEBUG] [TestEventLogger] java.lang.AssertionError: 
> 11:10:57.266 [DEBUG] [TestEventLogger] Expected: Expected checksum is 
> (f072786dde51dc09fc74bf38ffbfc27dcfdf0b96)
> 11:10:57.266 [DEBUG] [TestEventLogger] but: was 
> (0549d1dc8821976121771aefcb0e2297177bdb88)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:138)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdf.runTfIdf(TfIdf.java:427)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdfIT.testE2ETfIdf(TfIdfIT.java:72)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> java.lang.reflect.Method.invoke(Method.java:498)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> 

[jira] [Commented] (BEAM-5389) [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]

[yifan zou (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615233#comment-16615233
 ] 

yifan zou commented on BEAM-5389:
-

I am looking on it.

> [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]
>  
> -
>
> Key: BEAM-5389
> URL: https://issues.apache.org/jira/browse/BEAM-5389
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Priority: Major
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1465/]
> Relevant log:
> 11:10:52.145 [LIFECYCLE] 
> [org.gradle.internal.operations.DefaultBuildOperationExecutor] > Task 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> 11:10:57.265 [DEBUG] [TestEventLogger] 
> org.apache.beam.examples.complete.TfIdfIT > testE2ETfIdf FAILED
> 11:10:57.266 [DEBUG] [TestEventLogger] java.lang.AssertionError: 
> 11:10:57.266 [DEBUG] [TestEventLogger] Expected: Expected checksum is 
> (f072786dde51dc09fc74bf38ffbfc27dcfdf0b96)
> 11:10:57.266 [DEBUG] [TestEventLogger] but: was 
> (0549d1dc8821976121771aefcb0e2297177bdb88)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:138)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> 11:10:57.266 [DEBUG] [TestEventLogger] at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdf.runTfIdf(TfIdf.java:427)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.apache.beam.examples.complete.TfIdfIT.testE2ETfIdf(TfIdfIT.java:72)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> java.lang.reflect.Method.invoke(Method.java:498)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 11:10:57.267 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 11:10:57.268 [DEBUG] [TestEventLogger] at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 11:10:57.269 [DEBUG] [TestEventLogger] at 
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 11:10:57.269 [DEBUG] [TestEventLogger] at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> 11:10:57.269 [DEBUG] [TestEventLogger] at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> 11:10:57.269 [DEBUG] [TestEventLogger] at 
> 

[jira] [Work logged] (BEAM-5337) [beam_PostCommit_Java_GradleBuild][:beam-runners-flink_2.11:test][Flake] Build times out in beam-runners-flink target

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5337?focusedWorklogId=144404=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144404
 ]

ASF GitHub Bot logged work on BEAM-5337:


Author: ASF GitHub Bot
Created on: 14/Sep/18 18:44
Start Date: 14/Sep/18 18:44
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6385: 
[BEAM-5337] Fix flaky test UnboundedSourceWrapperTest#testValueEmission
URL: https://github.com/apache/beam/pull/6385#discussion_r217807807
 
 

 ##
 File path: 
runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/UnboundedSourceWrapperTest.java
 ##
 @@ -134,11 +133,13 @@ public void testValueEmission() throws Exception {
   public void run() {
 while (true) {
   try {
-synchronized (testHarness.getCheckpointLock()) {
-  
testHarness.setProcessingTime(System.currentTimeMillis());
-}
+testHarness.setProcessingTime(System.currentTimeMillis());
 Thread.sleep(1000);
+  } catch (InterruptedException e) {
+// this is ok
+break;
   } catch (Exception e) {
+e.printStackTrace();
 
 Review comment:
   +1 If you want to open a PR I'd be happy to ack it!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144404)
Time Spent: 2h  (was: 1h 50m)

> [beam_PostCommit_Java_GradleBuild][:beam-runners-flink_2.11:test][Flake] 
> Build times out in beam-runners-flink target
> -
>
> Key: BEAM-5337
> URL: https://issues.apache.org/jira/browse/BEAM-5337
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink, test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Maximilian Michels
>Priority: Critical
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Job times out. 
>  Failing job url:
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1414/consoleFull]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1406/consoleFull]
> https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1408/consoleFull
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4783) Spark SourceRDD Not Designed With Dynamic Allocation In Mind

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4783?focusedWorklogId=144402=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144402
 ]

ASF GitHub Bot logged work on BEAM-4783:


Author: ASF GitHub Bot
Created on: 14/Sep/18 18:43
Start Date: 14/Sep/18 18:43
Worklog Time Spent: 10m 
  Work Description: kyle-winkelman commented on issue #6181: [BEAM-4783] 
Add bundleSize for splitting BoundedSources.
URL: https://github.com/apache/beam/pull/6181#issuecomment-421449558
 
 
   After looking further into 
[BEAM-1815](https://jira.apache.org/jira/browse/BEAM-1815) my understanding is 
that the double shuffle happened in streaming mode only because there is the 
groupByKey in GroupCombineFunctions.groupByKeyOnly followed by updateStateByKey 
in SparkGroupAlsoByWindowViaWindowSet.groupAlsoByWindow. This will not happen 
in batch mode so I believe we should always use groupByKey() without the 
hashpartitioner when in batch mode. That way we don't squash a large amount of 
data into a few partitions.
   
   If you agree I will change (in TransformTranslator):
   ```java
   JavaRDD 
groupedByKey;
   if 
(context.getSerializableOptions().get().as(SparkPipelineOptions.class).getBundleSize()
   > 0) {
 groupedByKey =
 GroupCombineFunctions.groupByKeyOnlyDefaultPartitioner(inRDD, 
keyCoder, wvCoder);
   } else {
 groupedByKey = GroupCombineFunctions.groupByKeyOnly(inRDD, 
keyCoder, wvCoder);
   }
   ```
   to:
   ```java
   JavaRDD 
groupedByKey =
 GroupCombineFunctions.groupByKeyOnlyDefaultPartitioner(inRDD, 
keyCoder, wvCoder);
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144402)
Time Spent: 2h 50m  (was: 2h 40m)

> Spark SourceRDD Not Designed With Dynamic Allocation In Mind
> 
>
> Key: BEAM-4783
> URL: https://issues.apache.org/jira/browse/BEAM-4783
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Affects Versions: 2.5.0
>Reporter: Kyle Winkelman
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>  Labels: newbie
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> When the spark-runner is used along with the configuration 
> spark.dynamicAllocation.enabled=true the SourceRDD does not detect this. It 
> then falls back to the value calculated in this description:
>   // when running on YARN/SparkDeploy it's the result of max(totalCores, 
> 2).
>   // when running on Mesos it's 8.
>   // when running local it's the total number of cores (local = 1, 
> local[N] = N,
>   // local[*] = estimation of the machine's cores).
>   // ** the configuration "spark.default.parallelism" takes precedence 
> over all of the above **
> So in most cases this default is quite small. This is an issue when using a 
> very large input file as it will only get split in half.
> I believe that when Dynamic Allocation is enable the SourceRDD should use the 
> DEFAULT_BUNDLE_SIZE and possibly expose a SparkPipelineOptions that allows 
> you to change this DEFAULT_BUNDLE_SIZE.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5378) Ensure all Go SDK examples run successfully

[Robert Burke (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615220#comment-16615220
 ] 

Robert Burke commented on BEAM-5378:


1. [~nfis...@junctionbox.ca] You're close! The universal runner (as currently 
written) is really a generic runner for any runner that supports Portable Job 
Submission, for example the Flink Runner. Note that the Flink implementation 
delegates directly to the universal runner

[https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/flink/flink.go#L33]

There's a name collision at present with the Go SDK's "universal" runner, and 
the Universal Local Runner, which is written in Java and serves to demonstrate 
correct Beam Model semantics as a reference to other runner implementations. As 
mentioned, that work to have an easy set up and "it just works" for the local 
runner hasn't been done yet. If a Go package were to exist for that, it would 
be called the "local" runner. That runner would need to be able to additionally 
"spin up" an instance of the ULR at pipeline submission time. It hasn't been 
done yet. I believe the intent is do something with containers to make that 
easier.

2. The graphx package is for marshalling the Go SDK internal representation 
into the Portability representation for Portable runners. Dataflow as an older 
service has other things it needs to do to specify pipelines in a way Dataflow 
understands. Hopefully that will go away.


3. Ideally, but presumably the "direct" runner would always be much lower 
overhead and faster to build and start up than anything that also needs to 
bring up a JVM as well, so the direct runner would continue to serve that 
purpose, at least as a learning tool.
 

> Ensure all Go SDK examples run successfully
> ---
>
> Key: BEAM-5378
> URL: https://issues.apache.org/jira/browse/BEAM-5378
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Tomas Roos
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> I've been spending a day or so running through the example available for the 
> Go SDK in order to see what works and on what runner (direct, dataflow), and 
> what doesn't and here's the results.
> All available examples for the go sdk. For me as a new developer on apache 
> beam and dataflow it would be a tremendous value to have all examples running 
> because many of them have legitimate use-cases behind them. 
> {code:java}
> ├── complete
> │   └── autocomplete
> │   └── autocomplete.go
> ├── contains
> │   └── contains.go
> ├── cookbook
> │   ├── combine
> │   │   └── combine.go
> │   ├── filter
> │   │   └── filter.go
> │   ├── join
> │   │   └── join.go
> │   ├── max
> │   │   └── max.go
> │   └── tornadoes
> │   └── tornadoes.go
> ├── debugging_wordcount
> │   └── debugging_wordcount.go
> ├── forest
> │   └── forest.go
> ├── grades
> │   └── grades.go
> ├── minimal_wordcount
> │   └── minimal_wordcount.go
> ├── multiout
> │   └── multiout.go
> ├── pingpong
> │   └── pingpong.go
> ├── streaming_wordcap
> │   └── wordcap.go
> ├── windowed_wordcount
> │   └── windowed_wordcount.go
> ├── wordcap
> │   └── wordcap.go
> ├── wordcount
> │   └── wordcount.go
> └── yatzy
> └── yatzy.go
> {code}
> All examples that are supposed to be runnable by the direct driver (not 
> depending on gcp platform services) are runnable.
> On the otherhand these are the tests that needs to be updated because its not 
> runnable on the dataflow platform for various reasons.
> I tried to figure them out and all I can do is to pin point at least where it 
> fails since my knowledge so far in the beam / dataflow internals is limited.
> .
> ├── complete
> │   └── autocomplete
> │   └── autocomplete.go
> Runs successfully if swapping the input to one of the shakespear data files 
> from gs://
> But when running this it yields a error from the top.Largest func (discussed 
> in another issue that top.Largest needs to have a serializeable combinator / 
> accumulator)
> ➜  autocomplete git:(master) ✗ ./autocomplete --project fair-app-213019 
> --runner dataflow --staging_location=gs://fair-app-213019/staging-test2 
> --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
>  
> 2018/09/11 15:35:26 Running autocomplete
> Unable to encode combiner for lifting: failed to encode custom coder: bad 
> underlying type: bad field type: bad element: unencodable type: interface 
> {}2018/09/11 15:35:26 Using running binary as worker binary: './autocomplete'
> 2018/09/11 15:35:26 Staging worker binary: ./autocomplete
> ├── contains
> │   └── contains.go
> Fails when running debug.Head for some mysterious reason, might have to do 
> with the param passing into the x,y iterator. Frankly I dont know 

[jira] [Work logged] (BEAM-4783) Spark SourceRDD Not Designed With Dynamic Allocation In Mind

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4783?focusedWorklogId=144397=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144397
 ]

ASF GitHub Bot logged work on BEAM-4783:


Author: ASF GitHub Bot
Created on: 14/Sep/18 18:36
Start Date: 14/Sep/18 18:36
Worklog Time Spent: 10m 
  Work Description: kyle-winkelman commented on issue #6181: [BEAM-4783] 
Add bundleSize for splitting BoundedSources.
URL: https://github.com/apache/beam/pull/6181#issuecomment-421447794
 
 
   In my use case we use spark.dynamicAllocation as a way to remove a knob 
(--num-executors) in our attempt to become knobless; when running in batch mode 
it will create the SourceRDDs and based on the number of partitions it will try 
to spin up that many executors. This completely backfires when the SourceRDD is 
partitioned based on defaultParallelism because that will now be equal to 2 
(default --num-executors).
   
   If you prefer we could prevent the bundleSize from being a knob and always 
use 64MB (Apache Hadoop default block size).
   
   I understand why streaming acts in this way, but for batch the users are 
going to have to guess how many executors they need. If they do not guess high 
enough it is entirely possibly to end up with >2GB of data in a partition 
(https://issues.apache.org/jira/browse/SPARK-6235). Starting at 64MB per 
partition does not eliminate this possibility but it does reduce the chances. 
For example if a user read a 10GB file with 1 executor it would fail if it ever 
tried to cache the partition, but by breaking it into 64MB partitions it has a 
chance of succeeding (depending on executor memory, etc.).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144397)
Time Spent: 2h 40m  (was: 2.5h)

> Spark SourceRDD Not Designed With Dynamic Allocation In Mind
> 
>
> Key: BEAM-4783
> URL: https://issues.apache.org/jira/browse/BEAM-4783
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Affects Versions: 2.5.0
>Reporter: Kyle Winkelman
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>  Labels: newbie
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> When the spark-runner is used along with the configuration 
> spark.dynamicAllocation.enabled=true the SourceRDD does not detect this. It 
> then falls back to the value calculated in this description:
>   // when running on YARN/SparkDeploy it's the result of max(totalCores, 
> 2).
>   // when running on Mesos it's 8.
>   // when running local it's the total number of cores (local = 1, 
> local[N] = N,
>   // local[*] = estimation of the machine's cores).
>   // ** the configuration "spark.default.parallelism" takes precedence 
> over all of the above **
> So in most cases this default is quite small. This is an issue when using a 
> very large input file as it will only get split in half.
> I believe that when Dynamic Allocation is enable the SourceRDD should use the 
> DEFAULT_BUNDLE_SIZE and possibly expose a SparkPipelineOptions that allows 
> you to change this DEFAULT_BUNDLE_SIZE.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5389) [beam_PostCommit_Java_GradleBuild][:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest]

[Batkhuyag Batsaikhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Batkhuyag Batsaikhan updated BEAM-5389:
---
Description: 
[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1465/]

Relevant log:

11:10:52.145 [LIFECYCLE] 
[org.gradle.internal.operations.DefaultBuildOperationExecutor] > Task 
:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest
11:10:57.265 [DEBUG] [TestEventLogger] 
11:10:57.265 [DEBUG] [TestEventLogger] 
org.apache.beam.examples.complete.TfIdfIT > testE2ETfIdf FAILED
11:10:57.266 [DEBUG] [TestEventLogger] java.lang.AssertionError: 
11:10:57.266 [DEBUG] [TestEventLogger] Expected: Expected checksum is 
(f072786dde51dc09fc74bf38ffbfc27dcfdf0b96)
11:10:57.266 [DEBUG] [TestEventLogger] but: was 
(0549d1dc8821976121771aefcb0e2297177bdb88)
11:10:57.266 [DEBUG] [TestEventLogger] at 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:10:57.266 [DEBUG] [TestEventLogger] at 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
11:10:57.266 [DEBUG] [TestEventLogger] at 
org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:138)
11:10:57.266 [DEBUG] [TestEventLogger] at 
org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
11:10:57.266 [DEBUG] [TestEventLogger] at 
org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
11:10:57.266 [DEBUG] [TestEventLogger] at 
org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
11:10:57.266 [DEBUG] [TestEventLogger] at 
org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
11:10:57.267 [DEBUG] [TestEventLogger] at 
org.apache.beam.examples.complete.TfIdf.runTfIdf(TfIdf.java:427)
11:10:57.267 [DEBUG] [TestEventLogger] at 
org.apache.beam.examples.complete.TfIdfIT.testE2ETfIdf(TfIdfIT.java:72)
11:10:57.267 [DEBUG] [TestEventLogger] at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:10:57.267 [DEBUG] [TestEventLogger] at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
11:10:57.267 [DEBUG] [TestEventLogger] at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:10:57.267 [DEBUG] [TestEventLogger] at 
java.lang.reflect.Method.invoke(Method.java:498)
11:10:57.267 [DEBUG] [TestEventLogger] at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
11:10:57.267 [DEBUG] [TestEventLogger] at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
11:10:57.267 [DEBUG] [TestEventLogger] at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
11:10:57.267 [DEBUG] [TestEventLogger] at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
11:10:57.268 [DEBUG] [TestEventLogger] at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
11:10:57.268 [DEBUG] [TestEventLogger] at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
11:10:57.268 [DEBUG] [TestEventLogger] at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
11:10:57.268 [DEBUG] [TestEventLogger] at 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
11:10:57.268 [DEBUG] [TestEventLogger] at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
11:10:57.268 [DEBUG] [TestEventLogger] at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
11:10:57.268 [DEBUG] [TestEventLogger] at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
11:10:57.268 [DEBUG] [TestEventLogger] at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
11:10:57.268 [DEBUG] [TestEventLogger] at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
11:10:57.269 [DEBUG] [TestEventLogger] at 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
11:10:57.269 [DEBUG] [TestEventLogger] at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
11:10:57.269 [DEBUG] [TestEventLogger] at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
11:10:57.269 [DEBUG] [TestEventLogger] at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
11:10:57.269 [DEBUG] [TestEventLogger] at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
11:10:57.269 [DEBUG] [TestEventLogger] at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
11:10:57.269 [DEBUG] [TestEventLogger] at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:10:57.269 [DEBUG] [TestEventLogger] at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
11:10:57.269 [DEBUG] 

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=144392=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144392
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 14/Sep/18 18:25
Start Date: 14/Sep/18 18:25
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #6382: [BEAM-4449] Finish 
splitting Aggregate and Project
URL: https://github.com/apache/beam/pull/6382#issuecomment-421444776
 
 
   R: @akedin
   cc: @amaliujia 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144392)
Time Spent: 6h 50m  (was: 6h 40m)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5378) Ensure all Go SDK examples run successfully

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5378?focusedWorklogId=144387=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144387
 ]

ASF GitHub Bot logged work on BEAM-5378:


Author: ASF GitHub Bot
Created on: 14/Sep/18 18:21
Start Date: 14/Sep/18 18:21
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #6395: 
[BEAM-5378] Update go wordcap example to work on Dataflow runner
URL: https://github.com/apache/beam/pull/6395#discussion_r217801313
 
 

 ##
 File path: sdks/go/examples/wordcap/wordcap.go
 ##
 @@ -31,7 +30,7 @@ import (
 )
 
 var (
-   input = flag.String("input", 
os.ExpandEnv("$GOPATH/src/github.com/apache/beam/sdks/go/data/haiku/old_pond.txt"),
 "Files to read.")
+   input = flag.String("input", 
"gs://apache-beam-samples/shakespeare/kinglear.txt", "File(s) to read.")
 
 Review comment:
   Yes, I tried that it does. (This is similar to the wordcount example.) I 
believe this is a public file in GCS and does not require any auth to read it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 144387)
Time Spent: 40m  (was: 0.5h)

> Ensure all Go SDK examples run successfully
> ---
>
> Key: BEAM-5378
> URL: https://issues.apache.org/jira/browse/BEAM-5378
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Tomas Roos
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> I've been spending a day or so running through the example available for the 
> Go SDK in order to see what works and on what runner (direct, dataflow), and 
> what doesn't and here's the results.
> All available examples for the go sdk. For me as a new developer on apache 
> beam and dataflow it would be a tremendous value to have all examples running 
> because many of them have legitimate use-cases behind them. 
> {code:java}
> ├── complete
> │   └── autocomplete
> │   └── autocomplete.go
> ├── contains
> │   └── contains.go
> ├── cookbook
> │   ├── combine
> │   │   └── combine.go
> │   ├── filter
> │   │   └── filter.go
> │   ├── join
> │   │   └── join.go
> │   ├── max
> │   │   └── max.go
> │   └── tornadoes
> │   └── tornadoes.go
> ├── debugging_wordcount
> │   └── debugging_wordcount.go
> ├── forest
> │   └── forest.go
> ├── grades
> │   └── grades.go
> ├── minimal_wordcount
> │   └── minimal_wordcount.go
> ├── multiout
> │   └── multiout.go
> ├── pingpong
> │   └── pingpong.go
> ├── streaming_wordcap
> │   └── wordcap.go
> ├── windowed_wordcount
> │   └── windowed_wordcount.go
> ├── wordcap
> │   └── wordcap.go
> ├── wordcount
> │   └── wordcount.go
> └── yatzy
> └── yatzy.go
> {code}
> All examples that are supposed to be runnable by the direct driver (not 
> depending on gcp platform services) are runnable.
> On the otherhand these are the tests that needs to be updated because its not 
> runnable on the dataflow platform for various reasons.
> I tried to figure them out and all I can do is to pin point at least where it 
> fails since my knowledge so far in the beam / dataflow internals is limited.
> .
> ├── complete
> │   └── autocomplete
> │   └── autocomplete.go
> Runs successfully if swapping the input to one of the shakespear data files 
> from gs://
> But when running this it yields a error from the top.Largest func (discussed 
> in another issue that top.Largest needs to have a serializeable combinator / 
> accumulator)
> ➜  autocomplete git:(master) ✗ ./autocomplete --project fair-app-213019 
> --runner dataflow --staging_location=gs://fair-app-213019/staging-test2 
> --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
>  
> 2018/09/11 15:35:26 Running autocomplete
> Unable to encode combiner for lifting: failed to encode custom coder: bad 
> underlying type: bad field type: bad element: unencodable type: interface 
> {}2018/09/11 15:35:26 Using running binary as worker binary: './autocomplete'
> 2018/09/11 15:35:26 Staging worker binary: ./autocomplete
> ├── contains
> │   └── contains.go
> Fails when running debug.Head for some mysterious reason, might have to do 
> with the param passing into the x,y iterator. Frankly I dont know and could 
> not figure.
> But removing the debug.Head call everything works as expected and succeeds.
> ├── cookbook
> │   ├── combine
> │   │   └── combine.go
> Fails because of extractFn which is a struct is not