[jira] [Work logged] (BEAM-5355) Create GroupByKey load test for Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5355?focusedWorklogId=143377=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143377
 ]

ASF GitHub Bot logged work on BEAM-5355:


Author: ASF GitHub Bot
Created on: 12/Sep/18 05:49
Start Date: 12/Sep/18 05:49
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #6361: [BEAM-5355] 
GroupByKey Load IT
URL: https://github.com/apache/beam/pull/6361#issuecomment-420520337
 
 
   This looks good to me. I'd just add one or a couple distribution metrics to 
count time spent blocked, waiting to load the iterable with the values (e.g. 
time of first iteration, time of all iterations). WDYT?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143377)
Time Spent: 0.5h  (was: 20m)

> Create GroupByKey load test for Java SDK
> 
>
> Key: BEAM-5355
> URL: https://issues.apache.org/jira/browse/BEAM-5355
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: Lukasz Gajowy
>Assignee: Lukasz Gajowy
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> This is more thoroughly described in this proposal: 
> [https://docs.google.com/document/d/1PuIQv4v06eosKKwT76u7S6IP88AnXhTf870Rcj1AHt4/edit?usp=sharing]
>  
> In short: this ticket is about implementing the GroupByKeyLoadIT that uses 
> SyntheticStep and Synthetic source to create load on the pipeline. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #1014

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=143366=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143366
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 12/Sep/18 04:31
Start Date: 12/Sep/18 04:31
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #6316: 
[BEAM-4461] Add Unnest transform.
URL: https://github.com/apache/beam/pull/6316#discussion_r216893940
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
 ##
 @@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.Lists;
+import java.util.List;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+/**
+ * A {@link PTransform} to unnest nested rows.
+ *
+ * For example, consider a Row with the following nestedschema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location: LatLong
+ *
+ * LatLong Schema: latitude: DOUBLE longitude: DOUBLE
+ *
+ * After unnesting, all of the rows will be converted to rows satisfying 
the following schema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location.latitude: 
DOUBLE
+ * location.longitude: DOUBLE
+ *
+ * By default nested names are concatenated to generated the unnested name, 
however {@link
+ * Unnest.Inner#withFieldNameFunction} can be used to specify a custom naming 
policy.
+ *
+ * Note that currently array and map values are not unnested.
+ */
+@Experimental(Kind.SCHEMAS)
+public class Unnest {
+  public static  Inner create() {
+return new AutoValue_Unnest_Inner.Builder()
+.setMaxLevels(Integer.MAX_VALUE)
+.setFieldNameFunction(CONCAT_FIELD_NAMES)
+.build();
+  }
+  /**
+   * This is the default naming policy for naming fields. Every field name in 
the path to a given
+   * field is concated with . characters.
+   */
+  public static final SerializableFunction, String> 
CONCAT_FIELD_NAMES =
+  l -> {
+return String.join(".", l);
+  };
+  /**
+   * This policy keeps the raw nested field name. If two differently-nested 
fields have the same
+   * name, unnesting will fail with this policy.
+   */
+  public static final SerializableFunction, String> 
KEEP_NESTED_NAME =
+  l -> {
+return l.get(l.size() - 1);
+  };
+  /** Returns the result of unnesting the given schema. The default naming 
policy is used. */
+  static Schema getUnnestedSchema(Schema schema, int maxLevels) {
+List nameComponents = Lists.newArrayList();
+return getUnnestedSchema(schema, nameComponents, CONCAT_FIELD_NAMES, 
maxLevels, 0);
+  }
+  /** Returns the result of unnesting the given schema with the given naming 
policy. */
+  static Schema getUnnestedSchema(
+  Schema schema, int maxLevels, SerializableFunction, String> 
fn) {
+List nameComponents = Lists.newArrayList();
+return getUnnestedSchema(schema, nameComponents, fn, maxLevels, 0);
+  }
+
+  private static Schema getUnnestedSchema(
+  Schema schema,
+  List nameComponents,
+  SerializableFunction, String> fn,
+  int maxLevel,
+  int currentLevel) {
+Schema.Builder builder = Schema.builder();
+for (Field field : schema.getFields()) {
+  nameComponents.add(field.getName());
+  if (field.getType().getTypeName().isCompositeType() && currentLevel < 
maxLevel) {
 
 Review comment:
   Why? 


This is an automated message from the Apache Git Service.
To respond to the 

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=143365=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143365
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 12/Sep/18 04:30
Start Date: 12/Sep/18 04:30
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #6316: 
[BEAM-4461] Add Unnest transform.
URL: https://github.com/apache/beam/pull/6316#discussion_r216893853
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
 ##
 @@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.Lists;
+import java.util.List;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+/**
+ * A {@link PTransform} to unnest nested rows.
+ *
+ * For example, consider a Row with the following nestedschema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location: LatLong
+ *
+ * LatLong Schema: latitude: DOUBLE longitude: DOUBLE
+ *
+ * After unnesting, all of the rows will be converted to rows satisfying 
the following schema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location.latitude: 
DOUBLE
+ * location.longitude: DOUBLE
+ *
+ * By default nested names are concatenated to generated the unnested name, 
however {@link
+ * Unnest.Inner#withFieldNameFunction} can be used to specify a custom naming 
policy.
+ *
+ * Note that currently array and map values are not unnested.
+ */
+@Experimental(Kind.SCHEMAS)
+public class Unnest {
+  public static  Inner create() {
+return new AutoValue_Unnest_Inner.Builder()
+.setMaxLevels(Integer.MAX_VALUE)
+.setFieldNameFunction(CONCAT_FIELD_NAMES)
+.build();
+  }
+  /**
+   * This is the default naming policy for naming fields. Every field name in 
the path to a given
+   * field is concated with . characters.
+   */
+  public static final SerializableFunction, String> 
CONCAT_FIELD_NAMES =
+  l -> {
+return String.join(".", l);
+  };
+  /**
+   * This policy keeps the raw nested field name. If two differently-nested 
fields have the same
+   * name, unnesting will fail with this policy.
+   */
+  public static final SerializableFunction, String> 
KEEP_NESTED_NAME =
+  l -> {
+return l.get(l.size() - 1);
+  };
+  /** Returns the result of unnesting the given schema. The default naming 
policy is used. */
+  static Schema getUnnestedSchema(Schema schema, int maxLevels) {
+List nameComponents = Lists.newArrayList();
+return getUnnestedSchema(schema, nameComponents, CONCAT_FIELD_NAMES, 
maxLevels, 0);
+  }
+  /** Returns the result of unnesting the given schema with the given naming 
policy. */
+  static Schema getUnnestedSchema(
+  Schema schema, int maxLevels, SerializableFunction, String> 
fn) {
+List nameComponents = Lists.newArrayList();
+return getUnnestedSchema(schema, nameComponents, fn, maxLevels, 0);
+  }
+
+  private static Schema getUnnestedSchema(
+  Schema schema,
+  List nameComponents,
+  SerializableFunction, String> fn,
+  int maxLevel,
+  int currentLevel) {
+Schema.Builder builder = Schema.builder();
+for (Field field : schema.getFields()) {
+  nameComponents.add(field.getName());
+  if (field.getType().getTypeName().isCompositeType() && currentLevel < 
maxLevel) {
+Schema nestedSchema =
+getUnnestedSchema(
+field.getType().getRowSchema(), nameComponents, fn, maxLevel, 
currentLevel + 1);
+for 

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=143364=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143364
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 12/Sep/18 04:30
Start Date: 12/Sep/18 04:30
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #6316: 
[BEAM-4461] Add Unnest transform.
URL: https://github.com/apache/beam/pull/6316#discussion_r216893831
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
 ##
 @@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.Lists;
+import java.util.List;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+/**
+ * A {@link PTransform} to unnest nested rows.
+ *
+ * For example, consider a Row with the following nestedschema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location: LatLong
+ *
+ * LatLong Schema: latitude: DOUBLE longitude: DOUBLE
+ *
+ * After unnesting, all of the rows will be converted to rows satisfying 
the following schema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location.latitude: 
DOUBLE
+ * location.longitude: DOUBLE
+ *
+ * By default nested names are concatenated to generated the unnested name, 
however {@link
+ * Unnest.Inner#withFieldNameFunction} can be used to specify a custom naming 
policy.
+ *
+ * Note that currently array and map values are not unnested.
+ */
+@Experimental(Kind.SCHEMAS)
+public class Unnest {
+  public static  Inner create() {
+return new AutoValue_Unnest_Inner.Builder()
+.setMaxLevels(Integer.MAX_VALUE)
+.setFieldNameFunction(CONCAT_FIELD_NAMES)
+.build();
+  }
+  /**
+   * This is the default naming policy for naming fields. Every field name in 
the path to a given
+   * field is concated with . characters.
+   */
+  public static final SerializableFunction, String> 
CONCAT_FIELD_NAMES =
+  l -> {
+return String.join(".", l);
+  };
+  /**
+   * This policy keeps the raw nested field name. If two differently-nested 
fields have the same
+   * name, unnesting will fail with this policy.
+   */
+  public static final SerializableFunction, String> 
KEEP_NESTED_NAME =
 
 Review comment:
   Disagree here - I added this because I intend to use it fairly soon. Also I 
think a transform like this that munges names needs to be customizable - a 
hard-coded name strategy always runs into problems. Worth noting that this is 
basically the same as SQL (e.g. select X.Y AS NewName)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143364)
Time Spent: 7h 20m  (was: 7h 10m)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was 

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=143363=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143363
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 12/Sep/18 04:25
Start Date: 12/Sep/18 04:25
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #6316: 
[BEAM-4461] Add Unnest transform.
URL: https://github.com/apache/beam/pull/6316#discussion_r216893261
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
 ##
 @@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.Lists;
+import java.util.List;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+/**
+ * A {@link PTransform} to unnest nested rows.
+ *
+ * For example, consider a Row with the following nestedschema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location: LatLong
+ *
+ * LatLong Schema: latitude: DOUBLE longitude: DOUBLE
+ *
+ * After unnesting, all of the rows will be converted to rows satisfying 
the following schema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location.latitude: 
DOUBLE
+ * location.longitude: DOUBLE
+ *
+ * By default nested names are concatenated to generated the unnested name, 
however {@link
+ * Unnest.Inner#withFieldNameFunction} can be used to specify a custom naming 
policy.
+ *
+ * Note that currently array and map values are not unnested.
+ */
+@Experimental(Kind.SCHEMAS)
+public class Unnest {
+  public static  Inner create() {
+return new AutoValue_Unnest_Inner.Builder()
+.setMaxLevels(Integer.MAX_VALUE)
+.setFieldNameFunction(CONCAT_FIELD_NAMES)
+.build();
+  }
+  /**
+   * This is the default naming policy for naming fields. Every field name in 
the path to a given
+   * field is concated with . characters.
+   */
+  public static final SerializableFunction, String> 
CONCAT_FIELD_NAMES =
+  l -> {
+return String.join(".", l);
 
 Review comment:
   But isn't "." separated fields exactly what SQL select syntax wants?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143363)
Time Spent: 7h 10m  (was: 7h)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=143362=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143362
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 12/Sep/18 04:22
Start Date: 12/Sep/18 04:22
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #6317: 
[BEAM-4461]  Add mapping between FieldType and Java types.
URL: https://github.com/apache/beam/pull/6317#discussion_r216892966
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldTypeDescriptors.java
 ##
 @@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.BiMap;
+import com.google.common.collect.ImmutableBiMap;
+import java.lang.reflect.ParameterizedType;
+import java.util.Collection;
+import java.util.Map;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.joda.time.Instant;
+/**
+ * Utilities for converting between {@link Schema} field types and {@link 
TypeDescriptor}s that
+ * define Java objects which can represent these field types.
+ */
+public class FieldTypeDescriptors {
+  private static final BiMap PRIMITIVE_MAPPING =
+  ImmutableBiMap.builder()
+  .put(TypeName.BYTE, TypeDescriptors.bytes())
+  .put(TypeName.INT16, TypeDescriptors.shorts())
+  .put(TypeName.INT32, TypeDescriptors.integers())
+  .put(TypeName.INT64, TypeDescriptors.longs())
+  .put(TypeName.DECIMAL, TypeDescriptors.bigdecimals())
+  .put(TypeName.FLOAT, TypeDescriptors.floats())
+  .put(TypeName.DOUBLE, TypeDescriptors.doubles())
+  .put(TypeName.STRING, TypeDescriptors.strings())
+  .put(TypeName.DATETIME, TypeDescriptor.of(Instant.class))
+  .put(TypeName.BOOLEAN, TypeDescriptors.booleans())
+  .put(TypeName.BYTES, TypeDescriptor.of(byte[].class))
+  .build();
+  /** Get a {@link TypeDescriptor} from a {@link FieldType}. */
+  public static TypeDescriptor javaTypeForFieldType(FieldType fieldType) {
+switch (fieldType.getTypeName()) {
+  case ARRAY:
+return 
TypeDescriptors.lists(javaTypeForFieldType(fieldType.getCollectionElementType()));
+  case MAP:
+return TypeDescriptors.maps(
+javaTypeForFieldType(fieldType.getMapKeyType()),
+javaTypeForFieldType(fieldType.getMapValueType()));
+  case ROW:
+return TypeDescriptors.rows();
+  default:
+return PRIMITIVE_MAPPING.get(fieldType.getTypeName());
+}
+  }
+  /** Get a {@link FieldType} from a {@link TypeDescriptor}. */
+  public static FieldType fieldTypeForJavaType(TypeDescriptor typeDescriptor) {
+if (typeDescriptor.isArray()
+|| typeDescriptor.isSubtypeOf(TypeDescriptor.of(Collection.class))) {
+  return getArrayFieldType(typeDescriptor);
+} else if (typeDescriptor.isSubtypeOf(TypeDescriptor.of(Map.class))) {
+  return getMapFieldType(typeDescriptor);
+} else if (typeDescriptor.isSubtypeOf(TypeDescriptor.of(Row.class))) {
+  throw new IllegalArgumentException(
+  "Cannot automatically determine a field type from a Row class"
+  + " as we cannot determine the schema. You should set a field 
type explicitly.");
+} else {
+  TypeName typeName = PRIMITIVE_MAPPING.inverse().get(typeDescriptor);
+  if (typeName == null) {
+throw new RuntimeException("Couldn't find field type for " + 
typeDescriptor);
+  }
+  FieldType fieldType = FieldType.of(typeName);
+  return fieldType;
+}
+  }
+
+  private static FieldType getArrayFieldType(TypeDescriptor typeDescriptor) {
+if (typeDescriptor.isArray()) {
+  if (typeDescriptor.getComponentType().getType().equals(byte.class)) {
+

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=143361=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143361
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 12/Sep/18 04:20
Start Date: 12/Sep/18 04:20
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #6317: 
[BEAM-4461]  Add mapping between FieldType and Java types.
URL: https://github.com/apache/beam/pull/6317#discussion_r216892734
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldTypeDescriptors.java
 ##
 @@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.BiMap;
+import com.google.common.collect.ImmutableBiMap;
+import java.lang.reflect.ParameterizedType;
+import java.util.Collection;
+import java.util.Map;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.joda.time.Instant;
+/**
+ * Utilities for converting between {@link Schema} field types and {@link 
TypeDescriptor}s that
+ * define Java objects which can represent these field types.
+ */
+public class FieldTypeDescriptors {
+  private static final BiMap PRIMITIVE_MAPPING =
+  ImmutableBiMap.builder()
+  .put(TypeName.BYTE, TypeDescriptors.bytes())
+  .put(TypeName.INT16, TypeDescriptors.shorts())
+  .put(TypeName.INT32, TypeDescriptors.integers())
+  .put(TypeName.INT64, TypeDescriptors.longs())
+  .put(TypeName.DECIMAL, TypeDescriptors.bigdecimals())
+  .put(TypeName.FLOAT, TypeDescriptors.floats())
+  .put(TypeName.DOUBLE, TypeDescriptors.doubles())
+  .put(TypeName.STRING, TypeDescriptors.strings())
+  .put(TypeName.DATETIME, TypeDescriptor.of(Instant.class))
+  .put(TypeName.BOOLEAN, TypeDescriptors.booleans())
+  .put(TypeName.BYTES, TypeDescriptor.of(byte[].class))
+  .build();
+  /** Get a {@link TypeDescriptor} from a {@link FieldType}. */
+  public static TypeDescriptor javaTypeForFieldType(FieldType fieldType) {
+switch (fieldType.getTypeName()) {
+  case ARRAY:
+return 
TypeDescriptors.lists(javaTypeForFieldType(fieldType.getCollectionElementType()));
+  case MAP:
+return TypeDescriptors.maps(
+javaTypeForFieldType(fieldType.getMapKeyType()),
+javaTypeForFieldType(fieldType.getMapValueType()));
+  case ROW:
+return TypeDescriptors.rows();
+  default:
+return PRIMITIVE_MAPPING.get(fieldType.getTypeName());
+}
+  }
+  /** Get a {@link FieldType} from a {@link TypeDescriptor}. */
+  public static FieldType fieldTypeForJavaType(TypeDescriptor typeDescriptor) {
+if (typeDescriptor.isArray()
+|| typeDescriptor.isSubtypeOf(TypeDescriptor.of(Collection.class))) {
+  return getArrayFieldType(typeDescriptor);
+} else if (typeDescriptor.isSubtypeOf(TypeDescriptor.of(Map.class))) {
+  return getMapFieldType(typeDescriptor);
+} else if (typeDescriptor.isSubtypeOf(TypeDescriptor.of(Row.class))) {
+  throw new IllegalArgumentException(
+  "Cannot automatically determine a field type from a Row class"
+  + " as we cannot determine the schema. You should set a field 
type explicitly.");
+} else {
+  TypeName typeName = PRIMITIVE_MAPPING.inverse().get(typeDescriptor);
+  if (typeName == null) {
+throw new RuntimeException("Couldn't find field type for " + 
typeDescriptor);
+  }
+  FieldType fieldType = FieldType.of(typeName);
+  return fieldType;
+}
+  }
+
+  private static FieldType getArrayFieldType(TypeDescriptor typeDescriptor) {
+if (typeDescriptor.isArray()) {
+  if (typeDescriptor.getComponentType().getType().equals(byte.class)) {
+

Build failed in Jenkins: beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle #81

[Apache Jenkins Server]

See 


Changes:

[relax] Add convenience methods to create TopCombineFns. Type-specific methods

[relax] Allow Coders to be explicitly specified in composed CombineFns for the

[relax] Apply spotless.

[relax] Address comments.

[relax] Revert unexpected changes.

--
[...truncated 141.38 KB...]
Skip updating github.com/golang/mock#b3e60bcdc577185fce3cf625fc96b62857ce5574 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/mock/a5aba77edff874e2e3d9ad331994556c.
Skip updating 
github.com/golang/protobuf#3a3da3a4e26776cc22a79ef46d5d58477532dede in 
/home/jenkins/.gradle/go/repo/github.com/golang/protobuf/76cd502133aa2c5520c2b3e1debe630b.
Skip updating github.com/golang/snappy#553a641470496b2327abcac10b36396bd98e45c9 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/snappy/96c568bbd0eca3e3f6a41e15a438261a.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/google/go-cmp#3af367b6b30c263d47e8895973edcca9a49cf029 
in 
/home/jenkins/.gradle/go/repo/github.com/google/go-cmp/0e30ce638fec37a4c0a4040d380290b4.
Skip updating github.com/google/pprof#a8f279b7952b27edbcb72e5a6c69ee9be4c8ad93 
in 
/home/jenkins/.gradle/go/repo/github.com/google/pprof/580224cd7b76319263fe32ca011a0650.
Skip updating 
github.com/googleapis/gax-go#317e0006254c44a0ac427cc52a0e083ff0b9622f in 
/home/jenkins/.gradle/go/repo/github.com/googleapis/gax-go/cfbdb0eaeda3f1ea6b48e4ebedd6b7ac.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/hashicorp/hcl#23c074d0eceb2b8a5bfdbb271ab780cde70f05a8 
in 
/home/jenkins/.gradle/go/repo/github.com/hashicorp/hcl/37e0bb8b36d14da77f9f8c5babf91e48.
Skip updating 
github.com/ianlancetaylor/demangle#4883227f66371e02c4948937d3e2be1664d9be38 in 
/home/jenkins/.gradle/go/repo/github.com/ianlancetaylor/demangle/da9250878eadca7fe1a997f2c1e465b3.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/kr/fs#2788f0dbd16903de03cb8186e5c7d97b69ad387b in 
/home/jenkins/.gradle/go/repo/github.com/kr/fs/ef69c46d4366e9d326896066a4b027d4.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/magiconair/properties#49d762b9817ba1c2e9d0c69183c2b4a8b8f1d934 in 
/home/jenkins/.gradle/go/repo/github.com/magiconair/properties/34883f6c3f37cca1c4582d137f9fc020.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/mitchellh/go-homedir#b8bc1bf767474819792c23f32d8286a45736f1c6 in 
/home/jenkins/.gradle/go/repo/github.com/mitchellh/go-homedir/11e21d791561444efcbd8f83f3617926.
Skip updating 
github.com/mitchellh/mapstructure#a4e142e9c047c904fa2f1e144d9a84e6133024bc in 
/home/jenkins/.gradle/go/repo/github.com/mitchellh/mapstructure/8f497225aea52894fde79ad66bc0eb05.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/openzipkin/zipkin-go#3741243b287094fda649c7f0fa74bd51f37dc122 in 
/home/jenkins/.gradle/go/repo/github.com/openzipkin/zipkin-go/e53ea28b1632f7d2624f50816a45fa4e.
Skip updating 
github.com/pelletier/go-toml#acdc4509485b587f5e675510c4f2c63e90ff68a8 in 
/home/jenkins/.gradle/go/repo/github.com/pelletier/go-toml/fcb4bcb69fca9eab0a7b3cb341106167.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/pierrec/lz4#ed8d4cc3b461464e69798080a0092bd028910298 
in 

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=143360=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143360
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 12/Sep/18 04:19
Start Date: 12/Sep/18 04:19
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #6317: 
[BEAM-4461]  Add mapping between FieldType and Java types.
URL: https://github.com/apache/beam/pull/6317#discussion_r216892651
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldTypeDescriptors.java
 ##
 @@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.BiMap;
+import com.google.common.collect.ImmutableBiMap;
+import java.lang.reflect.ParameterizedType;
+import java.util.Collection;
+import java.util.Map;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.joda.time.Instant;
+/**
+ * Utilities for converting between {@link Schema} field types and {@link 
TypeDescriptor}s that
+ * define Java objects which can represent these field types.
+ */
+public class FieldTypeDescriptors {
+  private static final BiMap PRIMITIVE_MAPPING =
+  ImmutableBiMap.builder()
+  .put(TypeName.BYTE, TypeDescriptors.bytes())
+  .put(TypeName.INT16, TypeDescriptors.shorts())
+  .put(TypeName.INT32, TypeDescriptors.integers())
+  .put(TypeName.INT64, TypeDescriptors.longs())
+  .put(TypeName.DECIMAL, TypeDescriptors.bigdecimals())
+  .put(TypeName.FLOAT, TypeDescriptors.floats())
+  .put(TypeName.DOUBLE, TypeDescriptors.doubles())
+  .put(TypeName.STRING, TypeDescriptors.strings())
+  .put(TypeName.DATETIME, TypeDescriptor.of(Instant.class))
+  .put(TypeName.BOOLEAN, TypeDescriptors.booleans())
+  .put(TypeName.BYTES, TypeDescriptor.of(byte[].class))
+  .build();
+  /** Get a {@link TypeDescriptor} from a {@link FieldType}. */
+  public static TypeDescriptor javaTypeForFieldType(FieldType fieldType) {
+switch (fieldType.getTypeName()) {
+  case ARRAY:
+return 
TypeDescriptors.lists(javaTypeForFieldType(fieldType.getCollectionElementType()));
+  case MAP:
+return TypeDescriptors.maps(
+javaTypeForFieldType(fieldType.getMapKeyType()),
+javaTypeForFieldType(fieldType.getMapValueType()));
+  case ROW:
+return TypeDescriptors.rows();
+  default:
+return PRIMITIVE_MAPPING.get(fieldType.getTypeName());
+}
+  }
+  /** Get a {@link FieldType} from a {@link TypeDescriptor}. */
+  public static FieldType fieldTypeForJavaType(TypeDescriptor typeDescriptor) {
+if (typeDescriptor.isArray()
+|| typeDescriptor.isSubtypeOf(TypeDescriptor.of(Collection.class))) {
+  return getArrayFieldType(typeDescriptor);
+} else if (typeDescriptor.isSubtypeOf(TypeDescriptor.of(Map.class))) {
+  return getMapFieldType(typeDescriptor);
+} else if (typeDescriptor.isSubtypeOf(TypeDescriptor.of(Row.class))) {
+  throw new IllegalArgumentException(
+  "Cannot automatically determine a field type from a Row class"
+  + " as we cannot determine the schema. You should set a field 
type explicitly.");
 
 Review comment:
   Of course it's possible - that's essentially what proto buffers do. However 
it can be quite tricky to get right (especially when you start thinking about 
compatibility between different versions), so I'm loathe to go down that path 
just yet.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about 

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=143359=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143359
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 12/Sep/18 04:17
Start Date: 12/Sep/18 04:17
Worklog Time Spent: 10m 
  Work Description: reuvenlax closed pull request #6318: [BEAM-4461] Some 
fixes to Combiners needed for Schema support.
URL: https://github.com/apache/beam/pull/6318
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java
index c4f7813fbdb..b7e68d2a896 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java
@@ -20,6 +20,7 @@
 import static com.google.common.base.Preconditions.checkArgument;
 
 import com.google.common.base.Objects;
+import com.google.common.base.Optional;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.Lists;
@@ -32,6 +33,7 @@
 import java.util.List;
 import java.util.Map;
 import java.util.Map.Entry;
+import java.util.stream.Collectors;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.coders.CannotProvideCoderException;
 import org.apache.beam.sdk.coders.Coder;
@@ -112,6 +114,16 @@ public static ComposeCombineFnBuilder compose() {
   return new ComposedCombineFn().with(extractInputFn, combineFn, 
outputTag);
 }
 
+/** Like {@link #with(SimpleFunction, CombineFn, TupleTag)} but with an 
explicit input coder. */
+public  ComposedCombineFn with(
+SimpleFunction extractInputFn,
+Coder combineInputCoder,
+CombineFn combineFn,
+TupleTag outputTag) {
+  return new ComposedCombineFn()
+  .with(extractInputFn, combineInputCoder, combineFn, outputTag);
+}
+
 /**
  * Returns a {@link ComposedCombineFnWithContext} that can take additional 
{@link
  * GlobalCombineFn GlobalCombineFns} and apply them as a single combine 
function.
@@ -127,6 +139,16 @@ public static ComposeCombineFnBuilder compose() {
   return new ComposedCombineFnWithContext()
   .with(extractInputFn, combineFnWithContext, outputTag);
 }
+
+/** Like {@link #with(SimpleFunction, CombineFnWithContext, TupleTag)} but 
with input coder. */
+public  ComposedCombineFnWithContext with(
+SimpleFunction extractInputFn,
+Coder combineInputCoder,
+CombineFnWithContext combineFnWithContext,
+TupleTag outputTag) {
+  return new ComposedCombineFnWithContext()
+  .with(extractInputFn, combineInputCoder, combineFnWithContext, 
outputTag);
+}
   }
 
   /
@@ -212,12 +234,14 @@ public int hashCode() {
   public static class ComposedCombineFn extends CombineFn {
 
 private final List> combineFns;
+private final List> combineInputCoders;
 private final List> extractInputFns;
 private final List> outputTags;
 private final int combineFnCount;
 
 private ComposedCombineFn() {
   this.extractInputFns = ImmutableList.of();
+  this.combineInputCoders = ImmutableList.of();
   this.combineFns = ImmutableList.of();
   this.outputTags = ImmutableList.of();
   this.combineFnCount = 0;
@@ -225,11 +249,13 @@ private ComposedCombineFn() {
 
 private ComposedCombineFn(
 ImmutableList> extractInputFns,
+List> combineInputCoders,
 ImmutableList> combineFns,
 ImmutableList> outputTags) {
   @SuppressWarnings({"unchecked", "rawtypes"})
   List> castedExtractInputFns = (List) 
extractInputFns;
   this.extractInputFns = castedExtractInputFns;
+  this.combineInputCoders = combineInputCoders;
 
   @SuppressWarnings({"unchecked", "rawtypes"})
   List> castedCombineFns = (List) 
combineFns;
@@ -250,6 +276,10 @@ private ComposedCombineFn(
   .addAll(extractInputFns)
   .add(extractInputFn)
   .build(),
+  ImmutableList.>builder()
+  .addAll(combineInputCoders)
+  .add(Optional.absent())
+  .build(),
   ImmutableList.>builder().addAll(combineFns).add(combineFn).build(),
   
ImmutableList.>builder().addAll(outputTags).add(outputTag).build());
 }
@@ -272,6 +302,59 @@ private ComposedCombineFn(
   .addAll(extractInputFns)
   .add(extractInputFn)

[beam] branch master updated (8981a82 -> 3f0dcdc)

[reuvenlax]

This is an automated email from the ASF dual-hosted git repository.

reuvenlax pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 8981a82  Merge pull request #6370 from huygaa11/master
 add 345b0da  Add convenience methods to create TopCombineFns. 
Type-specific methods are added so that type inference can be done on the 
functions.
 add f09c249  Allow Coders to be explicitly specified in composed 
CombineFns for the case when coder inference does not work.
 add 2ea7d04  Apply spotless.
 add 2ed8c9a  Address comments.
 add eb18325  Revert unexpected changes.
 add 3f0dcdc  Merge pull request #6318: [BEAM-4461] Some fixes to Combiners 
needed for Schema support.

No new revisions were added by this update.

Summary of changes:
 .../org/apache/beam/sdk/transforms/CombineFns.java | 126 -
 .../java/org/apache/beam/sdk/transforms/Top.java   |  39 ++-
 2 files changed, 160 insertions(+), 5 deletions(-)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1444

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611444#comment-16611444
 ] 

Chamikara Jayalath commented on BEAM-5364:
--

Kevin, is this a regression from 2.6.0 if not this should probably not be a 
release blocker. Nevertheless agree that we should fix this soon.

>  BigtableIO source tries to validate table ID even though validation is 
> turned off
> --
>
> Key: BEAM-5364
> URL: https://issues.apache.org/jira/browse/BEAM-5364
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Blocker
>
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg]
> The validation can be turned off with following:
> BigtableIO.read()
>             .withoutValidation() // skip validation when constructing the 
> pipelline.
> A Dataflow template cannot be constructed due to this validation failure.
>  
> Error log when trying to construct a template:
> Exception in thread "main" java.lang.IllegalArgumentException: tableId was 
> not supplied
>         at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
>         at 
> com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5341) Migrate integration tests for TfIdf

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5341?focusedWorklogId=143331=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143331
 ]

ASF GitHub Bot logged work on BEAM-5341:


Author: ASF GitHub Bot
Created on: 12/Sep/18 01:26
Start Date: 12/Sep/18 01:26
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6371: [BEAM-5341] create 
IT test for TfIdf example
URL: https://github.com/apache/beam/pull/6371#issuecomment-420480308
 
 
   Run Java PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143331)
Time Spent: 50m  (was: 40m)

> Migrate integration tests for TfIdf
> ---
>
> Key: BEAM-5341
> URL: https://issues.apache.org/jira/browse/BEAM-5341
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PreCommit_Java_Cron #331

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #1426

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Interactive Beam -- comments for pipeline_graph_renderer.

[ptomasroos] Revert worker compatible binary check since its failing on some 
systems

[batbat] Fixed spotless java error in KinesisRecord.java causing precommit

[batbat] Removed beam/website check from the Beam build dependency.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 8981a826d6ad8a885a433bafe417fe0fea518e0f (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8981a826d6ad8a885a433bafe417fe0fea518e0f
Commit message: "Merge pull request #6370 from huygaa11/master"
 > git rev-list --no-walk 251dec50e7ab01838c93c8e91c94f25aca3b624c # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4762961368493831155.sh
+ rm -rf 

[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3267167419240571874.sh
+ rm -rf 

[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2932006499095803481.sh
+ virtualenv 

New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2618574591329053189.sh
+ 

 install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./env/.perfkit_env/lib/python2.7/site-packages (40.2.0)
Requirement already up-to-date: pip in 
./env/.perfkit_env/lib/python2.7/site-packages (18.0)
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3840180941104566070.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git 

Cloning into 
'
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1720275943093724636.sh
+ 

 install -r 

Collecting absl-py (from -r 

 (line 14))
Collecting jinja2>=2.7 (from -r 

 (line 15))
  Using cached 
https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
Requirement already satisfied: setuptools in 
./env/.perfkit_env/lib/python2.7/site-packages (from -r 

 (line 16)) (40.2.0)
Collecting colorlog[windows]==2.6.0 (from -r 

 (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r 

 (line 18))
Collecting futures>=3.0.3 (from -r 

[jira] [Work logged] (BEAM-5288) Modify Environment to support non-dockerized SDK harness deployments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5288?focusedWorklogId=143328=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143328
 ]

ASF GitHub Bot logged work on BEAM-5288:


Author: ASF GitHub Bot
Created on: 12/Sep/18 00:40
Start Date: 12/Sep/18 00:40
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6373: [BEAM-5288] Enhance 
Environment proto to use support different types of environments
URL: https://github.com/apache/beam/pull/6373#issuecomment-420472714
 
 
   cc: @mxm @tweise 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143328)
Time Spent: 40m  (was: 0.5h)

> Modify Environment to support non-dockerized SDK harness deployments 
> -
>
> Key: BEAM-5288
> URL: https://issues.apache.org/jira/browse/BEAM-5288
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> As of mailing discussions and BEAM-5187, it has become clear that we need to 
> extend the Environment information. In addition to the Docker environment, 
> the extended environment holds deployment options for 1) a process-based 
> environment, 2) an externally managed environment.
> The proto definition, as of now, looks as follows:
> {noformat}
>  message Environment {
>// (Required) The URN of the payload
>string urn = 1;
>// (Optional) The data specifying any parameters to the URN. If
>// the URN does not require any arguments, this may be omitted.
>bytes payload = 2;
>  }
>  message StandardEnvironments {
>enum Environments {
>  DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
>  PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
>  EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
>}
>  }
>  // The payload of a Docker image
>  message DockerPayload {
>string container_image = 1;  // implicitly linux_amd64.
>  }
>  message ProcessPayload {
>string os = 1;  // "linux", "darwin", ..
>string arch = 2;  // "amd64", ..
>string command = 3; // process to execute
>map env = 4; // environment variables
>  }
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5288) Modify Environment to support non-dockerized SDK harness deployments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5288?focusedWorklogId=143327=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143327
 ]

ASF GitHub Bot logged work on BEAM-5288:


Author: ASF GitHub Bot
Created on: 12/Sep/18 00:40
Start Date: 12/Sep/18 00:40
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6373: [BEAM-5288] Enhance 
Environment proto to use support different types of environments
URL: https://github.com/apache/beam/pull/6373#issuecomment-420472663
 
 
   Run Java PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143327)
Time Spent: 0.5h  (was: 20m)

> Modify Environment to support non-dockerized SDK harness deployments 
> -
>
> Key: BEAM-5288
> URL: https://issues.apache.org/jira/browse/BEAM-5288
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> As of mailing discussions and BEAM-5187, it has become clear that we need to 
> extend the Environment information. In addition to the Docker environment, 
> the extended environment holds deployment options for 1) a process-based 
> environment, 2) an externally managed environment.
> The proto definition, as of now, looks as follows:
> {noformat}
>  message Environment {
>// (Required) The URN of the payload
>string urn = 1;
>// (Optional) The data specifying any parameters to the URN. If
>// the URN does not require any arguments, this may be omitted.
>bytes payload = 2;
>  }
>  message StandardEnvironments {
>enum Environments {
>  DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
>  PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
>  EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
>}
>  }
>  // The payload of a Docker image
>  message DockerPayload {
>string container_image = 1;  // implicitly linux_amd64.
>  }
>  message ProcessPayload {
>string os = 1;  // "linux", "darwin", ..
>string arch = 2;  // "amd64", ..
>string command = 3; // process to execute
>map env = 4; // environment variables
>  }
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5288) Modify Environment to support non-dockerized SDK harness deployments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5288?focusedWorklogId=143326=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143326
 ]

ASF GitHub Bot logged work on BEAM-5288:


Author: ASF GitHub Bot
Created on: 12/Sep/18 00:37
Start Date: 12/Sep/18 00:37
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6373: [BEAM-5288] Enhance 
Environment proto to use support different types of environments
URL: https://github.com/apache/beam/pull/6373#issuecomment-420472097
 
 
   Run Java PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143326)
Time Spent: 20m  (was: 10m)

> Modify Environment to support non-dockerized SDK harness deployments 
> -
>
> Key: BEAM-5288
> URL: https://issues.apache.org/jira/browse/BEAM-5288
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> As of mailing discussions and BEAM-5187, it has become clear that we need to 
> extend the Environment information. In addition to the Docker environment, 
> the extended environment holds deployment options for 1) a process-based 
> environment, 2) an externally managed environment.
> The proto definition, as of now, looks as follows:
> {noformat}
>  message Environment {
>// (Required) The URN of the payload
>string urn = 1;
>// (Optional) The data specifying any parameters to the URN. If
>// the URN does not require any arguments, this may be omitted.
>bytes payload = 2;
>  }
>  message StandardEnvironments {
>enum Environments {
>  DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
>  PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
>  EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
>}
>  }
>  // The payload of a Docker image
>  message DockerPayload {
>string container_image = 1;  // implicitly linux_amd64.
>  }
>  message ProcessPayload {
>string os = 1;  // "linux", "darwin", ..
>string arch = 2;  // "amd64", ..
>string command = 3; // process to execute
>map env = 4; // environment variables
>  }
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5288) Modify Environment to support non-dockerized SDK harness deployments

[Ankur Goenka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611413#comment-16611413
 ] 

Ankur Goenka commented on BEAM-5288:


It will be good to expose args separately in Process Environment to make sure 
we escape them correctly when we issue the command.

 

message ProcessPayload {
 string os = 1; // "linux", "darwin", ..
 string arch = 2; // "amd64", ..
 string command = 3; // process to execute
 map env = 4; // environment variables
 }

 

==>>

 

message ProcessPayload {
 string os = 1; // "linux", "darwin", ..
 string arch = 2; // "amd64", ..
 string command = 3; // process to execute
repeated string args = 4; // Arguments to the command
map env = 5; // Environment variables


 }

> Modify Environment to support non-dockerized SDK harness deployments 
> -
>
> Key: BEAM-5288
> URL: https://issues.apache.org/jira/browse/BEAM-5288
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> As of mailing discussions and BEAM-5187, it has become clear that we need to 
> extend the Environment information. In addition to the Docker environment, 
> the extended environment holds deployment options for 1) a process-based 
> environment, 2) an externally managed environment.
> The proto definition, as of now, looks as follows:
> {noformat}
>  message Environment {
>// (Required) The URN of the payload
>string urn = 1;
>// (Optional) The data specifying any parameters to the URN. If
>// the URN does not require any arguments, this may be omitted.
>bytes payload = 2;
>  }
>  message StandardEnvironments {
>enum Environments {
>  DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
>  PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
>  EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
>}
>  }
>  // The payload of a Docker image
>  message DockerPayload {
>string container_image = 1;  // implicitly linux_amd64.
>  }
>  message ProcessPayload {
>string os = 1;  // "linux", "darwin", ..
>string arch = 2;  // "amd64", ..
>string command = 3; // process to execute
>map env = 4; // environment variables
>  }
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5288) Modify Environment to support non-dockerized SDK harness deployments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5288?focusedWorklogId=143323=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143323
 ]

ASF GitHub Bot logged work on BEAM-5288:


Author: ASF GitHub Bot
Created on: 12/Sep/18 00:29
Start Date: 12/Sep/18 00:29
Worklog Time Spent: 10m 
  Work Description: angoenka opened a new pull request #6373: [BEAM-5288] 
Enhance Environment proto to use support different types of environments
URL: https://github.com/apache/beam/pull/6373
 
 
   Reference discussion 
https://lists.apache.org/thread.html/d8b81e9f74f77d74c8b883cda80fa48efdcaf6ac2ad313c4fe68795a@%3Cdev.beam.apache.org%3E
 
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143323)
Time Spent: 10m
Remaining Estimate: 0h

> Modify Environment to support non-dockerized SDK harness deployments 
> -
>
> Key: BEAM-5288
> URL: https://issues.apache.org/jira/browse/BEAM-5288
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Maximilian Michels
>

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #1013

[Apache Jenkins Server]

See 


--
[...truncated 74.12 KB...]
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
test_as_dict_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ERROR
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock 

[jira] [Commented] (BEAM-5357) Go check for IsWorkerCompatibleBinary is wrong

[Henning Rohde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611406#comment-16611406
 ] 

Henning Rohde commented on BEAM-5357:
-

De-prioritizing now that [~ptomasroos]'s fix is in.

> Go check for IsWorkerCompatibleBinary is wrong
> --
>
> Key: BEAM-5357
> URL: https://issues.apache.org/jira/browse/BEAM-5357
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Per BEAM-5253, The linux/amd64 check in IsWorkerCompatibleBinary is 
> insufficient:
> https://github.com/apache/beam/blob/609a42978405173a60e5d91f35170a5c0b5d5332/sdks/go/pkg/beam/runners/universal/runnerlib/compile.go#L37
> We need to see if we can do a better check here (such as looking up the 
> symbol table or similar) or disable this optimization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5357) Go check for IsWorkerCompatibleBinary is wrong

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-5357:

Priority: Major  (was: Critical)

> Go check for IsWorkerCompatibleBinary is wrong
> --
>
> Key: BEAM-5357
> URL: https://issues.apache.org/jira/browse/BEAM-5357
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Per BEAM-5253, The linux/amd64 check in IsWorkerCompatibleBinary is 
> insufficient:
> https://github.com/apache/beam/blob/609a42978405173a60e5d91f35170a5c0b5d5332/sdks/go/pkg/beam/runners/universal/runnerlib/compile.go#L37
> We need to see if we can do a better check here (such as looking up the 
> symbol table or similar) or disable this optimization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5122) [beam_PostCommit_Java_GradleBuild][org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq][Flake] Suspect on pubsub initialization timeout.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?focusedWorklogId=143322=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143322
 ]

ASF GitHub Bot logged work on BEAM-5122:


Author: ASF GitHub Bot
Created on: 12/Sep/18 00:04
Start Date: 12/Sep/18 00:04
Worklog Time Spent: 10m 
  Work Description: Ardagan removed a comment on issue #6368: [BEAM-5122] 
Disable testUsesDlq due to flakiness
URL: https://github.com/apache/beam/pull/6368#issuecomment-420466842
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143322)
Time Spent: 7h  (was: 6h 50m)

> [beam_PostCommit_Java_GradleBuild][org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq][Flake]
>  Suspect on pubsub initialization timeout.
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5122) [beam_PostCommit_Java_GradleBuild][org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq][Flake] Suspect on pubsub initialization timeout.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?focusedWorklogId=143321=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143321
 ]

ASF GitHub Bot logged work on BEAM-5122:


Author: ASF GitHub Bot
Created on: 12/Sep/18 00:04
Start Date: 12/Sep/18 00:04
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #6368: [BEAM-5122] Disable 
testUsesDlq due to flakiness
URL: https://github.com/apache/beam/pull/6368#issuecomment-420466842
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143321)
Time Spent: 6h 50m  (was: 6h 40m)

> [beam_PostCommit_Java_GradleBuild][org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq][Flake]
>  Suspect on pubsub initialization timeout.
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle #80

[Apache Jenkins Server]

See 


--
[...truncated 140.86 KB...]
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/golang/glog#23def4e6c14b4da8ac2ed8007337bc5eb5007998 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/glog/c359ae0c52cbded42897d83346c85981.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/golang/mock#b3e60bcdc577185fce3cf625fc96b62857ce5574 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/mock/a5aba77edff874e2e3d9ad331994556c.
Skip updating 
github.com/golang/protobuf#3a3da3a4e26776cc22a79ef46d5d58477532dede in 
/home/jenkins/.gradle/go/repo/github.com/golang/protobuf/76cd502133aa2c5520c2b3e1debe630b.
Skip updating github.com/golang/snappy#553a641470496b2327abcac10b36396bd98e45c9 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/snappy/96c568bbd0eca3e3f6a41e15a438261a.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/google/go-cmp#3af367b6b30c263d47e8895973edcca9a49cf029 
in 
/home/jenkins/.gradle/go/repo/github.com/google/go-cmp/0e30ce638fec37a4c0a4040d380290b4.
Skip updating github.com/google/pprof#a8f279b7952b27edbcb72e5a6c69ee9be4c8ad93 
in 
/home/jenkins/.gradle/go/repo/github.com/google/pprof/580224cd7b76319263fe32ca011a0650.
Skip updating 
github.com/googleapis/gax-go#317e0006254c44a0ac427cc52a0e083ff0b9622f in 
/home/jenkins/.gradle/go/repo/github.com/googleapis/gax-go/cfbdb0eaeda3f1ea6b48e4ebedd6b7ac.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/hashicorp/hcl#23c074d0eceb2b8a5bfdbb271ab780cde70f05a8 
in 
/home/jenkins/.gradle/go/repo/github.com/hashicorp/hcl/37e0bb8b36d14da77f9f8c5babf91e48.
Skip updating 
github.com/ianlancetaylor/demangle#4883227f66371e02c4948937d3e2be1664d9be38 in 
/home/jenkins/.gradle/go/repo/github.com/ianlancetaylor/demangle/da9250878eadca7fe1a997f2c1e465b3.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/kr/fs#2788f0dbd16903de03cb8186e5c7d97b69ad387b in 
/home/jenkins/.gradle/go/repo/github.com/kr/fs/ef69c46d4366e9d326896066a4b027d4.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/magiconair/properties#49d762b9817ba1c2e9d0c69183c2b4a8b8f1d934 in 
/home/jenkins/.gradle/go/repo/github.com/magiconair/properties/34883f6c3f37cca1c4582d137f9fc020.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/mitchellh/go-homedir#b8bc1bf767474819792c23f32d8286a45736f1c6 in 
/home/jenkins/.gradle/go/repo/github.com/mitchellh/go-homedir/11e21d791561444efcbd8f83f3617926.
Skip updating 
github.com/mitchellh/mapstructure#a4e142e9c047c904fa2f1e144d9a84e6133024bc in 
/home/jenkins/.gradle/go/repo/github.com/mitchellh/mapstructure/8f497225aea52894fde79ad66bc0eb05.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/openzipkin/zipkin-go#3741243b287094fda649c7f0fa74bd51f37dc122 in 
/home/jenkins/.gradle/go/repo/github.com/openzipkin/zipkin-go/e53ea28b1632f7d2624f50816a45fa4e.
Skip updating 
github.com/pelletier/go-toml#acdc4509485b587f5e675510c4f2c63e90ff68a8 in 

[jira] [Work logged] (BEAM-5281) There are several deadlinks in beam-site, please removed.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5281?focusedWorklogId=143319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143319
 ]

ASF GitHub Bot logged work on BEAM-5281:


Author: ASF GitHub Bot
Created on: 12/Sep/18 00:01
Start Date: 12/Sep/18 00:01
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #6369: [BEAM-5281] 
disable beam-website:testWebsite during build
URL: https://github.com/apache/beam/pull/6369#issuecomment-420466169
 
 
   Already merged in https://github.com/apache/beam/pull/6370


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143319)
Time Spent: 0.5h  (was: 20m)

> There are several deadlinks in beam-site, please removed.
> -
>
> Key: BEAM-5281
> URL: https://issues.apache.org/jira/browse/BEAM-5281
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Boyuan Zhang
>Assignee: Melissa Pashniak
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Deadlinks in beam-site cause nightly build failed: 
> https://scans.gradle.com/s/nzwfwj6iqlgrg/console-log?task=:beam-website:testWebsite#L13



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5281) There are several deadlinks in beam-site, please removed.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5281?focusedWorklogId=143320=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143320
 ]

ASF GitHub Bot logged work on BEAM-5281:


Author: ASF GitHub Bot
Created on: 12/Sep/18 00:01
Start Date: 12/Sep/18 00:01
Worklog Time Spent: 10m 
  Work Description: alanmyrvold closed pull request #6369: [BEAM-5281] 
disable beam-website:testWebsite during build
URL: https://github.com/apache/beam/pull/6369
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/website/build.gradle b/website/build.gradle
index 0a64e73ed50..211350197d1 100644
--- a/website/build.gradle
+++ b/website/build.gradle
@@ -94,7 +94,8 @@ task testWebsite(type: Exec) {
   bundle exec rake test"""
 }
 
-check.dependsOn testWebsite
+// Re-enable when [BEAM-5281] is fixed.
+// check.dependsOn testWebsite
 
 task preCommit {
   dependsOn testWebsite


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143320)
Time Spent: 40m  (was: 0.5h)

> There are several deadlinks in beam-site, please removed.
> -
>
> Key: BEAM-5281
> URL: https://issues.apache.org/jira/browse/BEAM-5281
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Boyuan Zhang
>Assignee: Melissa Pashniak
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Deadlinks in beam-site cause nightly build failed: 
> https://scans.gradle.com/s/nzwfwj6iqlgrg/console-log?task=:beam-website:testWebsite#L13



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Website_Cron #48

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Interactive Beam -- enable setting render option

[qinyeli] Interactive Beam -- comments for pipeline_graph_renderer.

[ptomasroos] Revert worker compatible binary check since its failing on some 
systems

[batbat] Fixed spotless java error in KinesisRecord.java causing precommit

[batbat] Removed beam/website check from the Beam build dependency.

--
[...truncated 8.79 KB...]
> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:spotlessGroovy (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
1.757 secs.
:spotlessGroovyCheck (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:spotlessGroovyCheck (Thread[Task worker for ':buildSrc',5,main]) completed. 
Took 0.0 secs.
:spotlessGroovyGradle (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:spotlessGroovyGradle (Thread[Task worker for ':buildSrc',5,main]) completed. 
Took 0.032 secs.
:spotlessGroovyGradleCheck (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:spotlessGroovyGradleCheck (Thread[Daemon worker,5,main]) completed. Took 0.0 
secs.
:spotlessCheck (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:spotlessCheck (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
:compileTestJava (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Daemon worker,5,main]) completed. Took 0.002 secs.
:compileTestGroovy (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Daemon worker,5,main]) completed. Took 0.002 secs.
:processTestResources (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Daemon worker,5,main]) completed. Took 0.002 secs.
:testClasses (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
:test (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Daemon worker,5,main]) completed. Took 0.007 secs.
:check (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:check
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
:build (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:build
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
Invalidating in-memory cache of 
/home/jenkins/.gradle/caches/4.8/fileHashes/fileHashes.bin
Invalidating in-memory cache of 
/home/jenkins/.gradle/caches/4.8/fileHashes/resourceHashesCache.bin
Settings evaluated using 

[jira] [Work logged] (BEAM-5341) Migrate integration tests for TfIdf

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5341?focusedWorklogId=143312=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143312
 ]

ASF GitHub Bot logged work on BEAM-5341:


Author: ASF GitHub Bot
Created on: 11/Sep/18 23:46
Start Date: 11/Sep/18 23:46
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6371: [BEAM-5341] 
create IT test for TfIdf example
URL: https://github.com/apache/beam/pull/6371#issuecomment-420460993
 
 
   Run Java PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143312)
Time Spent: 40m  (was: 0.5h)

> Migrate integration tests for TfIdf
> ---
>
> Key: BEAM-5341
> URL: https://issues.apache.org/jira/browse/BEAM-5341
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5341) Migrate integration tests for TfIdf

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5341?focusedWorklogId=143311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143311
 ]

ASF GitHub Bot logged work on BEAM-5341:


Author: ASF GitHub Bot
Created on: 11/Sep/18 23:46
Start Date: 11/Sep/18 23:46
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6371: [BEAM-5341] create 
IT test for TfIdf example
URL: https://github.com/apache/beam/pull/6371#issuecomment-420463594
 
 
   Run Java PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143311)
Time Spent: 0.5h  (was: 20m)

> Migrate integration tests for TfIdf
> ---
>
> Key: BEAM-5341
> URL: https://issues.apache.org/jira/browse/BEAM-5341
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5369) Portable wordcount java broken because of create_view usage

[Ankur Goenka (JIRA)]

Ankur Goenka created BEAM-5369:
--

 Summary: Portable wordcount java broken because of create_view 
usage
 Key: BEAM-5369
 URL: https://issues.apache.org/jira/browse/BEAM-5369
 Project: Beam
  Issue Type: Test
  Components: sdk-java-core
Reporter: Ankur Goenka
Assignee: Ankur Goenka


Portable Wordcount is broken on Flink since 
[https://github.com/apache/beam/pull/6208/files#diff-14d60e038b469ee6ce66ec6ec9d7d976L153]

SDK should stop using create_view URN as its depricated 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5341) Migrate integration tests for TfIdf

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5341?focusedWorklogId=143306=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143306
 ]

ASF GitHub Bot logged work on BEAM-5341:


Author: ASF GitHub Bot
Created on: 11/Sep/18 23:33
Start Date: 11/Sep/18 23:33
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6371: [BEAM-5341] create 
IT test for TfIdf example
URL: https://github.com/apache/beam/pull/6371#issuecomment-420460993
 
 
   Run Java PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143306)
Time Spent: 20m  (was: 10m)

> Migrate integration tests for TfIdf
> ---
>
> Key: BEAM-5341
> URL: https://issues.apache.org/jira/browse/BEAM-5341
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5341) Migrate integration tests for TfIdf

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5341?focusedWorklogId=143305=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143305
 ]

ASF GitHub Bot logged work on BEAM-5341:


Author: ASF GitHub Bot
Created on: 11/Sep/18 23:32
Start Date: 11/Sep/18 23:32
Worklog Time Spent: 10m 
  Work Description: yifanzou opened a new pull request #6371: [BEAM-5341] 
create IT test for TfIdf example
URL: https://github.com/apache/beam/pull/6371
 
 
   **Please** add a meaningful description for your change here
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143305)
Time Spent: 10m
Remaining Estimate: 0h

> Migrate integration tests for TfIdf
> ---
>
> Key: BEAM-5341
> URL: https://issues.apache.org/jira/browse/BEAM-5341
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle #79

[Apache Jenkins Server]

See 


Changes:

[batbat] Removed beam/website check from the Beam build dependency.

--
[...truncated 140.63 KB...]
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/golang/glog#23def4e6c14b4da8ac2ed8007337bc5eb5007998 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/glog/c359ae0c52cbded42897d83346c85981.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/golang/mock#b3e60bcdc577185fce3cf625fc96b62857ce5574 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/mock/a5aba77edff874e2e3d9ad331994556c.
Skip updating 
github.com/golang/protobuf#3a3da3a4e26776cc22a79ef46d5d58477532dede in 
/home/jenkins/.gradle/go/repo/github.com/golang/protobuf/76cd502133aa2c5520c2b3e1debe630b.
Skip updating github.com/golang/snappy#553a641470496b2327abcac10b36396bd98e45c9 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/snappy/96c568bbd0eca3e3f6a41e15a438261a.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/google/go-cmp#3af367b6b30c263d47e8895973edcca9a49cf029 
in 
/home/jenkins/.gradle/go/repo/github.com/google/go-cmp/0e30ce638fec37a4c0a4040d380290b4.
Skip updating github.com/google/pprof#a8f279b7952b27edbcb72e5a6c69ee9be4c8ad93 
in 
/home/jenkins/.gradle/go/repo/github.com/google/pprof/580224cd7b76319263fe32ca011a0650.
Skip updating 
github.com/googleapis/gax-go#317e0006254c44a0ac427cc52a0e083ff0b9622f in 
/home/jenkins/.gradle/go/repo/github.com/googleapis/gax-go/cfbdb0eaeda3f1ea6b48e4ebedd6b7ac.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/hashicorp/hcl#23c074d0eceb2b8a5bfdbb271ab780cde70f05a8 
in 
/home/jenkins/.gradle/go/repo/github.com/hashicorp/hcl/37e0bb8b36d14da77f9f8c5babf91e48.
Skip updating 
github.com/ianlancetaylor/demangle#4883227f66371e02c4948937d3e2be1664d9be38 in 
/home/jenkins/.gradle/go/repo/github.com/ianlancetaylor/demangle/da9250878eadca7fe1a997f2c1e465b3.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/kr/fs#2788f0dbd16903de03cb8186e5c7d97b69ad387b in 
/home/jenkins/.gradle/go/repo/github.com/kr/fs/ef69c46d4366e9d326896066a4b027d4.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/magiconair/properties#49d762b9817ba1c2e9d0c69183c2b4a8b8f1d934 in 
/home/jenkins/.gradle/go/repo/github.com/magiconair/properties/34883f6c3f37cca1c4582d137f9fc020.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/mitchellh/go-homedir#b8bc1bf767474819792c23f32d8286a45736f1c6 in 
/home/jenkins/.gradle/go/repo/github.com/mitchellh/go-homedir/11e21d791561444efcbd8f83f3617926.
Skip updating 
github.com/mitchellh/mapstructure#a4e142e9c047c904fa2f1e144d9a84e6133024bc in 
/home/jenkins/.gradle/go/repo/github.com/mitchellh/mapstructure/8f497225aea52894fde79ad66bc0eb05.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/openzipkin/zipkin-go#3741243b287094fda649c7f0fa74bd51f37dc122 in 
/home/jenkins/.gradle/go/repo/github.com/openzipkin/zipkin-go/e53ea28b1632f7d2624f50816a45fa4e.
Skip 

[jira] [Work logged] (BEAM-5367) [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website test failure

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5367?focusedWorklogId=143297=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143297
 ]

ASF GitHub Bot logged work on BEAM-5367:


Author: ASF GitHub Bot
Created on: 11/Sep/18 22:53
Start Date: 11/Sep/18 22:53
Worklog Time Spent: 10m 
  Work Description: pabloem closed pull request #6370: [BEAM-5367] Removed 
beam/website check from the Beam build dependency.
URL: https://github.com/apache/beam/pull/6370
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/website/build.gradle b/website/build.gradle
index 0a64e73ed50..7f5892600c2 100644
--- a/website/build.gradle
+++ b/website/build.gradle
@@ -94,7 +94,12 @@ task testWebsite(type: Exec) {
   bundle exec rake test"""
 }
 
-check.dependsOn testWebsite
+/**
+ * Removed testWebsite from the Beam build dependency because it is broken and 
obsolete.
+ * See https://issues.apache.org/jira/browse/BEAM-5367 for more info.
+ */
+
+// check.dependsOn testWebsite
 
 task preCommit {
   dependsOn testWebsite


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143297)
Time Spent: 1h  (was: 50m)

> [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website 
> test failure
> ---
>
> Key: BEAM-5367
> URL: https://issues.apache.org/jira/browse/BEAM-5367
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 2.7.0, 2.8.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> apache/website test is breaking nightly build. Since we don't have stable 
> apache/beam website, we should remove it from the build for now. Until we 
> have an actual plan to fully migrate from asf beam-site, we should disable 
> apache/website checks from Beam build.
> Failing job url: [https://scans.gradle.com/s/uxb6mdigqj4n4/console-log#L23465]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5367) [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website test failure

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5367?focusedWorklogId=143296=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143296
 ]

ASF GitHub Bot logged work on BEAM-5367:


Author: ASF GitHub Bot
Created on: 11/Sep/18 22:53
Start Date: 11/Sep/18 22:53
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #6370: [BEAM-5367] Removed 
beam/website check from the Beam build dependency.
URL: https://github.com/apache/beam/pull/6370#issuecomment-420453100
 
 
   Okay, this LGTM. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143296)
Time Spent: 50m  (was: 40m)

> [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website 
> test failure
> ---
>
> Key: BEAM-5367
> URL: https://issues.apache.org/jira/browse/BEAM-5367
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 2.7.0, 2.8.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> apache/website test is breaking nightly build. Since we don't have stable 
> apache/beam website, we should remove it from the build for now. Until we 
> have an actual plan to fully migrate from asf beam-site, we should disable 
> apache/website checks from Beam build.
> Failing job url: [https://scans.gradle.com/s/uxb6mdigqj4n4/console-log#L23465]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6370 from huygaa11/master

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8981a826d6ad8a885a433bafe417fe0fea518e0f
Merge: a3f6f7e a40a099
Author: Pablo 
AuthorDate: Tue Sep 11 15:53:38 2018 -0700

Merge pull request #6370 from huygaa11/master

[BEAM-5367] Removed beam/website check from the Beam build dependency.

 website/build.gradle | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

[beam] branch master updated (a3f6f7e -> 8981a82)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a3f6f7e  Merge pull request #6364: Fixed spotless java error in 
KinesisRecord.java
 add a40a099  Removed beam/website check from the Beam build dependency.
 new 8981a82  Merge pull request #6370 from huygaa11/master

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 website/build.gradle | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

[jira] [Work logged] (BEAM-5367) [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website test failure

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5367?focusedWorklogId=143293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143293
 ]

ASF GitHub Bot logged work on BEAM-5367:


Author: ASF GitHub Bot
Created on: 11/Sep/18 22:47
Start Date: 11/Sep/18 22:47
Worklog Time Spent: 10m 
  Work Description: huygaa11 commented on issue #6370: [BEAM-5367] Removed 
beam/website check from the Beam build dependency.
URL: https://github.com/apache/beam/pull/6370#issuecomment-420451822
 
 
   r: @pabloem 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143293)
Time Spent: 40m  (was: 0.5h)

> [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website 
> test failure
> ---
>
> Key: BEAM-5367
> URL: https://issues.apache.org/jira/browse/BEAM-5367
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 2.7.0, 2.8.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> apache/website test is breaking nightly build. Since we don't have stable 
> apache/beam website, we should remove it from the build for now. Until we 
> have an actual plan to fully migrate from asf beam-site, we should disable 
> apache/website checks from Beam build.
> Failing job url: [https://scans.gradle.com/s/uxb6mdigqj4n4/console-log#L23465]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5367) [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website test failure

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5367?focusedWorklogId=143292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143292
 ]

ASF GitHub Bot logged work on BEAM-5367:


Author: ASF GitHub Bot
Created on: 11/Sep/18 22:44
Start Date: 11/Sep/18 22:44
Worklog Time Spent: 10m 
  Work Description: huygaa11 commented on issue #6370: [BEAM-5367] Removed 
beam/website check from the Beam build dependency.
URL: https://github.com/apache/beam/pull/6370#issuecomment-420451099
 
 
   r: @Ardagan 
   cc: @lukecwik @swegner 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143292)
Time Spent: 0.5h  (was: 20m)

> [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website 
> test failure
> ---
>
> Key: BEAM-5367
> URL: https://issues.apache.org/jira/browse/BEAM-5367
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 2.7.0, 2.8.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> apache/website test is breaking nightly build. Since we don't have stable 
> apache/beam website, we should remove it from the build for now. Until we 
> have an actual plan to fully migrate from asf beam-site, we should disable 
> apache/website checks from Beam build.
> Failing job url: [https://scans.gradle.com/s/uxb6mdigqj4n4/console-log#L23465]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5367) [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website test failure

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5367?focusedWorklogId=143291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143291
 ]

ASF GitHub Bot logged work on BEAM-5367:


Author: ASF GitHub Bot
Created on: 11/Sep/18 22:42
Start Date: 11/Sep/18 22:42
Worklog Time Spent: 10m 
  Work Description: huygaa11 commented on issue #6370: [BEAM-5367] Removed 
beam/website check from the Beam build dependency.
URL: https://github.com/apache/beam/pull/6370#issuecomment-420450750
 
 
   Failure JIRA: https://issues.apache.org/jira/browse/BEAM-5367
   
   Added a JIRA to enable it once stable: 
https://issues.apache.org/jira/browse/BEAM-5368


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143291)
Time Spent: 20m  (was: 10m)

> [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website 
> test failure
> ---
>
> Key: BEAM-5367
> URL: https://issues.apache.org/jira/browse/BEAM-5367
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 2.7.0, 2.8.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> apache/website test is breaking nightly build. Since we don't have stable 
> apache/beam website, we should remove it from the build for now. Until we 
> have an actual plan to fully migrate from asf beam-site, we should disable 
> apache/website checks from Beam build.
> Failing job url: [https://scans.gradle.com/s/uxb6mdigqj4n4/console-log#L23465]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5368) Re-enable apache/website build once it is stable

[Batkhuyag Batsaikhan (JIRA)]

Batkhuyag Batsaikhan created BEAM-5368:
--

 Summary: Re-enable apache/website build once it is stable
 Key: BEAM-5368
 URL: https://issues.apache.org/jira/browse/BEAM-5368
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Batkhuyag Batsaikhan
Assignee: Scott Wegner


We are disabling beam check in 
[https://github.com/apache/beam/pull/6370|https://github.com/apache/beam/pull/6370.]
 due to https://issues.apache.org/jira/browse/BEAM-5367.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5337) [beam_PostCommit_Java_GradleBuild][:beam-runners-flink_2.11:test][Flake] Build times out in beam-runners-flink target

[Mikhail Gryzykhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611342#comment-16611342
 ] 

Mikhail Gryzykhin commented on BEAM-5337:
-

Synced with Ankur. He pointed that Aljoscha is a better candidate to look at 
this.

> [beam_PostCommit_Java_GradleBuild][:beam-runners-flink_2.11:test][Flake] 
> Build times out in beam-runners-flink target
> -
>
> Key: BEAM-5337
> URL: https://issues.apache.org/jira/browse/BEAM-5337
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Ankur Goenka
>Priority: Major
>
> Job times out. 
>  Failing job url:
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1414/consoleFull]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1406/consoleFull]
> https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1408/consoleFull
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5337) [beam_PostCommit_Java_GradleBuild][:beam-runners-flink_2.11:test][Flake] Build times out in beam-runners-flink target

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin reassigned BEAM-5337:
---

Assignee: Aljoscha Krettek  (was: Ankur Goenka)

> [beam_PostCommit_Java_GradleBuild][:beam-runners-flink_2.11:test][Flake] 
> Build times out in beam-runners-flink target
> -
>
> Key: BEAM-5337
> URL: https://issues.apache.org/jira/browse/BEAM-5337
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Aljoscha Krettek
>Priority: Major
>
> Job times out. 
>  Failing job url:
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1414/consoleFull]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1406/consoleFull]
> https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1408/consoleFull
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5122) [beam_PostCommit_Java_GradleBuild][org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq][Flake] Suspect on pubsub initialization timeout.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?focusedWorklogId=143290=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143290
 ]

ASF GitHub Bot logged work on BEAM-5122:


Author: ASF GitHub Bot
Created on: 11/Sep/18 22:37
Start Date: 11/Sep/18 22:37
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #6368: [BEAM-5122] Disable 
testUsesDlq due to flakiness
URL: https://github.com/apache/beam/pull/6368#issuecomment-420449659
 
 
   CC: @huygaa11 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143290)
Time Spent: 6h 40m  (was: 6.5h)

> [beam_PostCommit_Java_GradleBuild][org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq][Flake]
>  Suspect on pubsub initialization timeout.
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5367) [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website test failure

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5367?focusedWorklogId=143289=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143289
 ]

ASF GitHub Bot logged work on BEAM-5367:


Author: ASF GitHub Bot
Created on: 11/Sep/18 22:36
Start Date: 11/Sep/18 22:36
Worklog Time Spent: 10m 
  Work Description: huygaa11 opened a new pull request #6370: [BEAM-5367] 
Removed beam/website check from the Beam build dependency.
URL: https://github.com/apache/beam/pull/6370
 
 
   beam/website is currently broken and not actively maintained. However, it's 
test is included in Beam build and breaking nightly build. Removing it from 
Beam build until it is fixed.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143289)
Time Spent: 10m
Remaining Estimate: 0h

> [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website 
> test failure
> ---
>
> Key: BEAM-5367
> URL: https://issues.apache.org/jira/browse/BEAM-5367
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 2.7.0, 2.8.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> apache/website test is breaking nightly build. Since we don't have stable 
> apache/beam website, we should remove it from the build for now. Until we 
> have an actual plan to fully migrate from asf beam-site, we should disable 
> apache/website checks from Beam build.
> Failing job url: [https://scans.gradle.com/s/uxb6mdigqj4n4/console-log#L23465]



--
This message was sent by Atlassian JIRA

[jira] [Work logged] (BEAM-5281) There are several deadlinks in beam-site, please removed.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5281?focusedWorklogId=143288=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143288
 ]

ASF GitHub Bot logged work on BEAM-5281:


Author: ASF GitHub Bot
Created on: 11/Sep/18 22:34
Start Date: 11/Sep/18 22:34
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #6369: [BEAM-5281] 
disable beam-website:testWebsite during build
URL: https://github.com/apache/beam/pull/6369#issuecomment-420448920
 
 
   Run Gradle Publish


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143288)
Time Spent: 20m  (was: 10m)

> There are several deadlinks in beam-site, please removed.
> -
>
> Key: BEAM-5281
> URL: https://issues.apache.org/jira/browse/BEAM-5281
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Boyuan Zhang
>Assignee: Melissa Pashniak
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Deadlinks in beam-site cause nightly build failed: 
> https://scans.gradle.com/s/nzwfwj6iqlgrg/console-log?task=:beam-website:testWebsite#L13



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5281) There are several deadlinks in beam-site, please removed.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5281?focusedWorklogId=143287=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143287
 ]

ASF GitHub Bot logged work on BEAM-5281:


Author: ASF GitHub Bot
Created on: 11/Sep/18 22:33
Start Date: 11/Sep/18 22:33
Worklog Time Spent: 10m 
  Work Description: alanmyrvold opened a new pull request #6369: 
[BEAM-5281] disable beam-website:testWebsite during build
URL: https://github.com/apache/beam/pull/6369
 
 
   https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/ is 
failing due to beam-website:testWebsite, which is testing in a repository 
(beam) and branch (master), not used by the beam site and can be ignore.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143287)
Time Spent: 10m
Remaining Estimate: 0h

> There are several deadlinks in beam-site, please removed.
> -
>
> Key: BEAM-5281
> URL: https://issues.apache.org/jira/browse/BEAM-5281
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Boyuan Zhang
>

[jira] [Updated] (BEAM-5367) [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website test failure

[Batkhuyag Batsaikhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Batkhuyag Batsaikhan updated BEAM-5367:
---
Description: 
apache/website test is breaking nightly build. Since we don't have stable 
apache/beam website, we should remove it from the build for now. Until we have 
an actual plan to fully migrate from asf beam-site, we should disable 
apache/website checks from Beam build.

Failing job url: [https://scans.gradle.com/s/uxb6mdigqj4n4/console-log#L23465]

  was:
apache/beam website test is breaking nightly build. Since we don't have stable 
apache/beam website, we should remove it from the build for now.


 Failing job url: https://scans.gradle.com/s/uxb6mdigqj4n4/console-log#L23465


> [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website 
> test failure
> ---
>
> Key: BEAM-5367
> URL: https://issues.apache.org/jira/browse/BEAM-5367
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 2.7.0, 2.8.0
>
>
> apache/website test is breaking nightly build. Since we don't have stable 
> apache/beam website, we should remove it from the build for now. Until we 
> have an actual plan to fully migrate from asf beam-site, we should disable 
> apache/website checks from Beam build.
> Failing job url: [https://scans.gradle.com/s/uxb6mdigqj4n4/console-log#L23465]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5367) [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website test failure

[Batkhuyag Batsaikhan (JIRA)]

Batkhuyag Batsaikhan created BEAM-5367:
--

 Summary: [beam_Release_Gradle_NightlySnapshot] is broken due to 
apache/beam website test failure
 Key: BEAM-5367
 URL: https://issues.apache.org/jira/browse/BEAM-5367
 Project: Beam
  Issue Type: Bug
  Components: test-failures
Reporter: Batkhuyag Batsaikhan
Assignee: Batkhuyag Batsaikhan
 Fix For: 2.7.0, 2.8.0


apache/beam website test is breaking nightly build. Since we don't have stable 
apache/beam website, we should remove it from the build for now.


 Failing job url: https://scans.gradle.com/s/uxb6mdigqj4n4/console-log#L23465



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5122) [beam_PostCommit_Java_GradleBuild][org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq][Flake] Suspect on pubsub initialization timeout.

[Mikhail Gryzykhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611327#comment-16611327
 ] 

Mikhail Gryzykhin commented on BEAM-5122:
-

Synced with Anton. He suggested to disable the test for now while he is working 
on proper fix.

> [beam_PostCommit_Java_GradleBuild][org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq][Flake]
>  Suspect on pubsub initialization timeout.
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5366) Vendor gRPC and Protobuf separately from beam-model-* Java packages

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5366:
---

 Summary: Vendor gRPC and Protobuf separately from beam-model-* 
Java packages
 Key: BEAM-5366
 URL: https://issues.apache.org/jira/browse/BEAM-5366
 Project: Beam
  Issue Type: Improvement
  Components: beam-model, build-system
Reporter: Luke Cwik
Assignee: Kenneth Knowles


Each of the beam-model-* jars currently contains duplicate definitions of gRPC 
and protobuf and their transitive dependencies.

 

By migrating those packages to a separate artifact, we could have a single copy 
of those classes instead of 3.

 

This would reduce the size of the jars and prevent warnings such as:
 
{code:java}
[WARNING] beam-model-fn-execution-2.5.0.jar, 
beam-model-job-management-2.5.0.jar, beam-model-pipeline-2.5.0.jar define 6660 
overlapping classes:  [WARNING]   - 
org.apache.beam.vendor.netty.v4.io.netty.handler.codec.http.HttpClientCodec$1 
[WARNING]   - 
org.apache.beam.vendor.guava.v20.com.google.common.util.concurrent.AggregateFutureState$SafeAtomicHelper
 [WARNING]   - 
org.apache.beam.vendor.netty.v4.io.netty.util.concurrent.DefaultFutureListeners 
[WARNING]   - 
org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.OpenSslSessionContext$1 
[WARNING]   - 
org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.Java9SslUtils$4 [WARNING]  
 - 
org.apache.beam.vendor.guava.v20.com.google.common.collect.ImmutableMultimap$Builder
 [WARNING]   - 
org.apache.beam.vendor.netty.v4.io.netty.handler.codec.spdy.SpdyHeaders 
[WARNING]   - 
org.apache.beam.vendor.protobuf.v3.com.google.protobuf.DescriptorProtos$FieldDescriptorProtoOrBuilder
 [WARNING]   - 
org.apache.beam.vendor.guava.v20.com.google.common.collect.AbstractMultimap 
[WARNING]   - 
org.apache.beam.vendor.guava.v20.com.google.common.io.BaseEncoding$3{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle #78

[Apache Jenkins Server]

See 


Changes:

[batbat] Fixed spotless java error in KinesisRecord.java causing precommit

--
[...truncated 141.00 KB...]
Skip updating github.com/golang/glog#23def4e6c14b4da8ac2ed8007337bc5eb5007998 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/glog/c359ae0c52cbded42897d83346c85981.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/golang/mock#b3e60bcdc577185fce3cf625fc96b62857ce5574 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/mock/a5aba77edff874e2e3d9ad331994556c.
Skip updating 
github.com/golang/protobuf#3a3da3a4e26776cc22a79ef46d5d58477532dede in 
/home/jenkins/.gradle/go/repo/github.com/golang/protobuf/76cd502133aa2c5520c2b3e1debe630b.
Skip updating github.com/golang/snappy#553a641470496b2327abcac10b36396bd98e45c9 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/snappy/96c568bbd0eca3e3f6a41e15a438261a.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/google/go-cmp#3af367b6b30c263d47e8895973edcca9a49cf029 
in 
/home/jenkins/.gradle/go/repo/github.com/google/go-cmp/0e30ce638fec37a4c0a4040d380290b4.
Skip updating github.com/google/pprof#a8f279b7952b27edbcb72e5a6c69ee9be4c8ad93 
in 
/home/jenkins/.gradle/go/repo/github.com/google/pprof/580224cd7b76319263fe32ca011a0650.
Skip updating 
github.com/googleapis/gax-go#317e0006254c44a0ac427cc52a0e083ff0b9622f in 
/home/jenkins/.gradle/go/repo/github.com/googleapis/gax-go/cfbdb0eaeda3f1ea6b48e4ebedd6b7ac.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/hashicorp/hcl#23c074d0eceb2b8a5bfdbb271ab780cde70f05a8 
in 
/home/jenkins/.gradle/go/repo/github.com/hashicorp/hcl/37e0bb8b36d14da77f9f8c5babf91e48.
Skip updating 
github.com/ianlancetaylor/demangle#4883227f66371e02c4948937d3e2be1664d9be38 in 
/home/jenkins/.gradle/go/repo/github.com/ianlancetaylor/demangle/da9250878eadca7fe1a997f2c1e465b3.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/kr/fs#2788f0dbd16903de03cb8186e5c7d97b69ad387b in 
/home/jenkins/.gradle/go/repo/github.com/kr/fs/ef69c46d4366e9d326896066a4b027d4.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/magiconair/properties#49d762b9817ba1c2e9d0c69183c2b4a8b8f1d934 in 
/home/jenkins/.gradle/go/repo/github.com/magiconair/properties/34883f6c3f37cca1c4582d137f9fc020.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/mitchellh/go-homedir#b8bc1bf767474819792c23f32d8286a45736f1c6 in 
/home/jenkins/.gradle/go/repo/github.com/mitchellh/go-homedir/11e21d791561444efcbd8f83f3617926.
Skip updating 
github.com/mitchellh/mapstructure#a4e142e9c047c904fa2f1e144d9a84e6133024bc in 
/home/jenkins/.gradle/go/repo/github.com/mitchellh/mapstructure/8f497225aea52894fde79ad66bc0eb05.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/openzipkin/zipkin-go#3741243b287094fda649c7f0fa74bd51f37dc122 in 
/home/jenkins/.gradle/go/repo/github.com/openzipkin/zipkin-go/e53ea28b1632f7d2624f50816a45fa4e.
Skip updating 
github.com/pelletier/go-toml#acdc4509485b587f5e675510c4f2c63e90ff68a8 in 
/home/jenkins/.gradle/go/repo/github.com/pelletier/go-toml/fcb4bcb69fca9eab0a7b3cb341106167.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 

[beam] branch master updated (a1c50e1 -> a3f6f7e)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a1c50e1  Merge pull request #6362 from qinyeli/master
 add 4c8f457  Fixed spotless java error in KinesisRecord.java causing 
precommit failure
 new a3f6f7e  Merge pull request #6364: Fixed spotless java error in 
KinesisRecord.java

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/sdk/io/kinesis/KinesisRecord.java| 16 
 1 file changed, 4 insertions(+), 12 deletions(-)

[beam] 01/01: Merge pull request #6364: Fixed spotless java error in KinesisRecord.java

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a3f6f7e3b147f5a65e5b419d9baf24b35750974b
Merge: a1c50e1 4c8f457
Author: Ismaël Mejía 
AuthorDate: Tue Sep 11 23:32:04 2018 +0200

Merge pull request #6364: Fixed spotless java error in KinesisRecord.java

 .../org/apache/beam/sdk/io/kinesis/KinesisRecord.java| 16 
 1 file changed, 4 insertions(+), 12 deletions(-)

[jira] [Created] (BEAM-5365) Migrate integration tests for bigquery_tornadoes

[yifan zou (JIRA)]

yifan zou created BEAM-5365:
---

 Summary: Migrate integration tests for bigquery_tornadoes
 Key: BEAM-5365
 URL: https://issues.apache.org/jira/browse/BEAM-5365
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: yifan zou
Assignee: yifan zou






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5342) Migrate google-api-client libraries to 1.24.1

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5342?focusedWorklogId=143269=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143269
 ]

ASF GitHub Bot logged work on BEAM-5342:


Author: ASF GitHub Bot
Created on: 11/Sep/18 20:49
Start Date: 11/Sep/18 20:49
Worklog Time Spent: 10m 
  Work Description: chamikaramj opened a new pull request #6365: 
[BEAM-5342] Upgrades Google API Client libraries to 1.24.1
URL: https://github.com/apache/beam/pull/6365
 
 
   We currently use 1.23 libraries which is about an year old. We should 
migrate to more recent 1.24.1 which fixes several known issues.
   
   Statement according to https://beam.apache.org/contribute/dependencies/: 
This dependency change does not introduce any backwards incompatible API 
changes.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143269)
Time Spent: 10m
Remaining Estimate: 0h

> Migrate google-api-client libraries to 1.24.1
> -
>
> Key: BEAM-5342
> URL: https://issues.apache.org/jira/browse/BEAM-5342
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp, runner-dataflow
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We currently use 1.23 libraries which is about an year old. We should migrate 
> to more recent 1.24.1 which fixes several known issues.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #1010

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[Kevin Si (JIRA)]

Kevin Si created BEAM-5364:
--

 Summary:  BigtableIO source tries to validate table ID even though 
validation is turned off
 Key: BEAM-5364
 URL: https://issues.apache.org/jira/browse/BEAM-5364
 Project: Beam
  Issue Type: Bug
  Components: io-java-gcp
Reporter: Kevin Si
Assignee: Chamikara Jayalath


[https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg]

The validation can be turned off with following:
BigtableIO.read()
            .withoutValidation() // skip validation when constructing the 
pipelline.

A Dataflow template cannot be constructed due to this validation failure.

 

Error log when trying to construct a template:

Exception in thread "main" java.lang.IllegalArgumentException: tableId was not 
supplied
        at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
        at 
org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084)
        at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95)
        at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85)
        at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
        at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
        at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
        at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167)
        at 
org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423)
        at 
org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179)
        at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
        at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
        at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
        at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
        at 
com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #1009

[Apache Jenkins Server]

See 


Changes:

[ptomasroos] Revert worker compatible binary check since its failing on some 
systems

--
[...truncated 76.48 KB...]
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 

[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=143256=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143256
 ]

ASF GitHub Bot logged work on BEAM-5283:


Author: ASF GitHub Bot
Created on: 11/Sep/18 19:58
Start Date: 11/Sep/18 19:58
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing 
Flink Post commit jenkins task
URL: https://github.com/apache/beam/pull/6340#issuecomment-420404062
 
 
   Run Python Flink PortableValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143256)
Time Spent: 8h 40m  (was: 8.5h)

> Enable Python Portable Flink PostCommit Tests to Jenkins
> 
>
> Key: BEAM-5283
> URL: https://issues.apache.org/jira/browse/BEAM-5283
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Ankur Goenka
>Assignee: Jason Kuster
>Priority: Major
>  Labels: CI
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle #77

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Interactive Beam -- comments for pipeline_graph_renderer.

--
[...truncated 140.21 KB...]
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/golang/glog#23def4e6c14b4da8ac2ed8007337bc5eb5007998 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/glog/c359ae0c52cbded42897d83346c85981.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/golang/mock#b3e60bcdc577185fce3cf625fc96b62857ce5574 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/mock/a5aba77edff874e2e3d9ad331994556c.
Skip updating 
github.com/golang/protobuf#3a3da3a4e26776cc22a79ef46d5d58477532dede in 
/home/jenkins/.gradle/go/repo/github.com/golang/protobuf/76cd502133aa2c5520c2b3e1debe630b.
Skip updating github.com/golang/snappy#553a641470496b2327abcac10b36396bd98e45c9 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/snappy/96c568bbd0eca3e3f6a41e15a438261a.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/google/go-cmp#3af367b6b30c263d47e8895973edcca9a49cf029 
in 
/home/jenkins/.gradle/go/repo/github.com/google/go-cmp/0e30ce638fec37a4c0a4040d380290b4.
Skip updating github.com/google/pprof#a8f279b7952b27edbcb72e5a6c69ee9be4c8ad93 
in 
/home/jenkins/.gradle/go/repo/github.com/google/pprof/580224cd7b76319263fe32ca011a0650.
Skip updating 
github.com/googleapis/gax-go#317e0006254c44a0ac427cc52a0e083ff0b9622f in 
/home/jenkins/.gradle/go/repo/github.com/googleapis/gax-go/cfbdb0eaeda3f1ea6b48e4ebedd6b7ac.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/hashicorp/hcl#23c074d0eceb2b8a5bfdbb271ab780cde70f05a8 
in 
/home/jenkins/.gradle/go/repo/github.com/hashicorp/hcl/37e0bb8b36d14da77f9f8c5babf91e48.
Skip updating 
github.com/ianlancetaylor/demangle#4883227f66371e02c4948937d3e2be1664d9be38 in 
/home/jenkins/.gradle/go/repo/github.com/ianlancetaylor/demangle/da9250878eadca7fe1a997f2c1e465b3.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/kr/fs#2788f0dbd16903de03cb8186e5c7d97b69ad387b in 
/home/jenkins/.gradle/go/repo/github.com/kr/fs/ef69c46d4366e9d326896066a4b027d4.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/magiconair/properties#49d762b9817ba1c2e9d0c69183c2b4a8b8f1d934 in 
/home/jenkins/.gradle/go/repo/github.com/magiconair/properties/34883f6c3f37cca1c4582d137f9fc020.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/mitchellh/go-homedir#b8bc1bf767474819792c23f32d8286a45736f1c6 in 
/home/jenkins/.gradle/go/repo/github.com/mitchellh/go-homedir/11e21d791561444efcbd8f83f3617926.
Skip updating 
github.com/mitchellh/mapstructure#a4e142e9c047c904fa2f1e144d9a84e6133024bc in 
/home/jenkins/.gradle/go/repo/github.com/mitchellh/mapstructure/8f497225aea52894fde79ad66bc0eb05.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/openzipkin/zipkin-go#3741243b287094fda649c7f0fa74bd51f37dc122 in 
/home/jenkins/.gradle/go/repo/github.com/openzipkin/zipkin-go/e53ea28b1632f7d2624f50816a45fa4e.
Skip 

[beam] 01/01: Merge pull request #6362 from qinyeli/master

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a1c50e1483937613c1cca7e5b631a65de3714d2a
Merge: a25bbc7 91f7624
Author: Charles Chen 
AuthorDate: Tue Sep 11 12:55:35 2018 -0700

Merge pull request #6362 from qinyeli/master

Interactive Beam -- comments for pipeline_graph_renderer.

 .../apache_beam/runners/interactive/display/pipeline_graph_renderer.py | 3 +++
 1 file changed, 3 insertions(+)

[beam] branch master updated (a25bbc7 -> a1c50e1)

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a25bbc7  [BEAM-5357] Disable optimization for Go check 
IsWorkerCompatibleBinary
 add 91f7624  Interactive Beam -- comments for pipeline_graph_renderer.
 new a1c50e1  Merge pull request #6362 from qinyeli/master

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache_beam/runners/interactive/display/pipeline_graph_renderer.py | 3 +++
 1 file changed, 3 insertions(+)

Build failed in Jenkins: beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle #76

[Apache Jenkins Server]

See 


Changes:

[ptomasroos] Revert worker compatible binary check since its failing on some 
systems

--
[...truncated 141.49 KB...]
Skip updating github.com/golang/glog#23def4e6c14b4da8ac2ed8007337bc5eb5007998 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/glog/c359ae0c52cbded42897d83346c85981.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/golang/mock#b3e60bcdc577185fce3cf625fc96b62857ce5574 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/mock/a5aba77edff874e2e3d9ad331994556c.
Skip updating 
github.com/golang/protobuf#3a3da3a4e26776cc22a79ef46d5d58477532dede in 
/home/jenkins/.gradle/go/repo/github.com/golang/protobuf/76cd502133aa2c5520c2b3e1debe630b.
Skip updating github.com/golang/snappy#553a641470496b2327abcac10b36396bd98e45c9 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/snappy/96c568bbd0eca3e3f6a41e15a438261a.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/google/go-cmp#3af367b6b30c263d47e8895973edcca9a49cf029 
in 
/home/jenkins/.gradle/go/repo/github.com/google/go-cmp/0e30ce638fec37a4c0a4040d380290b4.
Skip updating github.com/google/pprof#a8f279b7952b27edbcb72e5a6c69ee9be4c8ad93 
in 
/home/jenkins/.gradle/go/repo/github.com/google/pprof/580224cd7b76319263fe32ca011a0650.
Skip updating 
github.com/googleapis/gax-go#317e0006254c44a0ac427cc52a0e083ff0b9622f in 
/home/jenkins/.gradle/go/repo/github.com/googleapis/gax-go/cfbdb0eaeda3f1ea6b48e4ebedd6b7ac.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/hashicorp/hcl#23c074d0eceb2b8a5bfdbb271ab780cde70f05a8 
in 
/home/jenkins/.gradle/go/repo/github.com/hashicorp/hcl/37e0bb8b36d14da77f9f8c5babf91e48.
Skip updating 
github.com/ianlancetaylor/demangle#4883227f66371e02c4948937d3e2be1664d9be38 in 
/home/jenkins/.gradle/go/repo/github.com/ianlancetaylor/demangle/da9250878eadca7fe1a997f2c1e465b3.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/kr/fs#2788f0dbd16903de03cb8186e5c7d97b69ad387b in 
/home/jenkins/.gradle/go/repo/github.com/kr/fs/ef69c46d4366e9d326896066a4b027d4.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/magiconair/properties#49d762b9817ba1c2e9d0c69183c2b4a8b8f1d934 in 
/home/jenkins/.gradle/go/repo/github.com/magiconair/properties/34883f6c3f37cca1c4582d137f9fc020.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/mitchellh/go-homedir#b8bc1bf767474819792c23f32d8286a45736f1c6 in 
/home/jenkins/.gradle/go/repo/github.com/mitchellh/go-homedir/11e21d791561444efcbd8f83f3617926.
Skip updating 
github.com/mitchellh/mapstructure#a4e142e9c047c904fa2f1e144d9a84e6133024bc in 
/home/jenkins/.gradle/go/repo/github.com/mitchellh/mapstructure/8f497225aea52894fde79ad66bc0eb05.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/openzipkin/zipkin-go#3741243b287094fda649c7f0fa74bd51f37dc122 in 
/home/jenkins/.gradle/go/repo/github.com/openzipkin/zipkin-go/e53ea28b1632f7d2624f50816a45fa4e.
Skip updating 
github.com/pelletier/go-toml#acdc4509485b587f5e675510c4f2c63e90ff68a8 in 
/home/jenkins/.gradle/go/repo/github.com/pelletier/go-toml/fcb4bcb69fca9eab0a7b3cb341106167.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 

[jira] [Work logged] (BEAM-5357) Go check for IsWorkerCompatibleBinary is wrong

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5357?focusedWorklogId=143253=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143253
 ]

ASF GitHub Bot logged work on BEAM-5357:


Author: ASF GitHub Bot
Created on: 11/Sep/18 19:45
Start Date: 11/Sep/18 19:45
Worklog Time Spent: 10m 
  Work Description: herohde closed pull request #6363: [BEAM-5357] Disable 
optimization for Go check IsWorkerCompatibleBinary
URL: https://github.com/apache/beam/pull/6363
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/go/pkg/beam/runners/universal/runnerlib/compile.go 
b/sdks/go/pkg/beam/runners/universal/runnerlib/compile.go
index 1dae66e1442..ecf5c1b4ab6 100644
--- a/sdks/go/pkg/beam/runners/universal/runnerlib/compile.go
+++ b/sdks/go/pkg/beam/runners/universal/runnerlib/compile.go
@@ -35,9 +35,6 @@ import (
 // IsWorkerCompatibleBinary returns the path to itself and true if running
 // a linux-amd64 binary that can directly be used as a worker binary.
 func IsWorkerCompatibleBinary() (string, bool) {
-   if runtime.GOOS == "linux" && runtime.GOARCH == "amd64" {
-   return os.Args[0], true
-   }
return "", false
 }
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143253)
Time Spent: 20m  (was: 10m)

> Go check for IsWorkerCompatibleBinary is wrong
> --
>
> Key: BEAM-5357
> URL: https://issues.apache.org/jira/browse/BEAM-5357
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Critical
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Per BEAM-5253, The linux/amd64 check in IsWorkerCompatibleBinary is 
> insufficient:
> https://github.com/apache/beam/blob/609a42978405173a60e5d91f35170a5c0b5d5332/sdks/go/pkg/beam/runners/universal/runnerlib/compile.go#L37
> We need to see if we can do a better check here (such as looking up the 
> symbol table or similar) or disable this optimization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: [BEAM-5357] Disable optimization for Go check IsWorkerCompatibleBinary

[herohde]

This is an automated email from the ASF dual-hosted git repository.

herohde pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a25bbc7cb36d102bec415dbfbb072c6319ff3397
Merge: 251dec5 6a00e2e
Author: Henning Rohde 
AuthorDate: Tue Sep 11 12:45:29 2018 -0700

[BEAM-5357] Disable optimization for Go check IsWorkerCompatibleBinary

 sdks/go/pkg/beam/runners/universal/runnerlib/compile.go | 3 ---
 1 file changed, 3 deletions(-)

[jira] [Work logged] (BEAM-5357) Go check for IsWorkerCompatibleBinary is wrong

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5357?focusedWorklogId=143254=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143254
 ]

ASF GitHub Bot logged work on BEAM-5357:


Author: ASF GitHub Bot
Created on: 11/Sep/18 19:45
Start Date: 11/Sep/18 19:45
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #6363: [BEAM-5357] Disable 
optimization for Go check IsWorkerCompatibleBinary
URL: https://github.com/apache/beam/pull/6363#issuecomment-420399903
 
 
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143254)
Time Spent: 0.5h  (was: 20m)

> Go check for IsWorkerCompatibleBinary is wrong
> --
>
> Key: BEAM-5357
> URL: https://issues.apache.org/jira/browse/BEAM-5357
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Critical
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Per BEAM-5253, The linux/amd64 check in IsWorkerCompatibleBinary is 
> insufficient:
> https://github.com/apache/beam/blob/609a42978405173a60e5d91f35170a5c0b5d5332/sdks/go/pkg/beam/runners/universal/runnerlib/compile.go#L37
> We need to see if we can do a better check here (such as looking up the 
> symbol table or similar) or disable this optimization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (251dec5 -> a25bbc7)

[herohde]

This is an automated email from the ASF dual-hosted git repository.

herohde pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 251dec5  Merge pull request #6356 from qinyeli/master
 add 6a00e2e  Revert worker compatible binary check since its failing on 
some systems
 new a25bbc7  [BEAM-5357] Disable optimization for Go check 
IsWorkerCompatibleBinary

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/runners/universal/runnerlib/compile.go | 3 ---
 1 file changed, 3 deletions(-)

[jira] [Work logged] (BEAM-5357) Go check for IsWorkerCompatibleBinary is wrong

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5357?focusedWorklogId=143252=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143252
 ]

ASF GitHub Bot logged work on BEAM-5357:


Author: ASF GitHub Bot
Created on: 11/Sep/18 19:38
Start Date: 11/Sep/18 19:38
Worklog Time Spent: 10m 
  Work Description: ptomasroos opened a new pull request #6363: [BEAM-5357] 
Disable optimization for Go check IsWorkerCompatibleBinary
URL: https://github.com/apache/beam/pull/6363
 
 
   This disabled the IsWorkerCompatibleBinary optimization since its failing on 
some systems.
   
   @herohde @lostluck 
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143252)
Time Spent: 10m
Remaining Estimate: 0h

> Go check for IsWorkerCompatibleBinary is wrong
> --
>
> Key: BEAM-5357
> URL: https://issues.apache.org/jira/browse/BEAM-5357
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Critical
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Per BEAM-5253, The linux/amd64 check in IsWorkerCompatibleBinary is 
> insufficient:
> https://github.com/apache/beam/blob/609a42978405173a60e5d91f35170a5c0b5d5332/sdks/go/pkg/beam/runners/universal/runnerlib/compile.go#L37
> We need to see if we can do a better check here (such as looking up the 
> symbol table or similar) or disable this optimization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5219) Expose OutboundMessage in PubSub client

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5219?focusedWorklogId=143251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143251
 ]

ASF GitHub Bot logged work on BEAM-5219:


Author: ASF GitHub Bot
Created on: 11/Sep/18 19:27
Start Date: 11/Sep/18 19:27
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6276: [BEAM-5219] Make 
OutgoingMessage public
URL: https://github.com/apache/beam/pull/6276#issuecomment-420393814
 
 
   Thanks @chamikaramj !


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143251)
Time Spent: 50m  (was: 40m)

> Expose OutboundMessage in PubSub client
> ---
>
> Key: BEAM-5219
> URL: https://issues.apache.org/jira/browse/BEAM-5219
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> publish method in org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java is 
> public but the argument OutboundMessage is not public which makes the api 
> unusable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Java_Cron #330

[Apache Jenkins Server]

See 


Changes:

[Jozef.Vilcek] Enable to configure latencyTrackingInterval

[iemejia] Update KinesisRecord based on Amazon documentation

[mxm] Apply whitespace fix for 0445c757d6103405cfe3ae5d4406d823f98aee73

--
[...truncated 19.50 MB...]
log4j:WARN No appenders could be found for logger 
(org.apache.beam.sdk.coders.CoderRegistry).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for 
more info.
Finished generating test XML results (0.0 secs) into: 

Generating HTML test report...
Finished generating test html results (0.001 secs) into: 

Packing task ':beam-sdks-java-io-hadoop-common:test'
:beam-sdks-java-io-hadoop-common:test (Thread[Task worker for ':' Thread 
5,5,main]) completed. Took 1.295 secs.
:beam-sdks-java-io-hadoop-common:validateShadedJarDoesntLeakNonOrgApacheBeamClasses
 (Thread[Task worker for ':' Thread 5,5,main]) started.

> Task 
> :beam-sdks-java-io-hadoop-common:validateShadedJarDoesntLeakNonOrgApacheBeamClasses
Caching disabled for task 
':beam-sdks-java-io-hadoop-common:validateShadedJarDoesntLeakNonOrgApacheBeamClasses':
 Caching has not been enabled for the task
Task 
':beam-sdks-java-io-hadoop-common:validateShadedJarDoesntLeakNonOrgApacheBeamClasses'
 is not up-to-date because:
  Task has not declared any outputs despite executing actions.
:beam-sdks-java-io-hadoop-common:validateShadedJarDoesntLeakNonOrgApacheBeamClasses
 (Thread[Task worker for ':' Thread 5,5,main]) completed. Took 0.001 secs.
:beam-sdks-java-io-hadoop-common:check (Thread[Task worker for ':' Thread 
5,5,main]) started.

> Task :beam-sdks-java-io-hadoop-common:check
Skipping task ':beam-sdks-java-io-hadoop-common:check' as it has no actions.
:beam-sdks-java-io-hadoop-common:check (Thread[Task worker for ':' Thread 
5,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-hadoop-common:build (Thread[Task worker for ':',5,main]) 
started.

> Task :beam-sdks-java-io-hadoop-common:build
Skipping task ':beam-sdks-java-io-hadoop-common:build' as it has no actions.
:beam-sdks-java-io-hadoop-common:build (Thread[Task worker for ':',5,main]) 
completed. Took 0.0 secs.
:beam-sdks-java-io-hadoop-common:buildDependents (Thread[Task worker for 
':',5,main]) started.

> Task :beam-sdks-java-io-hadoop-common:buildDependents
Caching disabled for task ':beam-sdks-java-io-hadoop-common:buildDependents': 
Caching has not been enabled for the task
Task ':beam-sdks-java-io-hadoop-common:buildDependents' is not up-to-date 
because:
  Task has not declared any outputs despite executing actions.
:beam-sdks-java-io-hadoop-common:buildDependents (Thread[Task worker for 
':',5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:jar (Thread[Task worker for 
':',5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:jar
Build cache key for task ':beam-vendor-sdks-java-extensions-protobuf:jar' is 
318184f3acfe150a3a38d605826c992a
Caching disabled for task ':beam-vendor-sdks-java-extensions-protobuf:jar': 
Caching has not been enabled for the task
Task ':beam-vendor-sdks-java-extensions-protobuf:jar' is not up-to-date because:
  No history is available.
:beam-vendor-sdks-java-extensions-protobuf:jar (Thread[Task worker for 
':',5,main]) completed. Took 0.006 secs.
:beam-vendor-sdks-java-extensions-protobuf:compileTestJava (Thread[Task worker 
for ':',5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:compileTestJava' as 
it has no source files and no previous output files.
:beam-vendor-sdks-java-extensions-protobuf:compileTestJava (Thread[Task worker 
for ':',5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:processTestResources (Thread[Task 
worker for ':',5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:processTestResources' 
as it has no source files and no previous output files.
:beam-vendor-sdks-java-extensions-protobuf:processTestResources (Thread[Task 
worker for ':',5,main]) completed. Took 0.001 secs.
:beam-vendor-sdks-java-extensions-protobuf:testClasses (Thread[Task worker for 
':',5,main]) started.

> Task 

Build failed in Jenkins: beam_PerformanceTests_Python #1425

[Apache Jenkins Server]

See 


Changes:

[Jozef.Vilcek] Enable to configure latencyTrackingInterval

[qinyeli] Interactive Beam -- enable setting render option

[mxm] Apply whitespace fix for 0445c757d6103405cfe3ae5d4406d823f98aee73

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam15 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 251dec50e7ab01838c93c8e91c94f25aca3b624c (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 251dec50e7ab01838c93c8e91c94f25aca3b624c
Commit message: "Merge pull request #6356 from qinyeli/master"
 > git rev-list --no-walk b82d80c4b3d2c250ac9ae89cc7543e1c170997d1 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4069796641908196160.sh
+ rm -rf 

[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8581431818106135268.sh
+ rm -rf 

[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2380551140104865140.sh
+ virtualenv 

New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2372763421150240581.sh
+ 

 install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./env/.perfkit_env/lib/python2.7/site-packages (40.2.0)
Requirement already up-to-date: pip in 
./env/.perfkit_env/lib/python2.7/site-packages (18.0)
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1460709699949866581.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git 

Cloning into 
'
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3984983650522256220.sh
+ 

 install -r 

Collecting absl-py (from -r 

 (line 14))
Collecting jinja2>=2.7 (from -r 

 (line 15))
  Using cached 
https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
Requirement already satisfied: setuptools in 
./env/.perfkit_env/lib/python2.7/site-packages (from -r 

 (line 16)) (40.2.0)
Collecting colorlog[windows]==2.6.0 (from -r 

 (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r 

 (line 18))
Collecting futures>=3.0.3 (from -r 

 (line 19))
  Using cached 

[jira] [Created] (BEAM-5363) Java DirectRunner should implement retry and support @RequiresStableInput

[Yueyang Qiu (JIRA)]

Yueyang Qiu created BEAM-5363:
-

 Summary: Java DirectRunner should implement retry and support 
@RequiresStableInput
 Key: BEAM-5363
 URL: https://issues.apache.org/jira/browse/BEAM-5363
 Project: Beam
  Issue Type: New Feature
  Components: runner-direct
Reporter: Yueyang Qiu
Assignee: Daniel Oliveira


[https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle #75

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Interactive Beam -- enable setting render option

--
[...truncated 141.84 KB...]
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/golang/glog#23def4e6c14b4da8ac2ed8007337bc5eb5007998 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/glog/c359ae0c52cbded42897d83346c85981.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/golang/mock#b3e60bcdc577185fce3cf625fc96b62857ce5574 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/mock/a5aba77edff874e2e3d9ad331994556c.
Skip updating 
github.com/golang/protobuf#3a3da3a4e26776cc22a79ef46d5d58477532dede in 
/home/jenkins/.gradle/go/repo/github.com/golang/protobuf/76cd502133aa2c5520c2b3e1debe630b.
Skip updating github.com/golang/snappy#553a641470496b2327abcac10b36396bd98e45c9 
in 
/home/jenkins/.gradle/go/repo/github.com/golang/snappy/96c568bbd0eca3e3f6a41e15a438261a.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/google/go-cmp#3af367b6b30c263d47e8895973edcca9a49cf029 
in 
/home/jenkins/.gradle/go/repo/github.com/google/go-cmp/0e30ce638fec37a4c0a4040d380290b4.
Skip updating github.com/google/pprof#a8f279b7952b27edbcb72e5a6c69ee9be4c8ad93 
in 
/home/jenkins/.gradle/go/repo/github.com/google/pprof/580224cd7b76319263fe32ca011a0650.
Skip updating 
github.com/googleapis/gax-go#317e0006254c44a0ac427cc52a0e083ff0b9622f in 
/home/jenkins/.gradle/go/repo/github.com/googleapis/gax-go/cfbdb0eaeda3f1ea6b48e4ebedd6b7ac.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/hashicorp/hcl#23c074d0eceb2b8a5bfdbb271ab780cde70f05a8 
in 
/home/jenkins/.gradle/go/repo/github.com/hashicorp/hcl/37e0bb8b36d14da77f9f8c5babf91e48.
Skip updating 
github.com/ianlancetaylor/demangle#4883227f66371e02c4948937d3e2be1664d9be38 in 
/home/jenkins/.gradle/go/repo/github.com/ianlancetaylor/demangle/da9250878eadca7fe1a997f2c1e465b3.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/kr/fs#2788f0dbd16903de03cb8186e5c7d97b69ad387b in 
/home/jenkins/.gradle/go/repo/github.com/kr/fs/ef69c46d4366e9d326896066a4b027d4.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/magiconair/properties#49d762b9817ba1c2e9d0c69183c2b4a8b8f1d934 in 
/home/jenkins/.gradle/go/repo/github.com/magiconair/properties/34883f6c3f37cca1c4582d137f9fc020.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/mitchellh/go-homedir#b8bc1bf767474819792c23f32d8286a45736f1c6 in 
/home/jenkins/.gradle/go/repo/github.com/mitchellh/go-homedir/11e21d791561444efcbd8f83f3617926.
Skip updating 
github.com/mitchellh/mapstructure#a4e142e9c047c904fa2f1e144d9a84e6133024bc in 
/home/jenkins/.gradle/go/repo/github.com/mitchellh/mapstructure/8f497225aea52894fde79ad66bc0eb05.
Skip updating github.com/coreos/etcd#11214aa33bf5a47d3d9d8dafe0f6b97237dfe921 
in 
/home/jenkins/.gradle/go/repo/github.com/coreos/etcd/e07ad470c42c48737a8fdaa60abf6316.
Skip updating 
github.com/openzipkin/zipkin-go#3741243b287094fda649c7f0fa74bd51f37dc122 in 
/home/jenkins/.gradle/go/repo/github.com/openzipkin/zipkin-go/e53ea28b1632f7d2624f50816a45fa4e.
Skip updating 

[beam] 01/01: Merge pull request #6356 from qinyeli/master

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 251dec50e7ab01838c93c8e91c94f25aca3b624c
Merge: 609a429 21f2160
Author: Charles Chen 
AuthorDate: Tue Sep 11 11:09:18 2018 -0700

Merge pull request #6356 from qinyeli/master

Interactive Beam -- enable setting render option

 .../runners/interactive/display/__init__.py|  16 +++
 .../interactive/{ => display}/display_manager.py   |  21 ++--
 .../{ => display}/interactive_pipeline_graph.py|   2 +-
 .../interactive/{ => display}/pipeline_graph.py|  10 +-
 .../interactive/display/pipeline_graph_renderer.py | 125 +
 .../runners/interactive/interactive_runner.py  |  27 -
 6 files changed, 181 insertions(+), 20 deletions(-)

[beam] branch master updated (609a429 -> 251dec5)

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 609a429  Apply whitespace fix for 
0445c757d6103405cfe3ae5d4406d823f98aee73
 add 21f2160  Interactive Beam -- enable setting render option
 new 251dec5  Merge pull request #6356 from qinyeli/master

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../interactive/display}/__init__.py   |   0
 .../interactive/{ => display}/display_manager.py   |  21 ++--
 .../{ => display}/interactive_pipeline_graph.py|   2 +-
 .../interactive/{ => display}/pipeline_graph.py|  10 +-
 .../interactive/display/pipeline_graph_renderer.py | 125 +
 .../runners/interactive/interactive_runner.py  |  27 -
 6 files changed, 165 insertions(+), 20 deletions(-)
 copy sdks/python/apache_beam/{testing/benchmarks/nexmark/queries => 
runners/interactive/display}/__init__.py (100%)
 rename sdks/python/apache_beam/runners/interactive/{ => 
display}/display_manager.py (90%)
 rename sdks/python/apache_beam/runners/interactive/{ => 
display}/interactive_pipeline_graph.py (98%)
 rename sdks/python/apache_beam/runners/interactive/{ => 
display}/pipeline_graph.py (95%)
 create mode 100644 
sdks/python/apache_beam/runners/interactive/display/pipeline_graph_renderer.py

[jira] [Created] (BEAM-5362) Support @RequiresStableInput on Samza runner

[Yueyang Qiu (JIRA)]

Yueyang Qiu created BEAM-5362:
-

 Summary: Support @RequiresStableInput on Samza runner
 Key: BEAM-5362
 URL: https://issues.apache.org/jira/browse/BEAM-5362
 Project: Beam
  Issue Type: New Feature
  Components: runner-samza
Reporter: Yueyang Qiu
Assignee: Xinyu Liu


[https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5361) Support @RequiresStableInput on Gearpump runner

[Yueyang Qiu (JIRA)]

Yueyang Qiu created BEAM-5361:
-

 Summary: Support @RequiresStableInput on Gearpump runner
 Key: BEAM-5361
 URL: https://issues.apache.org/jira/browse/BEAM-5361
 Project: Beam
  Issue Type: New Feature
  Components: runner-gearpump
Reporter: Yueyang Qiu
Assignee: Manu Zhang


[https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5360) Support @RequiresStableInput on Apex runner

[Yueyang Qiu (JIRA)]

Yueyang Qiu created BEAM-5360:
-

 Summary: Support @RequiresStableInput on Apex runner
 Key: BEAM-5360
 URL: https://issues.apache.org/jira/browse/BEAM-5360
 Project: Beam
  Issue Type: New Feature
  Components: runner-apex
Reporter: Yueyang Qiu


[https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5359) Support @RequiresStableInput on Flink runner

[Yueyang Qiu (JIRA)]

Yueyang Qiu created BEAM-5359:
-

 Summary: Support @RequiresStableInput on Flink runner
 Key: BEAM-5359
 URL: https://issues.apache.org/jira/browse/BEAM-5359
 Project: Beam
  Issue Type: New Feature
  Components: runner-flink
Reporter: Yueyang Qiu
Assignee: Aljoscha Krettek


[https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5358) Support @RequiresStableInput on Spark runner

[Yueyang Qiu (JIRA)]

Yueyang Qiu created BEAM-5358:
-

 Summary: Support @RequiresStableInput on Spark runner
 Key: BEAM-5358
 URL: https://issues.apache.org/jira/browse/BEAM-5358
 Project: Beam
  Issue Type: New Feature
  Components: runner-spark
Reporter: Yueyang Qiu
Assignee: Amit Sela


[https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle #74

[Apache Jenkins Server]

See 


--
[...truncated 250.25 KB...]
at 
com.github.blindpirate.gogradle.core.pack.MetadataPackagePathResolver.fetchViaWeb(MetadataPackagePathResolver.java:73)
at 
com.github.blindpirate.gogradle.core.pack.MetadataPackagePathResolver.produce(MetadataPackagePathResolver.java:63)
at 
com.github.blindpirate.gogradle.core.pack.MetadataPackagePathResolver$$EnhancerByGuice$$a0ad225f.CGLIB$produce$1()
at 
com.github.blindpirate.gogradle.core.pack.MetadataPackagePathResolver$$EnhancerByGuice$$a0ad225f$$FastClassByGuice$$8ae5c4a.invoke()
at 
com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
at 
com.github.blindpirate.gogradle.util.logging.DebugLogMethodInterceptor.invoke(DebugLogMethodInterceptor.java:38)
at 
com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
at 
com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
at 
com.github.blindpirate.gogradle.core.pack.MetadataPackagePathResolver$$EnhancerByGuice$$a0ad225f.produce()
at 
com.github.blindpirate.gogradle.core.pack.MetadataPackagePathResolver.produce(MetadataPackagePathResolver.java:44)
at 
com.github.blindpirate.gogradle.core.pack.MetadataPackagePathResolver$$EnhancerByGuice$$a0ad225f.produce()
at 
com.github.blindpirate.gogradle.util.FactoryUtil.lambda$produce$0(FactoryUtil.java:29)
at 
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at 
java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958)
at 
java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
at 
java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
at 
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at 
java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464)
at 
com.github.blindpirate.gogradle.util.FactoryUtil.produce(FactoryUtil.java:32)
at 
com.github.blindpirate.gogradle.core.pack.DefaultPackagePathResolver.produce(DefaultPackagePathResolver.java:61)
at 
com.github.blindpirate.gogradle.core.dependency.parse.DefaultMapNotationParser.determinePackage(DefaultMapNotationParser.java:105)
at 
com.github.blindpirate.gogradle.core.dependency.parse.DefaultMapNotationParser.parse(DefaultMapNotationParser.java:71)
at 
com.github.blindpirate.gogradle.util.DependencySetUtils.parseMany(DependencySetUtils.java:30)
at 
com.github.blindpirate.gogradle.core.dependency.produce.ExternalDependencyFactory.produce(ExternalDependencyFactory.java:73)
at 
com.github.blindpirate.gogradle.core.dependency.produce.DefaultDependencyVisitor.visitExternalDependencies(DefaultDependencyVisitor.java:64)
at 
com.github.blindpirate.gogradle.core.dependency.produce.DefaultDependencyVisitor$$EnhancerByGuice$$916a3e6.CGLIB$visitExternalDependencies$1()
at 
com.github.blindpirate.gogradle.core.dependency.produce.DefaultDependencyVisitor$$EnhancerByGuice$$916a3e6$$FastClassByGuice$$ab8b0f24.invoke()
at 
com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
at 
com.github.blindpirate.gogradle.util.logging.DebugLogMethodInterceptor.invoke(DebugLogMethodInterceptor.java:38)
at 
com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
at 
com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
at 
com.github.blindpirate.gogradle.core.dependency.produce.DefaultDependencyVisitor$$EnhancerByGuice$$916a3e6.visitExternalDependencies()
at 
com.github.blindpirate.gogradle.core.dependency.produce.strategy.DefaultDependencyProduceStrategy.visitAll(DefaultDependencyProduceStrategy.java:55)
at 
com.github.blindpirate.gogradle.core.dependency.produce.strategy.DefaultDependencyProduceStrategy.produce(DefaultDependencyProduceStrategy.java:46)
at 
com.github.blindpirate.gogradle.core.dependency.ResolveContext.lambda$produceTransitiveDependencies$0(ResolveContext.java:83)
at 

[jira] [Updated] (BEAM-3194) Support annotating that a DoFn requires stable / deterministic input for replay/retry

[Yueyang Qiu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yueyang Qiu updated BEAM-3194:
--
Component/s: sdk-java-core

> Support annotating that a DoFn requires stable / deterministic input for 
> replay/retry
> -
>
> Key: BEAM-3194
> URL: https://issues.apache.org/jira/browse/BEAM-3194
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Yueyang Qiu
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> See the thread: 
> https://lists.apache.org/thread.html/5fd81ce371aeaf642665348f8e6940e308e04275dd7072f380f9f945@%3Cdev.beam.apache.org%3E
> We need this in order to have truly cross-runner end-to-end exactly once via 
> replay + idempotence.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4904) Beam Dependency Update Request: de.flapdoodle.embed:de.flapdoodle.embed.mongo 2.1.1

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4904?focusedWorklogId=143220=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143220
 ]

ASF GitHub Bot logged work on BEAM-4904:


Author: ASF GitHub Bot
Created on: 11/Sep/18 18:01
Start Date: 11/Sep/18 18:01
Worklog Time Spent: 10m 
  Work Description: huygaa11 commented on issue #6281: 
[BEAM-4904][BEAM-4905] Upgrade Flapdoodle OSS dependencies
URL: https://github.com/apache/beam/pull/6281#issuecomment-420365065
 
 
   ping!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143220)
Time Spent: 0.5h  (was: 20m)

> Beam Dependency Update Request: de.flapdoodle.embed:de.flapdoodle.embed.mongo 
> 2.1.1
> ---
>
> Key: BEAM-4904
> URL: https://issues.apache.org/jira/browse/BEAM-4904
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Chamikara Jayalath
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> 2018-07-25 20:23:49.911490
> Please review and upgrade the 
> de.flapdoodle.embed:de.flapdoodle.embed.mongo to the latest version 2.1.1 
>  
> cc: 
> 2018-08-06 12:09:30.976479
> Please review and upgrade the 
> de.flapdoodle.embed:de.flapdoodle.embed.mongo to the latest version 2.1.1 
>  
> cc: 
> 2018-08-13 12:09:48.188897
> Please review and upgrade the 
> de.flapdoodle.embed:de.flapdoodle.embed.mongo to the latest version 2.1.1 
>  
> cc: 
> 2018-08-20 12:12:32.344889
> Please review and upgrade the 
> de.flapdoodle.embed:de.flapdoodle.embed.mongo to the latest version 2.1.1 
>  
> cc: 
> 2018-08-27 12:14:00.846640
> Please review and upgrade the 
> de.flapdoodle.embed:de.flapdoodle.embed.mongo to the latest version 2.1.1 
>  
> cc: 
> 2018-09-03 12:26:28.154799
> Please review and upgrade the 
> de.flapdoodle.embed:de.flapdoodle.embed.mongo to the latest version 2.1.1 
>  
> cc: 
> 2018-09-10 12:15:38.875437
> Please review and upgrade the 
> de.flapdoodle.embed:de.flapdoodle.embed.mongo to the latest version 2.1.1 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4684) Support @RequiresStableInput on Dataflow runner in Java SDK

[Yueyang Qiu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yueyang Qiu updated BEAM-4684:
--
Component/s: (was: sdk-java-core)
 runner-dataflow

> Support @RequiresStableInput on Dataflow runner in Java SDK
> ---
>
> Key: BEAM-4684
> URL: https://issues.apache.org/jira/browse/BEAM-4684
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Website_Cron #47

[Apache Jenkins Server]

See 


Changes:

[Jozef.Vilcek] Enable to configure latencyTrackingInterval

[iemejia] Update KinesisRecord based on Amazon documentation

[mxm] Apply whitespace fix for 0445c757d6103405cfe3ae5d4406d823f98aee73

--
[...truncated 8.06 KB...]
:assemble (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.0 secs.
:spotlessGroovy (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:spotlessGroovy (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
1.468 secs.
:spotlessGroovyCheck (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:spotlessGroovyCheck (Thread[Task worker for ':buildSrc',5,main]) completed. 
Took 0.0 secs.
:spotlessGroovyGradle (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:spotlessGroovyGradle (Thread[Task worker for ':buildSrc',5,main]) completed. 
Took 0.036 secs.
:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.0 secs.
:spotlessCheck (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:spotlessCheck (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.0 
secs.
:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.002 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.004 secs.
:processTestResources (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc',5,main]) completed. 
Took 0.003 secs.
:testClasses (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.001 
secs.
:test (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.005 secs.
:check (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:check
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.0 secs.
:build (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:build
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Task worker for ':buildSrc',5,main]) 

[jira] [Work logged] (BEAM-4780) Entry point for ULR JobService compatible with TestPortableRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4780?focusedWorklogId=143219=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143219
 ]

ASF GitHub Bot logged work on BEAM-4780:


Author: ASF GitHub Bot
Created on: 11/Sep/18 18:00
Start Date: 11/Sep/18 18:00
Worklog Time Spent: 10m 
  Work Description: huygaa11 commented on issue #6151: [BEAM-4780] Updating 
to DockerJobBundleFactory in ReferenceRunner.
URL: https://github.com/apache/beam/pull/6151#issuecomment-420364594
 
 
   @bsidhom could you take the next appropriate action? Thank you.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143219)
Time Spent: 1h 20m  (was: 1h 10m)

> Entry point for ULR JobService compatible with TestPortableRunner
> -
>
> Key: BEAM-4780
> URL: https://issues.apache.org/jira/browse/BEAM-4780
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Eugene Kirpichov
>Assignee: Daniel Oliveira
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/pull/5935] introduces a TestPortableRunner 
> that can run ValidatesRunner tests against a given portable runner, 
> identified by a class name of a shim that can start/stop its JobService 
> endpoint.
> For ULR to run VR tests, it needs to provide such a shim.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5299) Define max global window as a shared value in protos like URN enums.

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611025#comment-16611025
 ] 

Maximilian Michels commented on BEAM-5299:
--

Thanks for the pointer [~lcwik]. What I did was adding another EnumValueOption: 

{code:java}
extend google.protobuf.EnumValueOptions {
  string beam_urn = 185324356;
  int64 global_window_max_timestamp_millis = 185324357;
}
{code}
 
But it makes much more sense to have a string constant field. Or even to reuse 
beam_urn and make it generic:

{code:java}
extend google.protobuf.EnumValueOptions {
  string beam_constant = 185324356;
}

message Constants {
  enum Constants {
// The minimum timestamp in milliseconds since Jan 1, 1970
TIMESTAMP_MIN_MILLIS = 0 [(beam_constant) = "-9223372036854775"]; 
// The maximum timestamp in milliseconds since Jan 1, 1970
TIMESTAMP_MAX_MILLIS = 0 [(beam_constant) = "9223372036854775"]; 
// The maximum timestamp for the global window in milliseconds since Jan 1, 
1970
// Triggers use maxTimestamp to set timers' timestamp. Timers fires when
// the watermark passes their timestamps. So, the timestamp needs to be
// smaller than the TIMESTAMP_MAX_MILLIS.
GLOBAL_WINDOW_MAX_TIMESTAMP_MILLIS = 1 [(beam_constant) = 
"9223371950454775"];
  }
}
{code}

> Define max global window as a shared value in protos like URN enums.
> 
>
> Key: BEAM-5299
> URL: https://issues.apache.org/jira/browse/BEAM-5299
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
>Reporter: Luke Cwik
>Assignee: Maximilian Michels
>Priority: Minor
>  Labels: portability
>
> Instead of having each language define a max timestamp themselves, define the 
> max timestamps within proto to be shared across different languages.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4684) Support @RequiresStableInput on Dataflow runner in Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4684?focusedWorklogId=143218=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143218
 ]

ASF GitHub Bot logged work on BEAM-4684:


Author: ASF GitHub Bot
Created on: 11/Sep/18 17:58
Start Date: 11/Sep/18 17:58
Worklog Time Spent: 10m 
  Work Description: robinyqiu commented on issue #6220: [BEAM-4684] Add 
integration test for support of @RequiresStableInput
URL: https://github.com/apache/beam/pull/6220#issuecomment-420363978
 
 
   Change link to BEAM-4684 (A specific JIRA under BEAM-3194)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143218)
Time Spent: 2h 50m  (was: 2h 40m)

> Support @RequiresStableInput on Dataflow runner in Java SDK
> ---
>
> Key: BEAM-4684
> URL: https://issues.apache.org/jira/browse/BEAM-4684
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5253) Go SDK PubSub example currently broken

[Henning Rohde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611017#comment-16611017
 ] 

Henning Rohde commented on BEAM-5253:
-

Great. Thanks for investigating!  Opened 
https://issues.apache.org/jira/browse/BEAM-5357.

[~ptomasroos] Just to be clear: are you sending a PR with the short-term fix of 
just always returning false or would you like me to do it?

> Go SDK PubSub example currently broken
> --
>
> Key: BEAM-5253
> URL: https://issues.apache.org/jira/browse/BEAM-5253
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Sean Patrick Hagen
>Assignee: Robert Burke
>Priority: Critical
> Fix For: Not applicable
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The Go SDK contains an example for creating a streaming pipeline that reads 
> from pubsub and outputs the messages. It can be found here: 
> [https://github.com/apache/beam/blob/master/sdks/go/examples/streaming_wordcap/wordcap.go]
>  
> This example is broken and does not work. It fails with the error "failed to 
> execute job: translation failed: no root units" when I try to run it with the 
> direct runner, and it just fails with "Internal Issue (8ed815a0a259018f): 
> 65177287:8503 " when run in Google Dataflow.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4684) Support @RequiresStableInput on Dataflow runner in Java SDK

[Yueyang Qiu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yueyang Qiu updated BEAM-4684:
--
Issue Type: New Feature  (was: Bug)

> Support @RequiresStableInput on Dataflow runner in Java SDK
> ---
>
> Key: BEAM-4684
> URL: https://issues.apache.org/jira/browse/BEAM-4684
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5357) Go check for IsWorkerCompatibleBinary is wrong

[Henning Rohde (JIRA)]

Henning Rohde created BEAM-5357:
---

 Summary: Go check for IsWorkerCompatibleBinary is wrong
 Key: BEAM-5357
 URL: https://issues.apache.org/jira/browse/BEAM-5357
 Project: Beam
  Issue Type: Bug
  Components: sdk-go
Reporter: Henning Rohde
Assignee: Robert Burke


Per BEAM-5253, The linux/amd64 check in IsWorkerCompatibleBinary is 
insufficient:

https://github.com/apache/beam/blob/609a42978405173a60e5d91f35170a5c0b5d5332/sdks/go/pkg/beam/runners/universal/runnerlib/compile.go#L37

We need to see if we can do a better check here (such as looking up the symbol 
table or similar) or disable this optimization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5253) Go SDK PubSub example currently broken

[Tomas Roos (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611010#comment-16611010
 ] 

Tomas Roos commented on BEAM-5253:
--

When returning false from IsWorkerCompatibleBinary everything works as expected 
with go run wordcap.go 

So yes we should open a issue to work on a better behaviour for this lookup! 
Thanks Henning

> Go SDK PubSub example currently broken
> --
>
> Key: BEAM-5253
> URL: https://issues.apache.org/jira/browse/BEAM-5253
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Sean Patrick Hagen
>Assignee: Robert Burke
>Priority: Critical
> Fix For: Not applicable
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The Go SDK contains an example for creating a streaming pipeline that reads 
> from pubsub and outputs the messages. It can be found here: 
> [https://github.com/apache/beam/blob/master/sdks/go/examples/streaming_wordcap/wordcap.go]
>  
> This example is broken and does not work. It fails with the error "failed to 
> execute job: translation failed: no root units" when I try to run it with the 
> direct runner, and it just fails with "Internal Issue (8ed815a0a259018f): 
> 65177287:8503 " when run in Google Dataflow.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3098) Upgrade Java grpc version

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611006#comment-16611006
 ] 

Luke Cwik commented on BEAM-3098:
-

[~kbolton], the 2.7.0 release is ongoing right now. You can follow progress on 
the [d...@beam.apache.org|mailto:d...@beam.apache.org] list.

> Upgrade Java grpc version
> -
>
> Key: BEAM-3098
> URL: https://issues.apache.org/jira/browse/BEAM-3098
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Solomon Duskis
>Assignee: Garrett Jones
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Beam Java currently depends on grpc 1.2, which was released in March.  It 
> would be great if the dependency could be update to something newer, like 
> grpc 1.7.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5350) Running autocomplete.go on dataflow fails

[Robert Burke (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610990#comment-16610990
 ] 

Robert Burke commented on BEAM-5350:


BEAM-4472(https://issues.apache.org/jira/browse/BEAM-4472) is about the Top 
issue. The accumulator type is not presently encodeable and it wasn't used at 
all until Combiner Lifting was properly added.

A good avenue to fixing that would be to do something similar to beam.Create 
for handling encoding those values instead of what's currently written.

> Running autocomplete.go on dataflow fails
> -
>
> Key: BEAM-5350
> URL: https://issues.apache.org/jira/browse/BEAM-5350
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Tomas Roos
>Assignee: Henning Rohde
>Priority: Major
>
> I'm in the process as a external developer make sure that all examples are 
> runnable on both direct and the dataflow runner as its crucial for people 
> onboarding this project.
>  
> I've visted the projects before and some are runnable, some probably where 
> previously, and some are def not runnable.
>  
> So I started top down today, in order to make autocomplete.go run on dataflow 
> as well as the direct runner i changed the input in order to make it platform 
> independent instead of pointing to a local file.The reading of the source 
> from the public cloud storage went fine but it fails to run the top.Largest 
> anonymous less function (ran on id: go-job-1-1536575613531078735) failed with
>  
>  
> {code:java}
> RESP: instruction_id: "-205" error: "Invalid bundle desc: decode: bad userfn: 
> bad struct encoding: failed to decode data: decode: failed to find symbol 
> main.main.func1: main.main.func1 not found. Use runtime.RegisterFunction in 
> unit tests" register: < >
>  
> {code}
>  
> [https://github.com/apache/beam/blob/master/sdks/go/examples/complete/autocomplete/autocomplete.go#L63]
>  
> So in order to fix this I introduced the local func called lessFn and 
> registered in the init process. This though now instead when running
>  
> {code:java}
>  
> go run autocomplete.go --project fair-app-213019 --runner dataflow 
> --staging_location=gs://fair-app-213019/staging-test2 
> --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
> {code}
>  
>  
> fails with
>  
> {code:java}
> 2018/09/10 13:37:10 Running autocomplete
>  Unable to encode combiner for lifting: failed to encode custom coder: bad 
> underlying type: bad field type: bad element: unencodable type: interface 
> {}2018/09/10 13:37:10 Using running binary as worker binary: 
> '/tmp/go-build157286122/b001/exe/autocomplete'
>  2018/09/10 13:37:10 Staging worker binary: 
> /tmp/go-build157286122/b001/exe/autocomplete{code}
>  
> And I know this is when invoking the top.Largest since I've removed the piece 
> of code and then the job runs fine, could you please point me in the right 
> direction why my local func is not encoable as a interface {} and I will of 
> course happily send a PR when this is working on direct and the dataflow 
> direct so I can move on to the other examples
>  
> (All changes can be seen here) 
> [https://github.com/apache/beam/compare/master...ptomasroos:make-autocomplete-dataflowable?expand=1]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5262) JobState support for Reference Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5262?focusedWorklogId=143205=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143205
 ]

ASF GitHub Bot logged work on BEAM-5262:


Author: ASF GitHub Bot
Created on: 11/Sep/18 17:31
Start Date: 11/Sep/18 17:31
Worklog Time Spent: 10m 
  Work Description: youngoli commented on a change in pull request #6301: 
[BEAM-5262] Add Reference runner support for add state stream
URL: https://github.com/apache/beam/pull/6301#discussion_r216754306
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobService.java
 ##
 @@ -201,6 +204,36 @@ public void getState(
 responseObserver.onCompleted();
   }
 
+  @Override
+  public void getStateStream(
+  GetJobStateRequest request, StreamObserver 
responseObserver) {
+LOG.trace("{} {}", GetJobStateRequest.class.getSimpleName(), request);
+String invocationId = request.getJobId();
+try {
+  Thread.sleep(WAIT_MS);
+  Enum state = jobStates.getOrDefault(request.getJobId(), 
Enum.UNRECOGNIZED);
+  
responseObserver.onNext(GetJobStateResponse.newBuilder().setState(state).build());
+  while (Enum.RUNNING.equals(state)) {
+Thread.sleep(WAIT_MS);
+state = jobStates.getOrDefault(request.getJobId(), Enum.UNRECOGNIZED);
+  }
+  
responseObserver.onNext(GetJobStateResponse.newBuilder().setState(state).build());
 
 Review comment:
   Makes sense. And it seems that in the case of it not being in the running 
state, it's not an issue to republish the state, so there's no 
performance/correctness issues here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 143205)
Time Spent: 1h  (was: 50m)

> JobState support for Reference Runner
> -
>
> Key: BEAM-5262
> URL: https://issues.apache.org/jira/browse/BEAM-5262
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Minor
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Reference runner does not support getStateStream which is needed by portable 
> SDK



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)