[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support
[
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=341236=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341236
]
ASF GitHub Bot logged work on BEAM-7274:
Author: ASF GitHub Bot
Created on: 11/Nov/19 11:00
Start Date: 11/Nov/19 11:00
Worklog Time Spent: 10m
Work Description: alexvanboxel commented on pull request #8690:
[BEAM-7274] Implement the Protobuf schema provider
URL: https://github.com/apache/beam/pull/8690#discussion_r34465
##
File path:
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoFieldOverlay.java
##
@@ -0,0 +1,525 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.Descriptors.FieldDescriptor;
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import com.google.protobuf.Timestamp;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.FieldValueGetter;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.joda.time.Instant;
+
+/**
+ * Protobuf ProtoFieldOverlay is the interface that each implementation needs
to implement to handle
+ * a specific field types.
+ */
+@Experimental(Experimental.Kind.SCHEMAS)
+public interface ProtoFieldOverlay extends FieldValueGetter {
+
+ ValueT convertGetObject(FieldDescriptor fieldDescriptor, Object object);
+
+ /** Convert the Row field and set it on the overlayed field of the message.
*/
+ void set(Message.Builder object, ValueT value);
+
+ Object convertSetObject(FieldDescriptor fieldDescriptor, Object value);
+
+ /** Return the Beam Schema Field of this overlayed field. */
+ Schema.Field getSchemaField();
+
+ abstract class ProtoFieldOverlayBase implements
ProtoFieldOverlay {
+
+protected int number;
+
+private Schema.Field field;
+
+FieldDescriptor getFieldDescriptor(Message message) {
+ return message.getDescriptorForType().findFieldByNumber(number);
+}
+
+FieldDescriptor getFieldDescriptor(Message.Builder message) {
+ return message.getDescriptorForType().findFieldByNumber(number);
+}
+
+protected void setField(Schema.Field field) {
+ this.field = field;
+}
+
+ProtoFieldOverlayBase(ProtoSchema protoSchema, FieldDescriptor
fieldDescriptor) {
+ // this.fieldDescriptor = fieldDescriptor;
+ this.number = fieldDescriptor.getNumber();
+}
+
+@Override
+public String name() {
+ return field.getName();
+}
+
+@Override
+public Schema.Field getSchemaField() {
+ return field;
+}
+ }
+
+ /** Overlay for Protobuf primitive types. Primitive values are just passed
through. */
+ class PrimitiveOverlay extends ProtoFieldOverlayBase {
+PrimitiveOverlay(ProtoSchema protoSchema, FieldDescriptor
fieldDescriptor) {
+ // this.fieldDescriptor = fieldDescriptor;
+ super(protoSchema, fieldDescriptor);
+ setField(
+ Schema.Field.of(
+ fieldDescriptor.getName(),
+ ProtoSchema.convertType(fieldDescriptor.getType())
+ .withMetadata(protoSchema.convertOptions(fieldDescriptor;
+}
+
+@Override
+public Object get(Message message) {
+ FieldDescriptor fieldDescriptor = getFieldDescriptor(message);
+ return convertGetObject(fieldDescriptor,
message.getField(fieldDescriptor));
+}
+
+@Override
+public Object convertGetObject(FieldDescriptor fieldDescriptor, Object
object) {
+ return object;
+}
+
+@Override
+public void set(Message.Builder message, Object value) {
+ message.setField(getFieldDescriptor(message), value);
+}
+
+@Override
+public Object convertSetObject(FieldDescriptor fieldDescriptor, Object
value) {
+ return
[jira] [Updated] (BEAM-5753) Beam Dependency Update Request: org.eclipse.jetty:jetty-servlet
[ https://issues.apache.org/jira/browse/BEAM-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5753: Description: - 2018-10-15 12:13:19.256622 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-22 12:13:32.240729 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-29 12:17:15.388249 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-05 12:15:02.810121 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-12 12:15:02.741564 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-19 12:15:44.462009 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-26 12:14:43.766710 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-03 12:15:09.151091 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-10 12:17:38.451204 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-17 12:17:59.925429 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-31 15:24:18.611955 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-07 12:27:30.751791 - Please consider upgrading the dependency
[jira] [Updated] (BEAM-8002) Beam Dependency Update Request: Gradle:
[ https://issues.apache.org/jira/browse/BEAM-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8002: Description: - 2019-08-19 12:04:19.192688 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-26 12:04:17.802062 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-02 12:04:53.057461 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-09 12:03:53.298241 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-16 12:03:47.037326 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-23 12:04:12.360299 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-30 12:05:19.680040 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-10-07 12:04:48.065365 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-10-14 12:04:37.533903 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-10-21 12:05:15.993810 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1 -> 5.6.3. The latest version is 6.0-rc-1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-10-28 12:07:10.976374 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1 -> 5.6.3. The latest version is 6.0-rc-1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-11-04 12:05:57.793513 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.4 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-11-11 12:05:24.076889 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 6.0 cc: Please refer to [Beam
[jira] [Work logged] (BEAM-8605) Function display_graph() in example do not exist
[ https://issues.apache.org/jira/browse/BEAM-8605?focusedWorklogId=341226=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341226 ] ASF GitHub Bot logged work on BEAM-8605: Author: ASF GitHub Bot Created on: 11/Nov/19 10:29 Start Date: 11/Nov/19 10:29 Worklog Time Spent: 10m Work Description: RusOr10n commented on pull request #10059: [BEAM-8605] Function display_graph() in example do not exist URL: https://github.com/apache/beam/pull/10059 R: @qinyeli I have changed just comment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341226) Remaining Estimate: 0h Time Spent: 10m > Function display_graph() in example do not exist > > > Key: BEAM-8605 > URL: https://issues.apache.org/jira/browse/BEAM-8605 > Project: Beam > Issue Type: Bug > Components: runner-py-interactive, sdk-py-core >Reporter: Rustam Khalmurzaev >Priority: Trivial > Labels: easyfix > Time Spent: 10m > Remaining Estimate: 0h > > Here in code an example of using PipelineGraph class. Example in comments > contains function display_graph() which do not exist. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py#L47] > > Examples: graph = pipeline_graph.PipelineGraph(pipeline_proto) > graph.display_graph() > or > graph = pipeline_graph.PipelineGraph(pipeline) > graph.display_graph() -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support
[ https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=341243=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341243 ] ASF GitHub Bot logged work on BEAM-7274: Author: ASF GitHub Bot Created on: 11/Nov/19 11:06 Start Date: 11/Nov/19 11:06 Worklog Time Spent: 10m Work Description: alexvanboxel commented on pull request #8690: [BEAM-7274] Implement the Protobuf schema provider URL: https://github.com/apache/beam/pull/8690#discussion_r344661043 ## File path: sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchema.java ## @@ -0,0 +1,563 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.sdk.extensions.protobuf; + +import com.google.protobuf.DescriptorProtos; +import com.google.protobuf.Descriptors; +import com.google.protobuf.DynamicMessage; +import com.google.protobuf.Message; +import com.google.protobuf.UnknownFieldSet; +import java.io.IOException; +import java.io.ObjectInputStream; +import java.io.ObjectOutputStream; +import java.io.Serializable; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.math.BigInteger; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.UUID; +import java.util.stream.Collectors; +import org.apache.beam.sdk.annotations.Experimental; +import org.apache.beam.sdk.schemas.Factory; +import org.apache.beam.sdk.schemas.FieldValueGetter; +import org.apache.beam.sdk.schemas.Schema; +import org.apache.beam.sdk.schemas.SchemaCoder; +import org.apache.beam.sdk.transforms.SerializableFunction; +import org.apache.beam.sdk.values.Row; +import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap; + +/** + * ProtoSchema is a top level anchor point. It makes sure it can recreate the complete schema and + * overlay with just the Message raw type or if it's a DynamicMessage with the serialised + * Descriptor. + * + * ProtoDomain is an integral part of a ProtoSchema, it it contains all the information needed to + * iterpret and reconstruct messages. + * + * + * Protobuf oneOf fields are mapped to nullable fields and flattened into the parent row. + * Protobuf primitives are mapped to it's nullable counter part. + * Protobuf maps are mapped to nullable maps, where empty maps are mapped to the null value. + * Protobuf repeatables are mapped to nullable arrays, where empty arrays are mapped to the + * null value. + * Protobuf enums are mapped to non-nullable string values. + * Enum map to their string representation Review comment: Removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341243) Time Spent: 8.5h (was: 8h 20m) > Protobuf Beam Schema support > > > Key: BEAM-7274 > URL: https://issues.apache.org/jira/browse/BEAM-7274 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Minor > Time Spent: 8.5h > Remaining Estimate: 0h > > Add support for the new Beam Schema to the Protobuf extension. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support
[ https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=341247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341247 ] ASF GitHub Bot logged work on BEAM-7274: Author: ASF GitHub Bot Created on: 11/Nov/19 11:16 Start Date: 11/Nov/19 11:16 Worklog Time Spent: 10m Work Description: alexvanboxel commented on issue #8690: [BEAM-7274] Implement the Protobuf schema provider URL: https://github.com/apache/beam/pull/8690#issuecomment-552401829 This PR is now rebased against master.This removed ProtoDomain (and it's tests) from this PR as it was part of the DymamicMessage PR that is now part of master. I think I've incorporated most of the comment of the review in this PR as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341247) Time Spent: 9h (was: 8h 50m) > Protobuf Beam Schema support > > > Key: BEAM-7274 > URL: https://issues.apache.org/jira/browse/BEAM-7274 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Minor > Time Spent: 9h > Remaining Estimate: 0h > > Add support for the new Beam Schema to the Protobuf extension. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8585) Include path in error message in path_to_beam_jar
[ https://issues.apache.org/jira/browse/BEAM-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maximilian Michels resolved BEAM-8585. -- Fix Version/s: 2.18.0 Resolution: Fixed > Include path in error message in path_to_beam_jar > - > > Key: BEAM-8585 > URL: https://issues.apache.org/jira/browse/BEAM-8585 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Trivial > Labels: portability-flink > Fix For: 2.18.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Right now, the error message looks like this when the job server jar can't be > found: > 12:35:50 RuntimeError: Please build the server with > 12:35:50 cd > /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src; > ./gradlew runners:flink:1.9:job-server:shadowJar > I would like to know the path of the missing jar to help me debug. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8585) Include path in error message in path_to_beam_jar
[ https://issues.apache.org/jira/browse/BEAM-8585?focusedWorklogId=341255=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341255 ] ASF GitHub Bot logged work on BEAM-8585: Author: ASF GitHub Bot Created on: 11/Nov/19 11:38 Start Date: 11/Nov/19 11:38 Worklog Time Spent: 10m Work Description: mxm commented on pull request #10032: [BEAM-8585] Include path in error message in path_to_beam_jar URL: https://github.com/apache/beam/pull/10032 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341255) Time Spent: 20m (was: 10m) > Include path in error message in path_to_beam_jar > - > > Key: BEAM-8585 > URL: https://issues.apache.org/jira/browse/BEAM-8585 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Trivial > Labels: portability-flink > Fix For: 2.18.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Right now, the error message looks like this when the job server jar can't be > found: > 12:35:50 RuntimeError: Please build the server with > 12:35:50 cd > /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src; > ./gradlew runners:flink:1.9:job-server:shadowJar > I would like to know the path of the missing jar to help me debug. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8605) Function display_graph() in example do not exist
[ https://issues.apache.org/jira/browse/BEAM-8605?focusedWorklogId=341229=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341229 ] ASF GitHub Bot logged work on BEAM-8605: Author: ASF GitHub Bot Created on: 11/Nov/19 10:47 Start Date: 11/Nov/19 10:47 Worklog Time Spent: 10m Work Description: RusOr10n commented on issue #10059: [BEAM-8605] Function display_graph() in example do not exist URL: https://github.com/apache/beam/pull/10059#issuecomment-552391894 R: @youngoli This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341229) Time Spent: 20m (was: 10m) > Function display_graph() in example do not exist > > > Key: BEAM-8605 > URL: https://issues.apache.org/jira/browse/BEAM-8605 > Project: Beam > Issue Type: Bug > Components: runner-py-interactive, sdk-py-core >Reporter: Rustam Khalmurzaev >Priority: Trivial > Labels: easyfix > Time Spent: 20m > Remaining Estimate: 0h > > Here in code an example of using PipelineGraph class. Example in comments > contains function display_graph() which do not exist. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py#L47] > > Examples: graph = pipeline_graph.PipelineGraph(pipeline_proto) > graph.display_graph() > or > graph = pipeline_graph.PipelineGraph(pipeline) > graph.display_graph() -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support
[
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=341244=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341244
]
ASF GitHub Bot logged work on BEAM-7274:
Author: ASF GitHub Bot
Created on: 11/Nov/19 11:08
Start Date: 11/Nov/19 11:08
Worklog Time Spent: 10m
Work Description: alexvanboxel commented on pull request #8690:
[BEAM-7274] Implement the Protobuf schema provider
URL: https://github.com/apache/beam/pull/8690#discussion_r344661429
##
File path:
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaProvider.java
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import com.google.protobuf.DynamicMessage;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaProvider;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Schema provider for Protobuf messages. The provider is able to handle pre
compiled Message file
+ * without external help. For Dynamic Messages a Descriptor needs to be
registered up front on a
+ * specific URN.
+ *
+ * It's possible to inherit this class for a specific implementation that
communicates with an
+ * external registry that maps those URN's with Descriptors.
+ */
+@Experimental(Experimental.Kind.SCHEMAS)
+public class ProtoSchemaProvider implements SchemaProvider {
Review comment:
No, each data type has it's own Provider (eg. AvroSchemaProvider). This
class makes sure Proto can be auto mapped.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 341244)
Time Spent: 8h 40m (was: 8.5h)
> Protobuf Beam Schema support
>
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-8605) Function display_graph() in example do not exist
Rustam Khalmurzaev created BEAM-8605: Summary: Function display_graph() in example do not exist Key: BEAM-8605 URL: https://issues.apache.org/jira/browse/BEAM-8605 Project: Beam Issue Type: Bug Components: runner-py-interactive Reporter: Rustam Khalmurzaev Here in code an example of using PipelineGraph class. Example in comments contains function display_graph() which do not exist. [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py#L47] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support
[
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=341245=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341245
]
ASF GitHub Bot logged work on BEAM-7274:
Author: ASF GitHub Bot
Created on: 11/Nov/19 11:11
Start Date: 11/Nov/19 11:11
Worklog Time Spent: 10m
Work Description: alexvanboxel commented on pull request #8690:
[BEAM-7274] Implement the Protobuf schema provider
URL: https://github.com/apache/beam/pull/8690#discussion_r344662602
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
##
@@ -554,6 +555,12 @@ public Builder withFieldValueGetters(
return this;
}
+/** The FieldValueGetters will handle the conversion for Arrays, Maps and
Rows. */
+public Builder withFieldValueGettersHandleCollections(boolean
collectionHandledByGetter) {
+ this.collectionHandledByGetter = collectionHandledByGetter;
+ return this;
+}
Review comment:
I've refactored from your input: I've created a
RowWithGettersCachedCollection that inherits for RowWithGetters. This cached is
the default.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 341245)
Time Spent: 8h 50m (was: 8h 40m)
> Protobuf Beam Schema support
>
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Time Spent: 8h 50m
> Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-8606) Beam Dependency Update Request: pytest
Beam JIRA Bot created BEAM-8606: --- Summary: Beam Dependency Update Request: pytest Key: BEAM-8606 URL: https://issues.apache.org/jira/browse/BEAM-8606 Project: Beam Issue Type: Bug Components: dependencies Reporter: Beam JIRA Bot - 2019-11-11 12:03:32.745505 - Please consider upgrading the dependency pytest. The current version is 4.6.6. The latest version is 5.2.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6089) Beam Dependency Update Request: oauth2client
[ https://issues.apache.org/jira/browse/BEAM-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6089: Description: - 2018-11-19 12:11:53.801885 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-26 12:10:31.359164 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-03 12:11:18.194090 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-10 12:13:40.021791 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-17 12:12:09.88 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-31 15:20:14.935936 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-07 12:23:14.664558 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-14 12:12:14.081917 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-21 12:18:38.928775 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-28 12:10:22.371989 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-02-04 12:11:00.277439 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-02-11 12:11:25.422782 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-02-18 12:21:52.210498 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3
[jira] [Updated] (BEAM-5748) Beam Dependency Update Request: org.conscrypt:conscrypt-openjdk
[ https://issues.apache.org/jira/browse/BEAM-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5748: Description: - 2018-10-15 12:11:29.496210 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-22 12:11:12.117107 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-29 12:13:29.155766 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-05 12:12:12.225263 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-12 12:12:03.483941 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-19 12:12:49.618981 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-26 12:11:54.540783 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-03 12:12:15.486522 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-10 12:14:35.731081 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-17 12:15:07.608265 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-31 15:21:22.979404 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-07 12:24:30.102231 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not
[jira] [Updated] (BEAM-5752) Beam Dependency Update Request: org.eclipse.jetty:jetty-server
[ https://issues.apache.org/jira/browse/BEAM-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5752: Description: - 2018-10-15 12:13:17.575546 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-22 12:13:26.908573 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-29 12:17:10.104039 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-05 12:14:56.891103 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-12 12:14:57.086704 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-19 12:15:38.454968 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-26 12:14:38.329092 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-03 12:15:03.778648 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-10 12:17:33.026807 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-17 12:17:54.249911 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-31 15:24:13.071360 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-07 12:27:25.376755 - Please consider upgrading the dependency
[jira] [Updated] (BEAM-6951) Beam Dependency Update Request: com.github.spotbugs:spotbugs-annotations
[ https://issues.apache.org/jira/browse/BEAM-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6951: Description: - 2019-04-01 12:15:05.460427 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-04-08 12:15:37.305259 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-04-15 12:35:52.817108 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-04-22 12:13:25.261372 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-05-20 16:39:18.034675 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-05-20 16:54:09.180503 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-05-20 17:37:40.326607 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-05-27 12:06:48.236836 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-03 12:04:13.166255 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-10 12:04:08.241217 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-17 12:33:32.545061 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-24 12:04:02.498856 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current
[jira] [Updated] (BEAM-6645) Beam Dependency Update Request: com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
[ https://issues.apache.org/jira/browse/BEAM-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6645: Description: - 2019-02-11 12:12:17.710870 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.20.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-02-18 12:22:38.110512 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.20.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-02-25 12:12:13.220144 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.20.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-03-04 12:13:34.975335 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-03-11 12:13:20.550078 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-03-25 04:18:13.860760 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-04-01 12:09:02.686241 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-04-08 12:11:15.303458 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-04-15 12:30:47.279374 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-04-22 12:08:36.647085 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-05-20 16:38:23.480410 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide
[jira] [Created] (BEAM-8607) Beam Dependency Update Request: tenacity
Beam JIRA Bot created BEAM-8607: --- Summary: Beam Dependency Update Request: tenacity Key: BEAM-8607 URL: https://issues.apache.org/jira/browse/BEAM-8607 Project: Beam Issue Type: Bug Components: dependencies Reporter: Beam JIRA Bot - 2019-11-11 12:03:53.472802 - Please consider upgrading the dependency tenacity. The current version is 5.1.5. The latest version is 6.0.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5750) Beam Dependency Update Request: javax.servlet:javax.servlet-api
[ https://issues.apache.org/jira/browse/BEAM-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5750: Description: - 2018-10-15 12:13:14.253682 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-22 12:13:20.677351 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-29 12:17:03.954722 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-05 12:14:50.209888 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-12 12:14:50.401239 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-19 12:15:31.652290 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-26 12:14:29.396363 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-03 12:14:52.265627 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-10 12:17:21.331309 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-17 12:17:42.365110 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-31 15:24:01.296226 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-07 12:27:13.634593 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not
[jira] [Updated] (BEAM-7369) Beam Dependency Update Request: mock
[ https://issues.apache.org/jira/browse/BEAM-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7369: Description: - 2019-05-20 16:37:42.331317 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-17 12:32:01.319061 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-24 12:02:34.500201 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-01 12:03:46.078139 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-08 12:02:33.351570 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-15 12:02:47.637435 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-22 12:03:10.835914 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-29 12:04:50.359555 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-05 12:02:38.989136 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-12 12:03:38.282286 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-19 12:02:30.405376 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-26 12:02:24.272597 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-02 12:03:07.834005 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide
[jira] [Updated] (BEAM-7370) Beam Dependency Update Request: Sphinx
[ https://issues.apache.org/jira/browse/BEAM-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7370: Description: - 2019-05-20 16:38:07.937770 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-17 12:32:27.855338 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-24 12:02:59.052884 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-01 12:04:13.113613 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-08 12:03:15.091005 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-15 12:03:09.406918 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-22 12:03:31.157859 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-29 12:05:13.023604 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-05 12:03:03.242767 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-12 12:04:01.647619 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-19 12:02:52.342008 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.2.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-26 12:02:44.918642 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.2.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-02 12:03:28.319578 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.2.0 cc: Please refer to [Beam Dependency Guide
[jira] [Updated] (BEAM-7792) Beam Dependency Update Request: com.github.spotbugs:spotbugs
[ https://issues.apache.org/jira/browse/BEAM-7792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7792: Description: - 2019-07-22 12:15:41.216793 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.10. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-29 12:06:22.055418 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.10. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-05 12:03:56.940731 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.10. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-12 12:05:04.611973 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.10. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-19 12:03:13.035336 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-26 12:02:56.981076 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-02 12:03:52.780365 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-09 12:02:54.192199 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-16 12:02:39.812475 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-23 12:03:12.396932 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta4 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-30 12:04:13.122038 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta4 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-10-07 12:03:31.042633 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta4 cc: Please refer to [Beam Dependency Guide
[jira] [Work logged] (BEAM-8090) Add test for Self Setting Timers close to GC TImer
[ https://issues.apache.org/jira/browse/BEAM-8090?focusedWorklogId=341195=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341195 ] ASF GitHub Bot logged work on BEAM-8090: Author: ASF GitHub Bot Created on: 11/Nov/19 08:36 Start Date: 11/Nov/19 08:36 Worklog Time Spent: 10m Work Description: stale[bot] commented on issue #9429: [BEAM-8090] Create test for bug around self setting timers URL: https://github.com/apache/beam/pull/9429#issuecomment-552344510 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the d...@beam.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341195) Time Spent: 0.5h (was: 20m) > Add test for Self Setting Timers close to GC TImer > -- > > Key: BEAM-8090 > URL: https://issues.apache.org/jira/browse/BEAM-8090 > Project: Beam > Issue Type: Test > Components: sdk-java-core >Reporter: Reza ardeshir rokni >Assignee: Reza ardeshir rokni >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Create test for bug around self setting timers and interaction with GC > Timers. Create timer that is a milli behind GC so they likely show up in the > same bundle. > Scenario this is expected to catch: > If a user timer and the GC timer show up in the same bundle, and the user > timer resets itself, and the reset timer is not called (because it had to be > committed and then restored), then you have your failure. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8605) Function display_graph() in example do not exist
[ https://issues.apache.org/jira/browse/BEAM-8605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rustam Khalmurzaev updated BEAM-8605: - Component/s: sdk-py-core > Function display_graph() in example do not exist > > > Key: BEAM-8605 > URL: https://issues.apache.org/jira/browse/BEAM-8605 > Project: Beam > Issue Type: Bug > Components: runner-py-interactive, sdk-py-core >Reporter: Rustam Khalmurzaev >Priority: Trivial > Labels: easyfix > > Here in code an example of using PipelineGraph class. Example in comments > contains function display_graph() which do not exist. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py#L47] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8605) Function display_graph() in example do not exist
[ https://issues.apache.org/jira/browse/BEAM-8605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rustam Khalmurzaev updated BEAM-8605: - Description: Here in code an example of using PipelineGraph class. Example in comments contains function display_graph() which do not exist. [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py#L47] Examples: graph = pipeline_graph.PipelineGraph(pipeline_proto) graph.display_graph() or graph = pipeline_graph.PipelineGraph(pipeline) graph.display_graph() was: Here in code an example of using PipelineGraph class. Example in comments contains function display_graph() which do not exist. [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py#L47] > Function display_graph() in example do not exist > > > Key: BEAM-8605 > URL: https://issues.apache.org/jira/browse/BEAM-8605 > Project: Beam > Issue Type: Bug > Components: runner-py-interactive, sdk-py-core >Reporter: Rustam Khalmurzaev >Priority: Trivial > Labels: easyfix > > Here in code an example of using PipelineGraph class. Example in comments > contains function display_graph() which do not exist. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py#L47] > > Examples: graph = pipeline_graph.PipelineGraph(pipeline_proto) > graph.display_graph() > or > graph = pipeline_graph.PipelineGraph(pipeline) > graph.display_graph() -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8605) Function display_graph() in example do not exist
[ https://issues.apache.org/jira/browse/BEAM-8605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971415#comment-16971415 ] Rustam Khalmurzaev commented on BEAM-8605: -- I'll take this issue. > Function display_graph() in example do not exist > > > Key: BEAM-8605 > URL: https://issues.apache.org/jira/browse/BEAM-8605 > Project: Beam > Issue Type: Bug > Components: runner-py-interactive, sdk-py-core >Reporter: Rustam Khalmurzaev >Priority: Trivial > Labels: easyfix > > Here in code an example of using PipelineGraph class. Example in comments > contains function display_graph() which do not exist. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py#L47] > > Examples: graph = pipeline_graph.PipelineGraph(pipeline_proto) > graph.display_graph() > or > graph = pipeline_graph.PipelineGraph(pipeline) > graph.display_graph() -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support
[ https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=341238=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341238 ] ASF GitHub Bot logged work on BEAM-7274: Author: ASF GitHub Bot Created on: 11/Nov/19 11:04 Start Date: 11/Nov/19 11:04 Worklog Time Spent: 10m Work Description: alexvanboxel commented on pull request #8690: [BEAM-7274] Implement the Protobuf schema provider URL: https://github.com/apache/beam/pull/8690#discussion_r344660088 ## File path: sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchema.java ## @@ -0,0 +1,563 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.sdk.extensions.protobuf; + +import com.google.protobuf.DescriptorProtos; +import com.google.protobuf.Descriptors; +import com.google.protobuf.DynamicMessage; +import com.google.protobuf.Message; +import com.google.protobuf.UnknownFieldSet; +import java.io.IOException; +import java.io.ObjectInputStream; +import java.io.ObjectOutputStream; +import java.io.Serializable; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.math.BigInteger; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.UUID; +import java.util.stream.Collectors; +import org.apache.beam.sdk.annotations.Experimental; +import org.apache.beam.sdk.schemas.Factory; +import org.apache.beam.sdk.schemas.FieldValueGetter; +import org.apache.beam.sdk.schemas.Schema; +import org.apache.beam.sdk.schemas.SchemaCoder; +import org.apache.beam.sdk.transforms.SerializableFunction; +import org.apache.beam.sdk.values.Row; +import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap; + +/** + * ProtoSchema is a top level anchor point. It makes sure it can recreate the complete schema and + * overlay with just the Message raw type or if it's a DynamicMessage with the serialised + * Descriptor. + * + * ProtoDomain is an integral part of a ProtoSchema, it it contains all the information needed to + * iterpret and reconstruct messages. + * + * + * Protobuf oneOf fields are mapped to nullable fields and flattened into the parent row. + * Protobuf primitives are mapped to it's nullable counter part. + * Protobuf maps are mapped to nullable maps, where empty maps are mapped to the null value. + * Protobuf repeatables are mapped to nullable arrays, where empty arrays are mapped to the + * null value. Review comment: Business wise this makes more sense. As proto doesn't distinguish between empty map/array and not set. It's also more performant as if the default would be an empty map/array it would result in a bigger Row. It would be an option to make it later configurable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341238) Time Spent: 8h 10m (was: 8h) > Protobuf Beam Schema support > > > Key: BEAM-7274 > URL: https://issues.apache.org/jira/browse/BEAM-7274 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Minor > Time Spent: 8h 10m > Remaining Estimate: 0h > > Add support for the new Beam Schema to the Protobuf extension. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support
[ https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=341240=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341240 ] ASF GitHub Bot logged work on BEAM-7274: Author: ASF GitHub Bot Created on: 11/Nov/19 11:04 Start Date: 11/Nov/19 11:04 Worklog Time Spent: 10m Work Description: alexvanboxel commented on pull request #8690: [BEAM-7274] Implement the Protobuf schema provider URL: https://github.com/apache/beam/pull/8690#discussion_r344660088 ## File path: sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchema.java ## @@ -0,0 +1,563 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.sdk.extensions.protobuf; + +import com.google.protobuf.DescriptorProtos; +import com.google.protobuf.Descriptors; +import com.google.protobuf.DynamicMessage; +import com.google.protobuf.Message; +import com.google.protobuf.UnknownFieldSet; +import java.io.IOException; +import java.io.ObjectInputStream; +import java.io.ObjectOutputStream; +import java.io.Serializable; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.math.BigInteger; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.UUID; +import java.util.stream.Collectors; +import org.apache.beam.sdk.annotations.Experimental; +import org.apache.beam.sdk.schemas.Factory; +import org.apache.beam.sdk.schemas.FieldValueGetter; +import org.apache.beam.sdk.schemas.Schema; +import org.apache.beam.sdk.schemas.SchemaCoder; +import org.apache.beam.sdk.transforms.SerializableFunction; +import org.apache.beam.sdk.values.Row; +import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap; + +/** + * ProtoSchema is a top level anchor point. It makes sure it can recreate the complete schema and + * overlay with just the Message raw type or if it's a DynamicMessage with the serialised + * Descriptor. + * + * ProtoDomain is an integral part of a ProtoSchema, it it contains all the information needed to + * iterpret and reconstruct messages. + * + * + * Protobuf oneOf fields are mapped to nullable fields and flattened into the parent row. + * Protobuf primitives are mapped to it's nullable counter part. + * Protobuf maps are mapped to nullable maps, where empty maps are mapped to the null value. + * Protobuf repeatables are mapped to nullable arrays, where empty arrays are mapped to the + * null value. Review comment: Business wise this makes more sense. As proto doesn't distinguish between empty map/array and not set. It's also more performant as if the default would be an empty map/array it would result in a bigger Row. It would be an option to make it later configurable. Kept it as is. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341240) Time Spent: 8h 20m (was: 8h 10m) > Protobuf Beam Schema support > > > Key: BEAM-7274 > URL: https://issues.apache.org/jira/browse/BEAM-7274 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Minor > Time Spent: 8h 20m > Remaining Estimate: 0h > > Add support for the new Beam Schema to the Protobuf extension. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8442) Unify bundle register in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8442?focusedWorklogId=341242=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341242 ] ASF GitHub Bot logged work on BEAM-8442: Author: ASF GitHub Bot Created on: 11/Nov/19 11:04 Start Date: 11/Nov/19 11:04 Worklog Time Spent: 10m Work Description: mxm commented on issue #10004: [BEAM-8442] Unify bundle register in Python SDK harness URL: https://github.com/apache/beam/pull/10004#issuecomment-552397946 Run Java PortabilityApi PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341242) Time Spent: 3h 40m (was: 3.5h) > Unify bundle register in Python SDK harness > --- > > Key: BEAM-8442 > URL: https://issues.apache.org/jira/browse/BEAM-8442 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > There are two methods for bundle register in Python SDK harness: > `SdkHarness._request_register` and `SdkWorker.register.` It should be unfied. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8557) Clean up useless null check.
[
https://issues.apache.org/jira/browse/BEAM-8557?focusedWorklogId=341296=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341296
]
ASF GitHub Bot logged work on BEAM-8557:
Author: ASF GitHub Bot
Created on: 11/Nov/19 13:23
Start Date: 11/Nov/19 13:23
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on issue #9991: [BEAM-8557]Add
log for the dropped unknown response
URL: https://github.com/apache/beam/pull/9991#issuecomment-552443631
R: @kennknowles I have updated the PR, is that make sense to you :) Welcome
any feedback.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 341296)
Time Spent: 1h 50m (was: 1h 40m)
> Clean up useless null check.
>
>
> Key: BEAM-8557
> URL: https://issues.apache.org/jira/browse/BEAM-8557
> Project: Beam
> Issue Type: Sub-task
> Components: runner-core, sdk-java-harness
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> I think we do not need null check here:
> [https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
> Because before the the `onNext` call, the `Future` already put into the queue
> in `handle` method.
>
> I found the test as follows:
> {code:java}
> @Test
> public void testUnknownResponseIgnored() throws Exception{code}
> I do not know why we need test this case? I think it would be better if we
> throw the Exception for an UnknownResponse, otherwise, this may hidden a
> potential bug.
> Please correct me if there anything I misunderstand @kennknowles
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8157) Key encoding for state requests is not consistent across SDKs
[ https://issues.apache.org/jira/browse/BEAM-8157?focusedWorklogId=341341=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341341 ] ASF GitHub Bot logged work on BEAM-8157: Author: ASF GitHub Bot Created on: 11/Nov/19 15:21 Start Date: 11/Nov/19 15:21 Worklog Time Spent: 10m Work Description: mxm commented on issue #9997: [BEAM-8157] Fix key encoding issues for state requests with unknown coders / Improve debugging and testing URL: https://github.com/apache/beam/pull/9997#issuecomment-552486933 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341341) Time Spent: 10h 40m (was: 10.5h) > Key encoding for state requests is not consistent across SDKs > - > > Key: BEAM-8157 > URL: https://issues.apache.org/jira/browse/BEAM-8157 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.13.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Critical > Fix For: 2.17.0 > > Time Spent: 10h 40m > Remaining Estimate: 0h > > The Flink runner requires the internal key to be encoded without a length > prefix (OUTER context). The user state request handler exposes a serialized > version of the key to the Runner. This key is encoded with the NESTED context > which may add a length prefix. We need to convert it to OUTER context to > match the Flink runner's key encoding. > So far this has not caused the Flink Runner to behave incorrectly. However, > with the upcoming support for Flink 1.9, the state backend will not accept > requests for keys not part of any key group/partition of the operator. This > is very likely to happen with the encoding not being consistent. > **NOTE** This is only applicable to the Java SDK, as the Python SDK uses > OUTER encoding for the key in state requests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8568) Local file system does not match relative path with wildcards
[
https://issues.apache.org/jira/browse/BEAM-8568?focusedWorklogId=341327=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341327
]
ASF GitHub Bot logged work on BEAM-8568:
Author: ASF GitHub Bot
Created on: 11/Nov/19 14:54
Start Date: 11/Nov/19 14:54
Worklog Time Spent: 10m
Work Description: dmvk commented on issue #10028: [BEAM-8568] Fixed
problem that LocalFileSystem no longer supports wil…
URL: https://github.com/apache/beam/pull/10028#issuecomment-552477142
Run Java_Examples_Dataflow PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 341327)
Time Spent: 1.5h (was: 1h 20m)
> Local file system does not match relative path with wildcards
> -
>
> Key: BEAM-8568
> URL: https://issues.apache.org/jira/browse/BEAM-8568
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 2.16.0
>Reporter: Ondrej Cerny
>Priority: Major
> Fix For: 2.17.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> CWD structure:
> {code}
> src/test/resources/input/sometestfile.txt
> {code}
>
> Code:
> {code:java}
> input
> .apply(Create.of("src/test/resources/input/*))
> .apply(FileIO.matchAll())
> .apply(FileIO.readMatches())
> {code}
> The code above doesn't match any file starting Beam 2.16.0. The regression
> has been introduced in BEAM-7854.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8608) Chain DoFns in Flink batch runner when possible.
[ https://issues.apache.org/jira/browse/BEAM-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971687#comment-16971687 ] David Moravek commented on BEAM-8608: - Needed for optimal finishBundle implementation. > Chain DoFns in Flink batch runner when possible. > > > Key: BEAM-8608 > URL: https://issues.apache.org/jira/browse/BEAM-8608 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Affects Versions: 2.16.0 >Reporter: David Moravek >Assignee: David Moravek >Priority: Major > Attachments: Screen Shot 2019-11-07 at 10.35.07.png, Screen Shot > 2019-11-07 at 10.35.22.png > > > Right now, in Batch runner, DoFn is executed using MapPartition operator > (_FlinkDoFnFunction_), which doesn't have chained driver implementation. > We need to reimplement DoFnFunction with FlatMap to allow chaining. > Attached is the execution graph for the same pipeline, before and after the > patch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-6673) BigQueryIO.Read should automatically produce schemas
[
https://issues.apache.org/jira/browse/BEAM-6673?focusedWorklogId=341362=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341362
]
ASF GitHub Bot logged work on BEAM-6673:
Author: ASF GitHub Bot
Created on: 11/Nov/19 16:28
Start Date: 11/Nov/19 16:28
Worklog Time Spent: 10m
Work Description: hadpe-nitin commented on issue #8620: [BEAM-6673] Add
schema support to BigQuery reads
URL: https://github.com/apache/beam/pull/8620#issuecomment-552514703
**Converting BigQuery type NUMERIC to Beam type is unsupported.**
Can you please help me understand if we can add NUMERIC Type for schema
support to BigQuery table reads.
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
@ line 165...
> `switch (typeName) {`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 341362)
Time Spent: 2.5h (was: 2h 20m)
> BigQueryIO.Read should automatically produce schemas
>
>
> Key: BEAM-6673
> URL: https://issues.apache.org/jira/browse/BEAM-6673
> Project: Beam
> Issue Type: Sub-task
> Components: io-java-gcp
>Reporter: Reuven Lax
>Assignee: Charith Ellawala
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> The output PCollections should contain
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8608) Chain DoFns in Flink batch runner when possible.
[ https://issues.apache.org/jira/browse/BEAM-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Moravek updated BEAM-8608: Attachment: Screen Shot 2019-11-07 at 10.35.22.png > Chain DoFns in Flink batch runner when possible. > > > Key: BEAM-8608 > URL: https://issues.apache.org/jira/browse/BEAM-8608 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Affects Versions: 2.16.0 >Reporter: David Moravek >Assignee: David Moravek >Priority: Major > Attachments: Screen Shot 2019-11-07 at 10.35.07.png, Screen Shot > 2019-11-07 at 10.35.22.png > > > Right now, in Batch runner, DoFn is executed using MapPartition operator > (_FlinkDoFnFunction_), which doesn't have chained driver implementation. > We need to reimplement DoFnFunction with FlatMap to allow chaining. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8608) Chain DoFns in Flink batch runner when possible.
[ https://issues.apache.org/jira/browse/BEAM-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Moravek updated BEAM-8608: Attachment: Screen Shot 2019-11-07 at 10.35.07.png > Chain DoFns in Flink batch runner when possible. > > > Key: BEAM-8608 > URL: https://issues.apache.org/jira/browse/BEAM-8608 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Affects Versions: 2.16.0 >Reporter: David Moravek >Assignee: David Moravek >Priority: Major > Attachments: Screen Shot 2019-11-07 at 10.35.07.png, Screen Shot > 2019-11-07 at 10.35.22.png > > > Right now, in Batch runner, DoFn is executed using MapPartition operator > (_FlinkDoFnFunction_), which doesn't have chained driver implementation. > We need to reimplement DoFnFunction with FlatMap to allow chaining. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8608) Chain DoFns in Flink batch runner when possible.
David Moravek created BEAM-8608: --- Summary: Chain DoFns in Flink batch runner when possible. Key: BEAM-8608 URL: https://issues.apache.org/jira/browse/BEAM-8608 Project: Beam Issue Type: Improvement Components: runner-flink Affects Versions: 2.16.0 Reporter: David Moravek Assignee: David Moravek Right now, in Batch runner, DoFn is executed using MapPartition operator (_FlinkDoFnFunction_), which doesn't have chained driver implementation. We need to reimplement DoFnFunction with FlatMap to allow chaining. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8151) Allow the Python SDK to use many many threads
[ https://issues.apache.org/jira/browse/BEAM-8151?focusedWorklogId=341328=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341328 ] ASF GitHub Bot logged work on BEAM-8151: Author: ASF GitHub Bot Created on: 11/Nov/19 15:03 Start Date: 11/Nov/19 15:03 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #9477: [BEAM-8151, BEAM-7848] Up the max number of threads inside the SDK harness to a default of 10k URL: https://github.com/apache/beam/pull/9477#issuecomment-552480610 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341328) Time Spent: 8h 10m (was: 8h) > Allow the Python SDK to use many many threads > - > > Key: BEAM-8151 > URL: https://issues.apache.org/jira/browse/BEAM-8151 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core, sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Time Spent: 8h 10m > Remaining Estimate: 0h > > We need to use a thread pool which shrinks the number of active threads when > they are not being used. > > This is to prevent any stuckness issues related to a runner scheduling more > work items then there are "work" threads inside the SDK harness. > > By default the control plane should have all "requests" being processed in > parallel and the runner is responsible for not overloading the SDK with too > much work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8157) Key encoding for state requests is not consistent across SDKs
[ https://issues.apache.org/jira/browse/BEAM-8157?focusedWorklogId=341339=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341339 ] ASF GitHub Bot logged work on BEAM-8157: Author: ASF GitHub Bot Created on: 11/Nov/19 15:19 Start Date: 11/Nov/19 15:19 Worklog Time Spent: 10m Work Description: mxm commented on issue #9997: [BEAM-8157] Fix key encoding issues for state requests with unknown coders / Improve debugging and testing URL: https://github.com/apache/beam/pull/9997#issuecomment-552486933 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341339) Time Spent: 10.5h (was: 10h 20m) > Key encoding for state requests is not consistent across SDKs > - > > Key: BEAM-8157 > URL: https://issues.apache.org/jira/browse/BEAM-8157 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.13.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Critical > Fix For: 2.17.0 > > Time Spent: 10.5h > Remaining Estimate: 0h > > The Flink runner requires the internal key to be encoded without a length > prefix (OUTER context). The user state request handler exposes a serialized > version of the key to the Runner. This key is encoded with the NESTED context > which may add a length prefix. We need to convert it to OUTER context to > match the Flink runner's key encoding. > So far this has not caused the Flink Runner to behave incorrectly. However, > with the upcoming support for Flink 1.9, the state backend will not accept > requests for keys not part of any key group/partition of the operator. This > is very likely to happen with the encoding not being consistent. > **NOTE** This is only applicable to the Java SDK, as the Python SDK uses > OUTER encoding for the key in state requests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8553) Add a more detailed cross-language transforms roadmap
[ https://issues.apache.org/jira/browse/BEAM-8553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath resolved BEAM-8553. - Fix Version/s: Not applicable Resolution: Fixed > Add a more detailed cross-language transforms roadmap > - > > Key: BEAM-8553 > URL: https://issues.apache.org/jira/browse/BEAM-8553 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > Fix For: Not applicable > > Time Spent: 50m > Remaining Estimate: 0h > > We have a basic description at following location but this has to be expanded > to detail currently completed tasks and ongoing and future efforts related to > Beam and runners (Flink and Dataflow primarily currently). > [https://beam.apache.org/roadmap/connectors-multi-sdk/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8539) Clearly define the valid job state transitions
[ https://issues.apache.org/jira/browse/BEAM-8539?focusedWorklogId=341365=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341365 ] ASF GitHub Bot logged work on BEAM-8539: Author: ASF GitHub Bot Created on: 11/Nov/19 16:37 Start Date: 11/Nov/19 16:37 Worklog Time Spent: 10m Work Description: chadrik commented on issue #9965: [BEAM-8539] Make job state transitions in python-based runners consistent with java-based runners URL: https://github.com/apache/beam/pull/9965#issuecomment-552518135 Review notes have been addressed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341365) Time Spent: 5h 40m (was: 5.5h) > Clearly define the valid job state transitions > -- > > Key: BEAM-8539 > URL: https://issues.apache.org/jira/browse/BEAM-8539 > Project: Beam > Issue Type: Improvement > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Chad Dombrova >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > > The Beam job state transitions are ill-defined, which is big problem for > anything that relies on the values coming from JobAPI.GetStateStream. > I was hoping to find something like a state transition diagram in the docs so > that I could determine the start state, the terminal states, and the valid > transitions, but I could not find this. The code reveals that the SDKs differ > on the fundamentals: > Java InMemoryJobService: > * start state: *STOPPED* > * run - about to submit to executor: STARTING > * run - actually running on executor: RUNNING > * terminal states: DONE, FAILED, CANCELLED, DRAINED > Python AbstractJobServiceServicer / LocalJobServicer: > * start state: STARTING > * terminal states: DONE, FAILED, CANCELLED, *STOPPED* > I think it would be good to make python work like Java, so that there is a > difference in state between a job that has been prepared and one that has > additionally been run. > It's hard to tell how far this problem has spread within the various runners. > I think a simple thing that can be done to help standardize behavior is to > implement the terminal states as an enum in the beam_job_api.proto, or create > a utility function in each language for checking if a state is terminal, so > that it's not left up to each runner to reimplement this logic. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8442) Unify bundle register in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8442?focusedWorklogId=341320=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341320 ] ASF GitHub Bot logged work on BEAM-8442: Author: ASF GitHub Bot Created on: 11/Nov/19 14:39 Start Date: 11/Nov/19 14:39 Worklog Time Spent: 10m Work Description: mxm commented on issue #10004: [BEAM-8442] Unify bundle register in Python SDK harness URL: https://github.com/apache/beam/pull/10004#issuecomment-552471469 Thanks, failure looks unrelated. Test is passing in the second run. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341320) Time Spent: 4h 20m (was: 4h 10m) > Unify bundle register in Python SDK harness > --- > > Key: BEAM-8442 > URL: https://issues.apache.org/jira/browse/BEAM-8442 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > There are two methods for bundle register in Python SDK harness: > `SdkHarness._request_register` and `SdkWorker.register.` It should be unfied. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8442) Unify bundle register in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maximilian Michels resolved BEAM-8442. -- Fix Version/s: 2.18.0 Resolution: Fixed > Unify bundle register in Python SDK harness > --- > > Key: BEAM-8442 > URL: https://issues.apache.org/jira/browse/BEAM-8442 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.18.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > There are two methods for bundle register in Python SDK harness: > `SdkHarness._request_register` and `SdkWorker.register.` It should be unfied. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8442) Unify bundle register in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8442?focusedWorklogId=341321=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341321 ] ASF GitHub Bot logged work on BEAM-8442: Author: ASF GitHub Bot Created on: 11/Nov/19 14:39 Start Date: 11/Nov/19 14:39 Worklog Time Spent: 10m Work Description: mxm commented on pull request #10004: [BEAM-8442] Unify bundle register in Python SDK harness URL: https://github.com/apache/beam/pull/10004 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341321) Time Spent: 4.5h (was: 4h 20m) > Unify bundle register in Python SDK harness > --- > > Key: BEAM-8442 > URL: https://issues.apache.org/jira/browse/BEAM-8442 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > > There are two methods for bundle register in Python SDK harness: > `SdkHarness._request_register` and `SdkWorker.register.` It should be unfied. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8605) Function display_graph() in example do not exist
[ https://issues.apache.org/jira/browse/BEAM-8605?focusedWorklogId=341364=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341364 ] ASF GitHub Bot logged work on BEAM-8605: Author: ASF GitHub Bot Created on: 11/Nov/19 16:35 Start Date: 11/Nov/19 16:35 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10059: [BEAM-8605] Function display_graph() in example do not exist URL: https://github.com/apache/beam/pull/10059#issuecomment-552517500 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341364) Time Spent: 0.5h (was: 20m) > Function display_graph() in example do not exist > > > Key: BEAM-8605 > URL: https://issues.apache.org/jira/browse/BEAM-8605 > Project: Beam > Issue Type: Bug > Components: runner-py-interactive, sdk-py-core >Reporter: Rustam Khalmurzaev >Priority: Trivial > Labels: easyfix > Time Spent: 0.5h > Remaining Estimate: 0h > > Here in code an example of using PipelineGraph class. Example in comments > contains function display_graph() which do not exist. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py#L47] > > Examples: graph = pipeline_graph.PipelineGraph(pipeline_proto) > graph.display_graph() > or > graph = pipeline_graph.PipelineGraph(pipeline) > graph.display_graph() -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5600) Splitting for SplittableDoFn should be exposed within runner shared libraries
[ https://issues.apache.org/jira/browse/BEAM-5600?focusedWorklogId=341359=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341359 ] ASF GitHub Bot logged work on BEAM-5600: Author: ASF GitHub Bot Created on: 11/Nov/19 16:26 Start Date: 11/Nov/19 16:26 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10045: [BEAM-5600, BEAM-2939] Add SplittableParDo expansion logic to runner's core. URL: https://github.com/apache/beam/pull/10045#issuecomment-552513917 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341359) Time Spent: 20m (was: 10m) > Splitting for SplittableDoFn should be exposed within runner shared libraries > - > > Key: BEAM-5600 > URL: https://issues.apache.org/jira/browse/BEAM-5600 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Scott Wegner >Priority: Major > Labels: portability > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5600) Splitting for SplittableDoFn should be exposed within runner shared libraries
[ https://issues.apache.org/jira/browse/BEAM-5600?focusedWorklogId=341360=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341360 ] ASF GitHub Bot logged work on BEAM-5600: Author: ASF GitHub Bot Created on: 11/Nov/19 16:26 Start Date: 11/Nov/19 16:26 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10045: [BEAM-5600, BEAM-2939] Add SplittableParDo expansion logic to runner's core. URL: https://github.com/apache/beam/pull/10045#issuecomment-552513983 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341360) Time Spent: 0.5h (was: 20m) > Splitting for SplittableDoFn should be exposed within runner shared libraries > - > > Key: BEAM-5600 > URL: https://issues.apache.org/jira/browse/BEAM-5600 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Scott Wegner >Priority: Major > Labels: portability > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8442) Unify bundle register in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8442?focusedWorklogId=341294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341294 ] ASF GitHub Bot logged work on BEAM-8442: Author: ASF GitHub Bot Created on: 11/Nov/19 13:19 Start Date: 11/Nov/19 13:19 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10004: [BEAM-8442] Unify bundle register in Python SDK harness URL: https://github.com/apache/beam/pull/10004#issuecomment-552442004 @mxm Thanks a lot for triggering the tests. The test failure seems because of quota issues. I would like to re-trigger the test to see if it still happens. Detail can be found here: https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi_PR/47/testReport/org.apache.beam.examples.cookbook/BigQueryTornadoesIT/testE2EBigQueryTornadoesWithExport/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341294) Time Spent: 4h 10m (was: 4h) > Unify bundle register in Python SDK harness > --- > > Key: BEAM-8442 > URL: https://issues.apache.org/jira/browse/BEAM-8442 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > There are two methods for bundle register in Python SDK harness: > `SdkHarness._request_register` and `SdkWorker.register.` It should be unfied. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8442) Unify bundle register in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8442?focusedWorklogId=341292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341292 ] ASF GitHub Bot logged work on BEAM-8442: Author: ASF GitHub Bot Created on: 11/Nov/19 13:18 Start Date: 11/Nov/19 13:18 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10004: [BEAM-8442] Unify bundle register in Python SDK harness URL: https://github.com/apache/beam/pull/10004#issuecomment-552441937 Run Java PortabilityApi PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341292) Time Spent: 3h 50m (was: 3h 40m) > Unify bundle register in Python SDK harness > --- > > Key: BEAM-8442 > URL: https://issues.apache.org/jira/browse/BEAM-8442 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > There are two methods for bundle register in Python SDK harness: > `SdkHarness._request_register` and `SdkWorker.register.` It should be unfied. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8442) Unify bundle register in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8442?focusedWorklogId=341293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341293 ] ASF GitHub Bot logged work on BEAM-8442: Author: ASF GitHub Bot Created on: 11/Nov/19 13:18 Start Date: 11/Nov/19 13:18 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10004: [BEAM-8442] Unify bundle register in Python SDK harness URL: https://github.com/apache/beam/pull/10004#issuecomment-552442004 @mxm Thanks a lot for triggering the tests. The test failure seems because of quota issues. I'll re-trigger the test to see if it still happens. log的链接：https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi_PR/47/testReport/org.apache.beam.examples.cookbook/BigQueryTornadoesIT/testE2EBigQueryTornadoesWithExport/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341293) Time Spent: 4h (was: 3h 50m) > Unify bundle register in Python SDK harness > --- > > Key: BEAM-8442 > URL: https://issues.apache.org/jira/browse/BEAM-8442 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > There are two methods for bundle register in Python SDK harness: > `SdkHarness._request_register` and `SdkWorker.register.` It should be unfied. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8157) Key encoding for state requests is not consistent across SDKs
[ https://issues.apache.org/jira/browse/BEAM-8157?focusedWorklogId=341290=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341290 ] ASF GitHub Bot logged work on BEAM-8157: Author: ASF GitHub Bot Created on: 11/Nov/19 13:18 Start Date: 11/Nov/19 13:18 Worklog Time Spent: 10m Work Description: mxm commented on issue #9997: [BEAM-8157] Fix key encoding issues for state requests with unknown coders / Improve debugging and testing URL: https://github.com/apache/beam/pull/9997#issuecomment-552441803 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341290) Time Spent: 10h 10m (was: 10h) > Key encoding for state requests is not consistent across SDKs > - > > Key: BEAM-8157 > URL: https://issues.apache.org/jira/browse/BEAM-8157 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.13.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Critical > Fix For: 2.17.0 > > Time Spent: 10h 10m > Remaining Estimate: 0h > > The Flink runner requires the internal key to be encoded without a length > prefix (OUTER context). The user state request handler exposes a serialized > version of the key to the Runner. This key is encoded with the NESTED context > which may add a length prefix. We need to convert it to OUTER context to > match the Flink runner's key encoding. > So far this has not caused the Flink Runner to behave incorrectly. However, > with the upcoming support for Flink 1.9, the state backend will not accept > requests for keys not part of any key group/partition of the operator. This > is very likely to happen with the encoding not being consistent. > **NOTE** This is only applicable to the Java SDK, as the Python SDK uses > OUTER encoding for the key in state requests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8157) Key encoding for state requests is not consistent across SDKs
[ https://issues.apache.org/jira/browse/BEAM-8157?focusedWorklogId=341291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341291 ] ASF GitHub Bot logged work on BEAM-8157: Author: ASF GitHub Bot Created on: 11/Nov/19 13:18 Start Date: 11/Nov/19 13:18 Worklog Time Spent: 10m Work Description: mxm commented on issue #9997: [BEAM-8157] Fix key encoding issues for state requests with unknown coders / Improve debugging and testing URL: https://github.com/apache/beam/pull/9997#issuecomment-552441803 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341291) Time Spent: 10h 20m (was: 10h 10m) > Key encoding for state requests is not consistent across SDKs > - > > Key: BEAM-8157 > URL: https://issues.apache.org/jira/browse/BEAM-8157 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.13.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Critical > Fix For: 2.17.0 > > Time Spent: 10h 20m > Remaining Estimate: 0h > > The Flink runner requires the internal key to be encoded without a length > prefix (OUTER context). The user state request handler exposes a serialized > version of the key to the Runner. This key is encoded with the NESTED context > which may add a length prefix. We need to convert it to OUTER context to > match the Flink runner's key encoding. > So far this has not caused the Flink Runner to behave incorrectly. However, > with the upcoming support for Flink 1.9, the state backend will not accept > requests for keys not part of any key group/partition of the operator. This > is very likely to happen with the encoding not being consistent. > **NOTE** This is only applicable to the Java SDK, as the Python SDK uses > OUTER encoding for the key in state requests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8608) Chain DoFns in Flink batch runner when possible.
[ https://issues.apache.org/jira/browse/BEAM-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Moravek updated BEAM-8608: Description: Right now, in Batch runner, DoFn is executed using MapPartition operator (_FlinkDoFnFunction_), which doesn't have chained driver implementation. We need to reimplement DoFnFunction with FlatMap to allow chaining. Attached is the execution graph for the same pipeline, before and after the patch. was: Right now, in Batch runner, DoFn is executed using MapPartition operator (_FlinkDoFnFunction_), which doesn't have chained driver implementation. We need to reimplement DoFnFunction with FlatMap to allow chaining. > Chain DoFns in Flink batch runner when possible. > > > Key: BEAM-8608 > URL: https://issues.apache.org/jira/browse/BEAM-8608 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Affects Versions: 2.16.0 >Reporter: David Moravek >Assignee: David Moravek >Priority: Major > Attachments: Screen Shot 2019-11-07 at 10.35.07.png, Screen Shot > 2019-11-07 at 10.35.22.png > > > Right now, in Batch runner, DoFn is executed using MapPartition operator > (_FlinkDoFnFunction_), which doesn't have chained driver implementation. > We need to reimplement DoFnFunction with FlatMap to allow chaining. > Attached is the execution graph for the same pipeline, before and after the > patch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8602) Always use shadow configuration for direct runner dependencies
[ https://issues.apache.org/jira/browse/BEAM-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971734#comment-16971734 ] Luke Cwik commented on BEAM-8602: - ./sdks/java/testing/nexmark/build.gradle and ./sdks/java/testing/load-tests/build.gradle both dynamically "add" the runner as a dependency without selecting the appropriate configuration. I'm sure there are other places in our build files where we do something similar that will have this problem. > Always use shadow configuration for direct runner dependencies > -- > > Key: BEAM-8602 > URL: https://issues.apache.org/jira/browse/BEAM-8602 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, sdk-java-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8457) Instrument Dataflow jobs that are launched from Notebooks
[ https://issues.apache.org/jira/browse/BEAM-8457?focusedWorklogId=341407=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341407 ] ASF GitHub Bot logged work on BEAM-8457: Author: ASF GitHub Bot Created on: 11/Nov/19 18:17 Start Date: 11/Nov/19 18:17 Worklog Time Spent: 10m Work Description: pabloem commented on issue #9887: [release-2.17.0] Revert "Merge pull request #9854 from [BEAM-8457] Label Dataflow jobs… URL: https://github.com/apache/beam/pull/9887#issuecomment-552554913 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341407) Time Spent: 9.5h (was: 9h 20m) > Instrument Dataflow jobs that are launched from Notebooks > - > > Key: BEAM-8457 > URL: https://issues.apache.org/jira/browse/BEAM-8457 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Fix For: 2.17.0 > > Time Spent: 9.5h > Remaining Estimate: 0h > > Dataflow needs the capability to tell how many Dataflow jobs are launched > from the Notebook environment, i.e., the Interactive Runner. > # Change the pipeline.run() API to allow supply a runner and an option > parameter so that a pipeline initially bundled w/ an interactive runner can > be directly run by other runners from notebook. > # Implicitly add the necessary source information through user labels when > the user does p.run(runner=DataflowRunner()). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8539) Clearly define the valid job state transitions
[
https://issues.apache.org/jira/browse/BEAM-8539?focusedWorklogId=341410=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341410
]
ASF GitHub Bot logged work on BEAM-8539:
Author: ASF GitHub Bot
Created on: 11/Nov/19 18:19
Start Date: 11/Nov/19 18:19
Worklog Time Spent: 10m
Work Description: chadrik commented on pull request #9965: [BEAM-8539]
Make job state transitions in python-based runners consistent with java-based
runners
URL: https://github.com/apache/beam/pull/9965#discussion_r344843640
##
File path: model/job-management/src/main/proto/beam_job_api.proto
##
@@ -213,17 +213,40 @@ message JobMessagesResponse {
// without needing to pass through STARTING.
message JobState {
enum Enum {
+// The job state reported by a runner cannot be interpreted by the SDK.
UNSPECIFIED = 0;
+
+// The job has been paused, or has not yet started.
Review comment:
I removed "paused" from the description. I agree that the name STOPPED is
misleading, but renaming it would be a much bigger task that could leak into
non-portable runners. Hopefully the improvement surrounding the documentation
of these enums will suffice for now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 341410)
Time Spent: 6h (was: 5h 50m)
> Clearly define the valid job state transitions
> --
>
> Key: BEAM-8539
> URL: https://issues.apache.org/jira/browse/BEAM-8539
> Project: Beam
> Issue Type: Improvement
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Chad Dombrova
>Priority: Major
> Time Spent: 6h
> Remaining Estimate: 0h
>
> The Beam job state transitions are ill-defined, which is big problem for
> anything that relies on the values coming from JobAPI.GetStateStream.
> I was hoping to find something like a state transition diagram in the docs so
> that I could determine the start state, the terminal states, and the valid
> transitions, but I could not find this. The code reveals that the SDKs differ
> on the fundamentals:
> Java InMemoryJobService:
> * start state: *STOPPED*
> * run - about to submit to executor: STARTING
> * run - actually running on executor: RUNNING
> * terminal states: DONE, FAILED, CANCELLED, DRAINED
> Python AbstractJobServiceServicer / LocalJobServicer:
> * start state: STARTING
> * terminal states: DONE, FAILED, CANCELLED, *STOPPED*
> I think it would be good to make python work like Java, so that there is a
> difference in state between a job that has been prepared and one that has
> additionally been run.
> It's hard to tell how far this problem has spread within the various runners.
> I think a simple thing that can be done to help standardize behavior is to
> implement the terminal states as an enum in the beam_job_api.proto, or create
> a utility function in each language for checking if a state is terminal, so
> that it's not left up to each runner to reimplement this logic.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8457) Instrument Dataflow jobs that are launched from Notebooks
[ https://issues.apache.org/jira/browse/BEAM-8457?focusedWorklogId=341411=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341411 ] ASF GitHub Bot logged work on BEAM-8457: Author: ASF GitHub Bot Created on: 11/Nov/19 18:20 Start Date: 11/Nov/19 18:20 Worklog Time Spent: 10m Work Description: KevinGG commented on pull request #9885: [BEAM-8457] Label Dataflow jobs from Notebook URL: https://github.com/apache/beam/pull/9885#discussion_r344843896 ## File path: sdks/python/apache_beam/runners/interactive/interactive_environment.py ## @@ -93,17 +94,7 @@ def __init__(self, cache_manager=None): 'install apache-beam[interactive]` to install necessary ' 'dependencies to enable all data visualization features.') -self._is_in_ipython = False -self._is_in_notebook = False -# Check if the runtime is within an interactive environment, i.e., ipython. -try: - from IPython import get_ipython # pylint: disable=import-error - if get_ipython(): -self._is_in_ipython = True -if 'IPKernelApp' in get_ipython().config: - self._is_in_notebook = True -except ImportError: - pass +self._is_in_ipython, self._is_in_notebook = is_interactive() Review comment: Roger, will make it into 2 separate APIs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341411) Time Spent: 9h 40m (was: 9.5h) > Instrument Dataflow jobs that are launched from Notebooks > - > > Key: BEAM-8457 > URL: https://issues.apache.org/jira/browse/BEAM-8457 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Fix For: 2.17.0 > > Time Spent: 9h 40m > Remaining Estimate: 0h > > Dataflow needs the capability to tell how many Dataflow jobs are launched > from the Notebook environment, i.e., the Interactive Runner. > # Change the pipeline.run() API to allow supply a runner and an option > parameter so that a pipeline initially bundled w/ an interactive runner can > be directly run by other runners from notebook. > # Implicitly add the necessary source information through user labels when > the user does p.run(runner=DataflowRunner()). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-2572) Implement an S3 filesystem for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-2572?focusedWorklogId=341413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341413 ] ASF GitHub Bot logged work on BEAM-2572: Author: ASF GitHub Bot Created on: 11/Nov/19 18:28 Start Date: 11/Nov/19 18:28 Worklog Time Spent: 10m Work Description: pabloem commented on issue #9955: [BEAM-2572] Python SDK S3 Filesystem URL: https://github.com/apache/beam/pull/9955#issuecomment-552558567 I can review. Looking today. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341413) Time Spent: 1h 20m (was: 1h 10m) > Implement an S3 filesystem for Python SDK > - > > Key: BEAM-2572 > URL: https://issues.apache.org/jira/browse/BEAM-2572 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Dmitry Demeshchuk >Priority: Minor > Labels: GSoC2019, gsoc, gsoc2019, mentor, outreachy19dec > Time Spent: 1h 20m > Remaining Estimate: 0h > > There are two paths worth exploring, to my understanding: > 1. Sticking to the HDFS-based approach (like it's done in Java). > 2. Using boto/boto3 for accessing S3 through its common API endpoints. > I personally prefer the second approach, for a few reasons: > 1. In real life, HDFS and S3 have different consistency guarantees, therefore > their behaviors may contradict each other in some edge cases (say, we write > something to S3, but it's not immediately accessible for reading from another > end). > 2. There are other AWS-based sources and sinks we may want to create in the > future: DynamoDB, Kinesis, SQS, etc. > 3. boto3 already provides somewhat good logic for basic things like > reattempting. > Whatever path we choose, there's another problem related to this: we > currently cannot pass any global settings (say, pipeline options, or just an > arbitrary kwarg) to a filesystem. Because of that, we'd have to setup the > runner nodes to have AWS keys set up in the environment, which is not trivial > to achieve and doesn't look too clean either (I'd rather see one single place > for configuring the runner options). > Also, it's worth mentioning that I already have a janky S3 filesystem > implementation that only supports DirectRunner at the moment (because of the > previous paragraph). I'm perfectly fine finishing it myself, with some > guidance from the maintainers. > Where should I move on from here, and whose input should I be looking for? > Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8343) Add means for IO APIs to support predicate and/or project push-down when running SQL pipelines
[ https://issues.apache.org/jira/browse/BEAM-8343?focusedWorklogId=341452=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341452 ] ASF GitHub Bot logged work on BEAM-8343: Author: ASF GitHub Bot Created on: 11/Nov/19 19:29 Start Date: 11/Nov/19 19:29 Worklog Time Spent: 10m Work Description: 11moon11 commented on issue #10060: [BEAM-8343] Updated the cost model to favor IO with push-down. URL: https://github.com/apache/beam/pull/10060#issuecomment-552580854 Run sql postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341452) Time Spent: 6h (was: 5h 50m) > Add means for IO APIs to support predicate and/or project push-down when > running SQL pipelines > -- > > Key: BEAM-8343 > URL: https://issues.apache.org/jira/browse/BEAM-8343 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > > The objective is to create a universal way for Beam SQL IO APIs to support > predicate/project push-down. > A proposed way to achieve that is by introducing an interface responsible > for identifying what portion(s) of a Calc can be moved down to IO layer. > Also, adding following methods to a BeamSqlTable interface to pass necessary > parameters to IO APIs: > - BeamSqlTableFilter constructFilter(List filter) > - ProjectSupport supportsProjects() > - PCollection buildIOReader(PBegin begin, BeamSqlTableFilter filters, > List fieldNames) > > ProjectSupport is an enum with the following options: > * NONE > * WITHOUT_FIELD_REORDERING > * WITH_FIELD_REORDERING > > Design doc > [link|https://docs.google.com/document/d/1-ysD7U7qF3MAmSfkbXZO_5PLJBevAL9bktlLCerd_jE/edit?usp=sharing]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7926) Show PCollection with Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Kang updated BEAM-7926: Description: Support auto plotting / charting of materialized data of a given PCollection with Interactive Beam. Say an Interactive Beam pipeline defined as p = create_pipeline() pcoll = p | 'Transform' >> transform() The use can call a single function and get auto-magical charting of the data as materialized pcoll. e.g., show(pcoll) was: Support auto plotting / charting of materialized data of a given PCollection with Interactive Beam. Say an Interactive Beam pipeline defined as p = create_pipeline() pcoll = p | 'Transform' >> transform() The use can call a single function and get auto-magical charting of the data as materialized pcoll. e.g., visualize(pcoll) > Show PCollection with Interactive Beam > -- > > Key: BEAM-7926 > URL: https://issues.apache.org/jira/browse/BEAM-7926 > Project: Beam > Issue Type: New Feature > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Time Spent: 19h 50m > Remaining Estimate: 0h > > Support auto plotting / charting of materialized data of a given PCollection > with Interactive Beam. > Say an Interactive Beam pipeline defined as > p = create_pipeline() > pcoll = p | 'Transform' >> transform() > The use can call a single function and get auto-magical charting of the data > as materialized pcoll. > e.g., show(pcoll) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7926) Show PCollection with Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Kang updated BEAM-7926: Summary: Show PCollection with Interactive Beam (was: Visualize PCollection with Interactive Beam) > Show PCollection with Interactive Beam > -- > > Key: BEAM-7926 > URL: https://issues.apache.org/jira/browse/BEAM-7926 > Project: Beam > Issue Type: New Feature > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Time Spent: 19h 50m > Remaining Estimate: 0h > > Support auto plotting / charting of materialized data of a given PCollection > with Interactive Beam. > Say an Interactive Beam pipeline defined as > p = create_pipeline() > pcoll = p | 'Transform' >> transform() > The use can call a single function and get auto-magical charting of the data > as materialized pcoll. > e.g., visualize(pcoll) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8457) Instrument Dataflow jobs that are launched from Notebooks
[ https://issues.apache.org/jira/browse/BEAM-8457?focusedWorklogId=341401=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341401 ] ASF GitHub Bot logged work on BEAM-8457: Author: ASF GitHub Bot Created on: 11/Nov/19 18:11 Start Date: 11/Nov/19 18:11 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #9885: [BEAM-8457] Label Dataflow jobs from Notebook URL: https://github.com/apache/beam/pull/9885 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341401) Time Spent: 9h 10m (was: 9h) > Instrument Dataflow jobs that are launched from Notebooks > - > > Key: BEAM-8457 > URL: https://issues.apache.org/jira/browse/BEAM-8457 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Fix For: 2.17.0 > > Time Spent: 9h 10m > Remaining Estimate: 0h > > Dataflow needs the capability to tell how many Dataflow jobs are launched > from the Notebook environment, i.e., the Interactive Runner. > # Change the pipeline.run() API to allow supply a runner and an option > parameter so that a pipeline initially bundled w/ an interactive runner can > be directly run by other runners from notebook. > # Implicitly add the necessary source information through user labels when > the user does p.run(runner=DataflowRunner()). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8343) Add means for IO APIs to support predicate and/or project push-down when running SQL pipelines
[ https://issues.apache.org/jira/browse/BEAM-8343?focusedWorklogId=341412=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341412 ] ASF GitHub Bot logged work on BEAM-8343: Author: ASF GitHub Bot Created on: 11/Nov/19 18:25 Start Date: 11/Nov/19 18:25 Worklog Time Spent: 10m Work Description: 11moon11 commented on issue #10060: [BEAM-8343] Updated the cost model to favor IO with push-down. URL: https://github.com/apache/beam/pull/10060#issuecomment-552557479 Need to rebase after #10030 is merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341412) Time Spent: 5h 20m (was: 5h 10m) > Add means for IO APIs to support predicate and/or project push-down when > running SQL pipelines > -- > > Key: BEAM-8343 > URL: https://issues.apache.org/jira/browse/BEAM-8343 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > The objective is to create a universal way for Beam SQL IO APIs to support > predicate/project push-down. > A proposed way to achieve that is by introducing an interface responsible > for identifying what portion(s) of a Calc can be moved down to IO layer. > Also, adding following methods to a BeamSqlTable interface to pass necessary > parameters to IO APIs: > - BeamSqlTableFilter constructFilter(List filter) > - ProjectSupport supportsProjects() > - PCollection buildIOReader(PBegin begin, BeamSqlTableFilter filters, > List fieldNames) > > ProjectSupport is an enum with the following options: > * NONE > * WITHOUT_FIELD_REORDERING > * WITH_FIELD_REORDERING > > Design doc > [link|https://docs.google.com/document/d/1-ysD7U7qF3MAmSfkbXZO_5PLJBevAL9bktlLCerd_jE/edit?usp=sharing]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8368) [Python] libprotobuf-generated exception when importing apache_beam
[
https://issues.apache.org/jira/browse/BEAM-8368?focusedWorklogId=341433=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341433
]
ASF GitHub Bot logged work on BEAM-8368:
Author: ASF GitHub Bot
Created on: 11/Nov/19 19:03
Start Date: 11/Nov/19 19:03
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #9986: Merge pull request
#9970: [BEAM-8368] [BEAM-8392] Update pyarrow to the latest version 0.15.1
URL: https://github.com/apache/beam/pull/9986#issuecomment-552571069
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 341433)
Time Spent: 4h 20m (was: 4h 10m)
> [Python] libprotobuf-generated exception when importing apache_beam
> ---
>
> Key: BEAM-8368
> URL: https://issues.apache.org/jira/browse/BEAM-8368
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.15.0, 2.17.0
>Reporter: Ubaier Bhat
>Assignee: Brian Hulette
>Priority: Blocker
> Fix For: 2.17.0
>
> Attachments: error_log.txt
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Unable to import apache_beam after upgrading to macos 10.15 (Catalina).
> Cleared all the pipenvs and but can't get it working again.
> {code}
> import apache_beam as beam
> /Users/***/.local/share/virtualenvs/beam-etl-ims6DitU/lib/python3.7/site-packages/apache_beam/__init__.py:84:
> UserWarning: Some syntactic constructs of Python 3 are not yet fully
> supported by Apache Beam.
> 'Some syntactic constructs of Python 3 are not yet fully supported by '
> [libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already
> exists in database:
> [libprotobuf FATAL google/protobuf/descriptor.cc:1370] CHECK failed:
> GeneratedDatabase()->Add(encoded_file_descriptor, size):
> libc++abi.dylib: terminating with uncaught exception of type
> google::protobuf::FatalException: CHECK failed:
> GeneratedDatabase()->Add(encoded_file_descriptor, size):
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8539) Clearly define the valid job state transitions
[
https://issues.apache.org/jira/browse/BEAM-8539?focusedWorklogId=341392=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341392
]
ASF GitHub Bot logged work on BEAM-8539:
Author: ASF GitHub Bot
Created on: 11/Nov/19 17:52
Start Date: 11/Nov/19 17:52
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #9965: [BEAM-8539]
Make job state transitions in python-based runners consistent with java-based
runners
URL: https://github.com/apache/beam/pull/9965#discussion_r344833136
##
File path: model/job-management/src/main/proto/beam_job_api.proto
##
@@ -213,17 +213,40 @@ message JobMessagesResponse {
// without needing to pass through STARTING.
message JobState {
enum Enum {
+// The job state reported by a runner cannot be interpreted by the SDK.
UNSPECIFIED = 0;
+
+// The job has been paused, or has not yet started.
Review comment:
Didn't we decide that there is no pause/restart?
Stopped really sounds like the job was running at one time, until someone
decided to stop it. Perhaps this would be worth renaming to UNSTARTED?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 341392)
Time Spent: 5h 50m (was: 5h 40m)
> Clearly define the valid job state transitions
> --
>
> Key: BEAM-8539
> URL: https://issues.apache.org/jira/browse/BEAM-8539
> Project: Beam
> Issue Type: Improvement
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Chad Dombrova
>Priority: Major
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> The Beam job state transitions are ill-defined, which is big problem for
> anything that relies on the values coming from JobAPI.GetStateStream.
> I was hoping to find something like a state transition diagram in the docs so
> that I could determine the start state, the terminal states, and the valid
> transitions, but I could not find this. The code reveals that the SDKs differ
> on the fundamentals:
> Java InMemoryJobService:
> * start state: *STOPPED*
> * run - about to submit to executor: STARTING
> * run - actually running on executor: RUNNING
> * terminal states: DONE, FAILED, CANCELLED, DRAINED
> Python AbstractJobServiceServicer / LocalJobServicer:
> * start state: STARTING
> * terminal states: DONE, FAILED, CANCELLED, *STOPPED*
> I think it would be good to make python work like Java, so that there is a
> difference in state between a job that has been prepared and one that has
> additionally been run.
> It's hard to tell how far this problem has spread within the various runners.
> I think a simple thing that can be done to help standardize behavior is to
> implement the terminal states as an enum in the beam_job_api.proto, or create
> a utility function in each language for checking if a state is terminal, so
> that it's not left up to each runner to reimplement this logic.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8457) Instrument Dataflow jobs that are launched from Notebooks
[ https://issues.apache.org/jira/browse/BEAM-8457?focusedWorklogId=341402=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341402 ] ASF GitHub Bot logged work on BEAM-8457: Author: ASF GitHub Bot Created on: 11/Nov/19 18:14 Start Date: 11/Nov/19 18:14 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #9885: [BEAM-8457] Label Dataflow jobs from Notebook URL: https://github.com/apache/beam/pull/9885#discussion_r344841478 ## File path: sdks/python/apache_beam/runners/interactive/interactive_environment.py ## @@ -93,17 +94,7 @@ def __init__(self, cache_manager=None): 'install apache-beam[interactive]` to install necessary ' 'dependencies to enable all data visualization features.') -self._is_in_ipython = False -self._is_in_notebook = False -# Check if the runtime is within an interactive environment, i.e., ipython. -try: - from IPython import get_ipython # pylint: disable=import-error - if get_ipython(): -self._is_in_ipython = True -if 'IPKernelApp' in get_ipython().config: - self._is_in_notebook = True -except ImportError: - pass +self._is_in_ipython, self._is_in_notebook = is_interactive() Review comment: Conventionally, `is_xxx` functions return a boolean. Returning a pair will be especially surprising if one writes statements like `if is_interactive()` and the return value is `(False, False)` (which as a non-zero-length tuple evaluates to `True`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341402) Time Spent: 9h 20m (was: 9h 10m) > Instrument Dataflow jobs that are launched from Notebooks > - > > Key: BEAM-8457 > URL: https://issues.apache.org/jira/browse/BEAM-8457 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Fix For: 2.17.0 > > Time Spent: 9h 20m > Remaining Estimate: 0h > > Dataflow needs the capability to tell how many Dataflow jobs are launched > from the Notebook environment, i.e., the Interactive Runner. > # Change the pipeline.run() API to allow supply a runner and an option > parameter so that a pipeline initially bundled w/ an interactive runner can > be directly run by other runners from notebook. > # Implicitly add the necessary source information through user labels when > the user does p.run(runner=DataflowRunner()). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=341442=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341442 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 11/Nov/19 19:14 Start Date: 11/Nov/19 19:14 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #10050: [BEAM-8575] Add streaming test case for multi-triggered GBK as side input URL: https://github.com/apache/beam/pull/10050#issuecomment-552575277 CC: @brandonpollack23 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341442) Time Spent: 2h 10m (was: 2h) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8343) Add means for IO APIs to support predicate and/or project push-down when running SQL pipelines
[ https://issues.apache.org/jira/browse/BEAM-8343?focusedWorklogId=341451=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341451 ] ASF GitHub Bot logged work on BEAM-8343: Author: ASF GitHub Bot Created on: 11/Nov/19 19:29 Start Date: 11/Nov/19 19:29 Worklog Time Spent: 10m Work Description: 11moon11 commented on issue #10060: [BEAM-8343] Updated the cost model to favor IO with push-down. URL: https://github.com/apache/beam/pull/10060#issuecomment-552580854 Run sql postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341451) Time Spent: 5h 50m (was: 5h 40m) > Add means for IO APIs to support predicate and/or project push-down when > running SQL pipelines > -- > > Key: BEAM-8343 > URL: https://issues.apache.org/jira/browse/BEAM-8343 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > > The objective is to create a universal way for Beam SQL IO APIs to support > predicate/project push-down. > A proposed way to achieve that is by introducing an interface responsible > for identifying what portion(s) of a Calc can be moved down to IO layer. > Also, adding following methods to a BeamSqlTable interface to pass necessary > parameters to IO APIs: > - BeamSqlTableFilter constructFilter(List filter) > - ProjectSupport supportsProjects() > - PCollection buildIOReader(PBegin begin, BeamSqlTableFilter filters, > List fieldNames) > > ProjectSupport is an enum with the following options: > * NONE > * WITHOUT_FIELD_REORDERING > * WITH_FIELD_REORDERING > > Design doc > [link|https://docs.google.com/document/d/1-ysD7U7qF3MAmSfkbXZO_5PLJBevAL9bktlLCerd_jE/edit?usp=sharing]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8343) Add means for IO APIs to support predicate and/or project push-down when running SQL pipelines
[ https://issues.apache.org/jira/browse/BEAM-8343?focusedWorklogId=341449=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341449 ] ASF GitHub Bot logged work on BEAM-8343: Author: ASF GitHub Bot Created on: 11/Nov/19 19:29 Start Date: 11/Nov/19 19:29 Worklog Time Spent: 10m Work Description: 11moon11 commented on issue #10060: [BEAM-8343] Updated the cost model to favor IO with push-down. URL: https://github.com/apache/beam/pull/10060#issuecomment-552580620 Run sql postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341449) Time Spent: 5.5h (was: 5h 20m) > Add means for IO APIs to support predicate and/or project push-down when > running SQL pipelines > -- > > Key: BEAM-8343 > URL: https://issues.apache.org/jira/browse/BEAM-8343 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > > The objective is to create a universal way for Beam SQL IO APIs to support > predicate/project push-down. > A proposed way to achieve that is by introducing an interface responsible > for identifying what portion(s) of a Calc can be moved down to IO layer. > Also, adding following methods to a BeamSqlTable interface to pass necessary > parameters to IO APIs: > - BeamSqlTableFilter constructFilter(List filter) > - ProjectSupport supportsProjects() > - PCollection buildIOReader(PBegin begin, BeamSqlTableFilter filters, > List fieldNames) > > ProjectSupport is an enum with the following options: > * NONE > * WITHOUT_FIELD_REORDERING > * WITH_FIELD_REORDERING > > Design doc > [link|https://docs.google.com/document/d/1-ysD7U7qF3MAmSfkbXZO_5PLJBevAL9bktlLCerd_jE/edit?usp=sharing]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8343) Add means for IO APIs to support predicate and/or project push-down when running SQL pipelines
[ https://issues.apache.org/jira/browse/BEAM-8343?focusedWorklogId=341450=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341450 ] ASF GitHub Bot logged work on BEAM-8343: Author: ASF GitHub Bot Created on: 11/Nov/19 19:29 Start Date: 11/Nov/19 19:29 Worklog Time Spent: 10m Work Description: 11moon11 commented on issue #10060: [BEAM-8343] Updated the cost model to favor IO with push-down. URL: https://github.com/apache/beam/pull/10060#issuecomment-552580620 Run sql postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341450) Time Spent: 5h 40m (was: 5.5h) > Add means for IO APIs to support predicate and/or project push-down when > running SQL pipelines > -- > > Key: BEAM-8343 > URL: https://issues.apache.org/jira/browse/BEAM-8343 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > > The objective is to create a universal way for Beam SQL IO APIs to support > predicate/project push-down. > A proposed way to achieve that is by introducing an interface responsible > for identifying what portion(s) of a Calc can be moved down to IO layer. > Also, adding following methods to a BeamSqlTable interface to pass necessary > parameters to IO APIs: > - BeamSqlTableFilter constructFilter(List filter) > - ProjectSupport supportsProjects() > - PCollection buildIOReader(PBegin begin, BeamSqlTableFilter filters, > List fieldNames) > > ProjectSupport is an enum with the following options: > * NONE > * WITHOUT_FIELD_REORDERING > * WITH_FIELD_REORDERING > > Design doc > [link|https://docs.google.com/document/d/1-ysD7U7qF3MAmSfkbXZO_5PLJBevAL9bktlLCerd_jE/edit?usp=sharing]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8602) Always use shadow configuration for direct runner dependencies
[ https://issues.apache.org/jira/browse/BEAM-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik updated BEAM-8602: Status: Open (was: Triage Needed) > Always use shadow configuration for direct runner dependencies > -- > > Key: BEAM-8602 > URL: https://issues.apache.org/jira/browse/BEAM-8602 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, sdk-java-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8597) Allow TestStream trigger tests to run on other runners.
[ https://issues.apache.org/jira/browse/BEAM-8597?focusedWorklogId=341415=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341415 ] ASF GitHub Bot logged work on BEAM-8597: Author: ASF GitHub Bot Created on: 11/Nov/19 18:32 Start Date: 11/Nov/19 18:32 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10043: [BEAM-8597] Allow TestStream trigger tests to run on other runners. URL: https://github.com/apache/beam/pull/10043#issuecomment-552560153 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341415) Time Spent: 40m (was: 0.5h) > Allow TestStream trigger tests to run on other runners. > --- > > Key: BEAM-8597 > URL: https://issues.apache.org/jira/browse/BEAM-8597 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8598) TestStream broken across multiple stages in Flink
[ https://issues.apache.org/jira/browse/BEAM-8598?focusedWorklogId=341416=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341416 ] ASF GitHub Bot logged work on BEAM-8598: Author: ASF GitHub Bot Created on: 11/Nov/19 18:34 Start Date: 11/Nov/19 18:34 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10049: [BEAM-8598] Test triggering BEAM-8598 on FlinkRunner. URL: https://github.com/apache/beam/pull/10049#issuecomment-552560822 Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341416) Time Spent: 20m (was: 10m) > TestStream broken across multiple stages in Flink > - > > Key: BEAM-8598 > URL: https://issues.apache.org/jira/browse/BEAM-8598 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8598) TestStream broken across multiple stages in Flink
[ https://issues.apache.org/jira/browse/BEAM-8598?focusedWorklogId=341417=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341417 ] ASF GitHub Bot logged work on BEAM-8598: Author: ASF GitHub Bot Created on: 11/Nov/19 18:34 Start Date: 11/Nov/19 18:34 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10049: [BEAM-8598] Test triggering BEAM-8598 on FlinkRunner. URL: https://github.com/apache/beam/pull/10049#issuecomment-552560868 Run Java_Examples_Dataflow PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341417) Time Spent: 0.5h (was: 20m) > TestStream broken across multiple stages in Flink > - > > Key: BEAM-8598 > URL: https://issues.apache.org/jira/browse/BEAM-8598 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=341437=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341437 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 11/Nov/19 19:08 Start Date: 11/Nov/19 19:08 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10048: [BEAM-8472] test Java default GCP region URL: https://github.com/apache/beam/pull/10048#issuecomment-552573076 Going forward, it is much easier to review if you don't flatten the commits so I can see the usually trivial diff. I can flatten the commits when merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341437) Time Spent: 3h 40m (was: 3.5h) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=341436=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341436 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 11/Nov/19 19:08 Start Date: 11/Nov/19 19:08 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10048: [BEAM-8472] test Java default GCP region URL: https://github.com/apache/beam/pull/10048 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341436) Time Spent: 3.5h (was: 3h 20m) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8457) Instrument Dataflow jobs that are launched from Notebooks
[ https://issues.apache.org/jira/browse/BEAM-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Kang updated BEAM-8457: Fix Version/s: (was: 2.17.0) 2.18.0 Description: Dataflow needs the capability to tell how many Dataflow jobs are launched from the Notebook environment. We are doing it by checking if the current execution path is with ipython and if the ipython kernel is connected to a notebook frontend. was: Dataflow needs the capability to tell how many Dataflow jobs are launched from the Notebook environment, i.e., the Interactive Runner. # Change the pipeline.run() API to allow supply a runner and an option parameter so that a pipeline initially bundled w/ an interactive runner can be directly run by other runners from notebook. # Implicitly add the necessary source information through user labels when the user does p.run(runner=DataflowRunner()). > Instrument Dataflow jobs that are launched from Notebooks > - > > Key: BEAM-8457 > URL: https://issues.apache.org/jira/browse/BEAM-8457 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Fix For: 2.18.0 > > Time Spent: 9h 40m > Remaining Estimate: 0h > > Dataflow needs the capability to tell how many Dataflow jobs are launched > from the Notebook environment. > We are doing it by checking if the current execution path is with ipython and > if the ipython kernel is connected to a notebook frontend. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8151) Allow the Python SDK to use many many threads
[ https://issues.apache.org/jira/browse/BEAM-8151?focusedWorklogId=341380=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341380 ] ASF GitHub Bot logged work on BEAM-8151: Author: ASF GitHub Bot Created on: 11/Nov/19 17:11 Start Date: 11/Nov/19 17:11 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #9477: [BEAM-8151, BEAM-7848] Up the max number of threads inside the SDK harness to a default of 10k URL: https://github.com/apache/beam/pull/9477#issuecomment-552530990 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341380) Time Spent: 8h 20m (was: 8h 10m) > Allow the Python SDK to use many many threads > - > > Key: BEAM-8151 > URL: https://issues.apache.org/jira/browse/BEAM-8151 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core, sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Time Spent: 8h 20m > Remaining Estimate: 0h > > We need to use a thread pool which shrinks the number of active threads when > they are not being used. > > This is to prevent any stuckness issues related to a runner scheduling more > work items then there are "work" threads inside the SDK harness. > > By default the control plane should have all "requests" being processed in > parallel and the runner is responsible for not overloading the SDK with too > much work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8442) Unify bundle register in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8442?focusedWorklogId=341394=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341394 ] ASF GitHub Bot logged work on BEAM-8442: Author: ASF GitHub Bot Created on: 11/Nov/19 17:57 Start Date: 11/Nov/19 17:57 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10004: [BEAM-8442] Unify bundle register in Python SDK harness URL: https://github.com/apache/beam/pull/10004#issuecomment-552547680 The control plane is asynchronous, but still ordered. IIRC bundle registration was synchronous before such that if a bundle was not registered, an error could be thrown rather than waiting around for a registration that may never come. What is the behavior after this CL if the registration does not complete before the process bundle instruction is seen? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341394) Time Spent: 4h 40m (was: 4.5h) > Unify bundle register in Python SDK harness > --- > > Key: BEAM-8442 > URL: https://issues.apache.org/jira/browse/BEAM-8442 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.18.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > There are two methods for bundle register in Python SDK harness: > `SdkHarness._request_register` and `SdkWorker.register.` It should be unfied. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=341399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341399 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 11/Nov/19 18:09 Start Date: 11/Nov/19 18:09 Worklog Time Spent: 10m Work Description: ibzib commented on issue #10048: [BEAM-8472] test Java default GCP region URL: https://github.com/apache/beam/pull/10048#issuecomment-552551911 > It looks like there is a checkstyle and spotbugs violation, please fix. Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341399) Time Spent: 3h 20m (was: 3h 10m) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8343) Add means for IO APIs to support predicate and/or project push-down when running SQL pipelines
[ https://issues.apache.org/jira/browse/BEAM-8343?focusedWorklogId=341400=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341400 ] ASF GitHub Bot logged work on BEAM-8343: Author: ASF GitHub Bot Created on: 11/Nov/19 18:11 Start Date: 11/Nov/19 18:11 Worklog Time Spent: 10m Work Description: 11moon11 commented on pull request #10060: [BEAM-8343] Updated the cost model to favor IO with push-down. URL: https://github.com/apache/beam/pull/10060 - Create a new `BeamPushDownIOSourceRel` class. - Perform push-down rule only on `BeamIOSourceRel`. - Cost for `BeamPushDownIOSourceRel` should reflect pushed-down filters and projects. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-8598) TestStream broken across multiple stages in Flink
[ https://issues.apache.org/jira/browse/BEAM-8598?focusedWorklogId=341418=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341418 ] ASF GitHub Bot logged work on BEAM-8598: Author: ASF GitHub Bot Created on: 11/Nov/19 18:35 Start Date: 11/Nov/19 18:35 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10049: [BEAM-8598] Test triggering BEAM-8598 on FlinkRunner. URL: https://github.com/apache/beam/pull/10049#issuecomment-552560901 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341418) Time Spent: 40m (was: 0.5h) > TestStream broken across multiple stages in Flink > - > > Key: BEAM-8598 > URL: https://issues.apache.org/jira/browse/BEAM-8598 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8605) Function display_graph() in example do not exist
[ https://issues.apache.org/jira/browse/BEAM-8605?focusedWorklogId=341439=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341439 ] ASF GitHub Bot logged work on BEAM-8605: Author: ASF GitHub Bot Created on: 11/Nov/19 19:12 Start Date: 11/Nov/19 19:12 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10059: [BEAM-8605] Function display_graph() in example do not exist URL: https://github.com/apache/beam/pull/10059 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341439) Time Spent: 40m (was: 0.5h) > Function display_graph() in example do not exist > > > Key: BEAM-8605 > URL: https://issues.apache.org/jira/browse/BEAM-8605 > Project: Beam > Issue Type: Bug > Components: runner-py-interactive, sdk-py-core >Reporter: Rustam Khalmurzaev >Priority: Trivial > Labels: easyfix > Time Spent: 40m > Remaining Estimate: 0h > > Here in code an example of using PipelineGraph class. Example in comments > contains function display_graph() which do not exist. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py#L47] > > Examples: graph = pipeline_graph.PipelineGraph(pipeline_proto) > graph.display_graph() > or > graph = pipeline_graph.PipelineGraph(pipeline) > graph.display_graph() -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7013) A new count distinct transform based on BigQuery compatible HyperLogLog++ implementation
[ https://issues.apache.org/jira/browse/BEAM-7013?focusedWorklogId=341438=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341438 ] ASF GitHub Bot logged work on BEAM-7013: Author: ASF GitHub Bot Created on: 11/Nov/19 19:11 Start Date: 11/Nov/19 19:11 Worklog Time Spent: 10m Work Description: boyuanzz commented on pull request #9778: [BEAM-7013] Update BigQueryHllSketchCompatibilityIT to cover empty sketch cases URL: https://github.com/apache/beam/pull/9778 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341438) Time Spent: 37h 20m (was: 37h 10m) > A new count distinct transform based on BigQuery compatible HyperLogLog++ > implementation > > > Key: BEAM-7013 > URL: https://issues.apache.org/jira/browse/BEAM-7013 > Project: Beam > Issue Type: New Feature > Components: extensions-java-sketching, sdk-java-core >Reporter: Yueyang Qiu >Assignee: Yueyang Qiu >Priority: Major > Fix For: 2.16.0 > > Time Spent: 37h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8151) Allow the Python SDK to use many many threads
[ https://issues.apache.org/jira/browse/BEAM-8151?focusedWorklogId=341440=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341440 ] ASF GitHub Bot logged work on BEAM-8151: Author: ASF GitHub Bot Created on: 11/Nov/19 19:13 Start Date: 11/Nov/19 19:13 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #9477: [BEAM-8151, BEAM-7848] Up the max number of threads inside the SDK harness to a default of 10k URL: https://github.com/apache/beam/pull/9477#issuecomment-552574998 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341440) Time Spent: 8.5h (was: 8h 20m) > Allow the Python SDK to use many many threads > - > > Key: BEAM-8151 > URL: https://issues.apache.org/jira/browse/BEAM-8151 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core, sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > > We need to use a thread pool which shrinks the number of active threads when > they are not being used. > > This is to prevent any stuckness issues related to a runner scheduling more > work items then there are "work" threads inside the SDK harness. > > By default the control plane should have all "requests" being processed in > parallel and the runner is responsible for not overloading the SDK with too > much work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7926) Show PCollection with Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971824#comment-16971824 ] Ning Kang commented on BEAM-7926: - Implementation has been added. Mark it as resolved. > Show PCollection with Interactive Beam > -- > > Key: BEAM-7926 > URL: https://issues.apache.org/jira/browse/BEAM-7926 > Project: Beam > Issue Type: New Feature > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Time Spent: 19h 50m > Remaining Estimate: 0h > > Support auto plotting / charting of materialized data of a given PCollection > with Interactive Beam. > Say an Interactive Beam pipeline defined as > p = create_pipeline() > pcoll = p | 'Transform' >> transform() > The use can call a single function and get auto-magical charting of the data > as materialized pcoll. > e.g., show(pcoll) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8586) Add a server for MongoDb Integration Test
[ https://issues.apache.org/jira/browse/BEAM-8586?focusedWorklogId=341428=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341428 ] ASF GitHub Bot logged work on BEAM-8586: Author: ASF GitHub Bot Created on: 11/Nov/19 18:58 Start Date: 11/Nov/19 18:58 Worklog Time Spent: 10m Work Description: 11moon11 commented on pull request #10061: [BEAM-8586] [SQL] Fix MongoDb integration tests URL: https://github.com/apache/beam/pull/10061 - MongoDb Sql integrations tests should start and use a local instance of a MongoDb service. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-8157) Key encoding for state requests is not consistent across SDKs
[ https://issues.apache.org/jira/browse/BEAM-8157?focusedWorklogId=341458=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341458 ] ASF GitHub Bot logged work on BEAM-8157: Author: ASF GitHub Bot Created on: 11/Nov/19 19:56 Start Date: 11/Nov/19 19:56 Worklog Time Spent: 10m Work Description: mxm commented on issue #9997: [BEAM-8157] Fix key encoding issues for state requests with unknown coders / Improve debugging and testing URL: https://github.com/apache/beam/pull/9997#issuecomment-552590564 As usual, Python PreCommit times out :) https://builds.apache.org/job/beam_PreCommit_Python_Commit/9508/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341458) Time Spent: 10h 50m (was: 10h 40m) > Key encoding for state requests is not consistent across SDKs > - > > Key: BEAM-8157 > URL: https://issues.apache.org/jira/browse/BEAM-8157 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.13.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Critical > Fix For: 2.17.0 > > Time Spent: 10h 50m > Remaining Estimate: 0h > > The Flink runner requires the internal key to be encoded without a length > prefix (OUTER context). The user state request handler exposes a serialized > version of the key to the Runner. This key is encoded with the NESTED context > which may add a length prefix. We need to convert it to OUTER context to > match the Flink runner's key encoding. > So far this has not caused the Flink Runner to behave incorrectly. However, > with the upcoming support for Flink 1.9, the state backend will not accept > requests for keys not part of any key group/partition of the operator. This > is very likely to happen with the encoding not being consistent. > **NOTE** This is only applicable to the Java SDK, as the Python SDK uses > OUTER encoding for the key in state requests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-1440) Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-1440?focusedWorklogId=341473=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341473 ] ASF GitHub Bot logged work on BEAM-1440: Author: ASF GitHub Bot Created on: 11/Nov/19 21:03 Start Date: 11/Nov/19 21:03 Worklog Time Spent: 10m Work Description: pabloem commented on issue #9772: [BEAM-1440] Create a BigQuery source that implements iobase.BoundedSource for Python URL: https://github.com/apache/beam/pull/9772#issuecomment-552614792 I had been traveling. I'll take look now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341473) Time Spent: 9h 40m (was: 9.5h) > Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK > -- > > Key: BEAM-1440 > URL: https://issues.apache.org/jira/browse/BEAM-1440 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Kamil Wasilewski >Priority: Major > Time Spent: 9h 40m > Remaining Estimate: 0h > > Currently we have a BigQuery native source for Python SDK [1]. > This can only be used by Dataflow runner. > We should implement a Beam BigQuery source that implements > iobase.BoundedSource [2] interface so that other runners that try to use > Python SDK can read from BigQuery as well. Java SDK already has a Beam > BigQuery source [3]. > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py > [2] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py#L70 > [3] > https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1189 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-6335) GroupByKey uses data insertion pipeline in streaming tests
[ https://issues.apache.org/jira/browse/BEAM-6335?focusedWorklogId=341488=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341488 ] ASF GitHub Bot logged work on BEAM-6335: Author: ASF GitHub Bot Created on: 11/Nov/19 21:33 Start Date: 11/Nov/19 21:33 Worklog Time Spent: 10m Work Description: kkucharc commented on issue #10029: [BEAM-6335] Test GBK streaming reading SyntheticSources URL: https://github.com/apache/beam/pull/10029#issuecomment-552626159 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341488) Time Spent: 20m (was: 10m) > GroupByKey uses data insertion pipeline in streaming tests > -- > > Key: BEAM-6335 > URL: https://issues.apache.org/jira/browse/BEAM-6335 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Kasia Kucharczyk >Assignee: Kasia Kucharczyk >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Uses prepared Java Data Insertion Pipeline to update GroupByKey in Python to > load test streaming. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8335) Add streaming support to Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=341499=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341499 ] ASF GitHub Bot logged work on BEAM-8335: Author: ASF GitHub Bot Created on: 11/Nov/19 21:52 Start Date: 11/Nov/19 21:52 Worklog Time Spent: 10m Work Description: rohdesamuel commented on pull request #9954: [BEAM-8335] Add the PTuple URL: https://github.com/apache/beam/pull/9954#discussion_r344926475 ## File path: sdks/python/apache_beam/pvalue.py ## @@ -201,6 +201,43 @@ class PDone(PValue): pass +class PTuple(object): + """An object grouping multiple PCollections. + + This class is useful for returning a named tuple of PCollections from a + composite. + """ + + def __init__(self, pcoll_dict): +"""Initializes this named tuple with a dictionary of tagged PCollections. +""" +self._pcolls = pcoll_dict + + def __str__(self): +return '<%s>' % self._str_internal() + + def __repr__(self): +return '<%s at %s>' % (self._str_internal(), hex(id(self))) Review comment: Ack, done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341499) Time Spent: 19h 50m (was: 19h 40m) > Add streaming support to Interactive Beam > - > > Key: BEAM-8335 > URL: https://issues.apache.org/jira/browse/BEAM-8335 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 19h 50m > Remaining Estimate: 0h > > This issue tracks the work items to introduce streaming support to the > Interactive Beam experience. This will allow users to: > * Write and run a streaming job in IPython > * Automatically cache records from unbounded sources > * Add a replay experience that replays all cached records to simulate the > original pipeline execution > * Add controls to play/pause/stop/step individual elements from the cached > records > * Add ability to inspect/visualize unbounded PCollections -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8335) Add streaming support to Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=341498=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341498 ] ASF GitHub Bot logged work on BEAM-8335: Author: ASF GitHub Bot Created on: 11/Nov/19 21:52 Start Date: 11/Nov/19 21:52 Worklog Time Spent: 10m Work Description: rohdesamuel commented on pull request #9954: [BEAM-8335] Add the PTuple URL: https://github.com/apache/beam/pull/9954#discussion_r344926414 ## File path: sdks/python/apache_beam/pvalue.py ## @@ -201,6 +201,43 @@ class PDone(PValue): pass +class PTuple(object): + """An object grouping multiple PCollections. + + This class is useful for returning a named tuple of PCollections from a + composite. + """ + + def __init__(self, pcoll_dict): +"""Initializes this named tuple with a dictionary of tagged PCollections. +""" +self._pcolls = pcoll_dict + + def __str__(self): +return '<%s>' % self._str_internal() + + def __repr__(self): +return '<%s at %s>' % (self._str_internal(), hex(id(self))) + + def _str_internal(self): +return '%s pcollections=%s' % ( +self.__class__.__name__, self._pcolls) + + def __iter__(self): +for tag in self._pcolls: + yield self[tag] + + def __getattr__(self, tag): +# Special methods which may be accessed before the object is Review comment: Removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341498) Time Spent: 19h 40m (was: 19.5h) > Add streaming support to Interactive Beam > - > > Key: BEAM-8335 > URL: https://issues.apache.org/jira/browse/BEAM-8335 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 19h 40m > Remaining Estimate: 0h > > This issue tracks the work items to introduce streaming support to the > Interactive Beam experience. This will allow users to: > * Write and run a streaming job in IPython > * Automatically cache records from unbounded sources > * Add a replay experience that replays all cached records to simulate the > original pipeline execution > * Add controls to play/pause/stop/step individual elements from the cached > records > * Add ability to inspect/visualize unbounded PCollections -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8427) [SQL] Add support for MongoDB source
[ https://issues.apache.org/jira/browse/BEAM-8427?focusedWorklogId=341501=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341501 ] ASF GitHub Bot logged work on BEAM-8427: Author: ASF GitHub Bot Created on: 11/Nov/19 21:53 Start Date: 11/Nov/19 21:53 Worklog Time Spent: 10m Work Description: 11moon11 commented on issue #9892: [BEAM-8427] [SQL] buildIOWrite for MongoDb Table URL: https://github.com/apache/beam/pull/9892#issuecomment-552633435 Run sql postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341501) Time Spent: 7h 10m (was: 7h) > [SQL] Add support for MongoDB source > > > Key: BEAM-8427 > URL: https://issues.apache.org/jira/browse/BEAM-8427 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 7h 10m > Remaining Estimate: 0h > > In progress: > * Create a MongoDB table and table provider. > * Implement buildIOReader > * Support primitive types > Still needs to be done: > * Implement buildIOWrite > * improve getTableStatistics -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8427) [SQL] Add support for MongoDB source
[ https://issues.apache.org/jira/browse/BEAM-8427?focusedWorklogId=341500=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341500 ] ASF GitHub Bot logged work on BEAM-8427: Author: ASF GitHub Bot Created on: 11/Nov/19 21:53 Start Date: 11/Nov/19 21:53 Worklog Time Spent: 10m Work Description: 11moon11 commented on issue #9892: [BEAM-8427] [SQL] buildIOWrite for MongoDb Table URL: https://github.com/apache/beam/pull/9892#issuecomment-552633435 Run sql postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341500) Time Spent: 7h (was: 6h 50m) > [SQL] Add support for MongoDB source > > > Key: BEAM-8427 > URL: https://issues.apache.org/jira/browse/BEAM-8427 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 7h > Remaining Estimate: 0h > > In progress: > * Create a MongoDB table and table provider. > * Implement buildIOReader > * Support primitive types > Still needs to be done: > * Implement buildIOWrite > * improve getTableStatistics -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8427) [SQL] Add support for MongoDB source
[ https://issues.apache.org/jira/browse/BEAM-8427?focusedWorklogId=341502=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341502 ] ASF GitHub Bot logged work on BEAM-8427: Author: ASF GitHub Bot Created on: 11/Nov/19 21:53 Start Date: 11/Nov/19 21:53 Worklog Time Spent: 10m Work Description: 11moon11 commented on issue #9892: [BEAM-8427] [SQL] buildIOWrite for MongoDb Table URL: https://github.com/apache/beam/pull/9892#issuecomment-552037374 Waiting on #10031 to get merged before merging this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341502) Time Spent: 7h 20m (was: 7h 10m) > [SQL] Add support for MongoDB source > > > Key: BEAM-8427 > URL: https://issues.apache.org/jira/browse/BEAM-8427 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 7h 20m > Remaining Estimate: 0h > > In progress: > * Create a MongoDB table and table provider. > * Implement buildIOReader > * Support primitive types > Still needs to be done: > * Implement buildIOWrite > * improve getTableStatistics -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8598) TestStream broken across multiple stages in Flink
[ https://issues.apache.org/jira/browse/BEAM-8598?focusedWorklogId=341506=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341506 ] ASF GitHub Bot logged work on BEAM-8598: Author: ASF GitHub Bot Created on: 11/Nov/19 21:57 Start Date: 11/Nov/19 21:57 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10049: [BEAM-8598] Test triggering BEAM-8598 on FlinkRunner. URL: https://github.com/apache/beam/pull/10049#issuecomment-552634814 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341506) Time Spent: 1h (was: 50m) > TestStream broken across multiple stages in Flink > - > > Key: BEAM-8598 > URL: https://issues.apache.org/jira/browse/BEAM-8598 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8587) Add TestStream support for Dataflow runner
[ https://issues.apache.org/jira/browse/BEAM-8587?focusedWorklogId=341515=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341515 ] ASF GitHub Bot logged work on BEAM-8587: Author: ASF GitHub Bot Created on: 11/Nov/19 22:13 Start Date: 11/Nov/19 22:13 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10041: [BEAM-8587] TestStream for Dataflow runner URL: https://github.com/apache/beam/pull/10041 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341515) Time Spent: 1.5h (was: 1h 20m) > Add TestStream support for Dataflow runner > -- > > Key: BEAM-8587 > URL: https://issues.apache.org/jira/browse/BEAM-8587 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow, testing >Reporter: Andrew Crites >Assignee: Andrew Crites >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > TestStream support needed to test features like late data and processing time > triggers on local Dataflow runner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8589) Add instrumentation to portable runner to print pipeline proto and options when logging level is set to Debug.
[ https://issues.apache.org/jira/browse/BEAM-8589?focusedWorklogId=341521=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341521 ] ASF GitHub Bot logged work on BEAM-8589: Author: ASF GitHub Bot Created on: 11/Nov/19 22:25 Start Date: 11/Nov/19 22:25 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #10036: [BEAM-8589] Print pipeline proto and pipeline options in DEBUG loglevel. URL: https://github.com/apache/beam/pull/10036 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341521) Time Spent: 20m (was: 10m) > Add instrumentation to portable runner to print pipeline proto and options > when logging level is set to Debug. > -- > > Key: BEAM-8589 > URL: https://issues.apache.org/jira/browse/BEAM-8589 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Major > Fix For: Not applicable > > Time Spent: 20m > Remaining Estimate: 0h > > Similar capability in Dataflow runner: > https://github.com/apache/beam/blob/90d587843172143c15ed392513e396b74569a98c/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py#L567. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-8589) Add instrumentation to portable runner to print pipeline proto and options when logging level is set to Debug.
[ https://issues.apache.org/jira/browse/BEAM-8589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentyn Tymofieiev closed BEAM-8589. - Fix Version/s: Not applicable Resolution: Fixed > Add instrumentation to portable runner to print pipeline proto and options > when logging level is set to Debug. > -- > > Key: BEAM-8589 > URL: https://issues.apache.org/jira/browse/BEAM-8589 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Major > Fix For: Not applicable > > Time Spent: 20m > Remaining Estimate: 0h > > Similar capability in Dataflow runner: > https://github.com/apache/beam/blob/90d587843172143c15ed392513e396b74569a98c/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py#L567. -- This message was sent by Atlassian Jira (v8.3.4#803005)
