[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=393056=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-393056
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 25/Feb/20 23:40
Start Date: 25/Feb/20 23:40
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10502: 
[BEAM-7274] Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 393056)
Time Spent: 26h 40m  (was: 26.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Fix For: 2.21.0
>
>  Time Spent: 26h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=393055=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-393055
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 25/Feb/20 23:36
Start Date: 25/Feb/20 23:36
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10502: 
[BEAM-7274] Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#discussion_r384192806
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
 ##
 @@ -0,0 +1,854 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getFieldNumber;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMapKeyMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMapValueMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.withFieldNumber;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.withMessageName;
+
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.Descriptors.FieldDescriptor;
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import java.io.Serializable;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.schemas.logicaltypes.EnumerationType;
+import org.apache.beam.sdk.schemas.logicaltypes.NanosDuration;
+import org.apache.beam.sdk.schemas.logicaltypes.NanosInstant;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+@Experimental(Experimental.Kind.SCHEMAS)
+public class ProtoDynamicMessageSchema implements Serializable {
+  public static final long serialVersionUID = 1L;
+
+  /**
+   * Context of the schema, the context can be generated from a source schema 
or descriptors. The
+   * ability of converting back from Row to proto depends on the type of 
context.
+   */
+  private final Context context;
+
+  /** The toRow function to convert the Message to a Row. */
+  private transient SerializableFunction toRowFunction;
+
+  /** The fromRow function to convert the Row to a Message. */
+  private transient SerializableFunction fromRowFunction;
+
+  /** List of field converters for each field in the row. */
+  private transient List converters;
+
+  private ProtoDynamicMessageSchema(String messageName, ProtoDomain domain) {
+this.context = new DescriptorContext(messageName, domain);
+readResolve();
+  }
+
+  private ProtoDynamicMessageSchema(Context context) {
+this.context = context;
+readResolve();
+  }
+
+  /**
+   * Create a new ProtoDynamicMessageSchema from a {@link ProtoDomain} and for 
a message. The
+   * message need to be in the domain and needs to be the fully qualified name.
+   */
+  public static ProtoDynamicMessageSchema forDescriptor(ProtoDomain domain, 
String messageName) {
+return new ProtoDynamicMessageSchema(messageName, domain);
+  }
+
+  /**
+   * Create a new ProtoDynamicMessageSchema from a {@link ProtoDomain} and for 
a descriptor. The
+   * descriptor is only used for it's name, that name will be used for a 
search in the domain.
+   */
+  public static ProtoDynamicMessageSchema forDescriptor(
+  ProtoDomain domain, Descriptors.Descriptor descriptor) {
+return new ProtoDynamicMessageSchema<>(descriptor.getFullName(), 

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=392692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392692
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 25/Feb/20 17:11
Start Date: 25/Feb/20 17:11
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10502: [BEAM-7274] 
Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#discussion_r384004274
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
 ##
 @@ -0,0 +1,854 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getFieldNumber;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMapKeyMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMapValueMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.withFieldNumber;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.withMessageName;
+
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.Descriptors.FieldDescriptor;
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import java.io.Serializable;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.schemas.logicaltypes.EnumerationType;
+import org.apache.beam.sdk.schemas.logicaltypes.NanosDuration;
+import org.apache.beam.sdk.schemas.logicaltypes.NanosInstant;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+@Experimental(Experimental.Kind.SCHEMAS)
+public class ProtoDynamicMessageSchema implements Serializable {
+  public static final long serialVersionUID = 1L;
+
+  /**
+   * Context of the schema, the context can be generated from a source schema 
or descriptors. The
+   * ability of converting back from Row to proto depends on the type of 
context.
+   */
+  private final Context context;
+
+  /** The toRow function to convert the Message to a Row. */
+  private transient SerializableFunction toRowFunction;
+
+  /** The fromRow function to convert the Row to a Message. */
+  private transient SerializableFunction fromRowFunction;
+
+  /** List of field converters for each field in the row. */
+  private transient List converters;
+
+  private ProtoDynamicMessageSchema(String messageName, ProtoDomain domain) {
+this.context = new DescriptorContext(messageName, domain);
+readResolve();
+  }
+
+  private ProtoDynamicMessageSchema(Context context) {
+this.context = context;
+readResolve();
+  }
+
+  /**
+   * Create a new ProtoDynamicMessageSchema from a {@link ProtoDomain} and for 
a message. The
+   * message need to be in the domain and needs to be the fully qualified name.
+   */
+  public static ProtoDynamicMessageSchema forDescriptor(ProtoDomain domain, 
String messageName) {
+return new ProtoDynamicMessageSchema(messageName, domain);
+  }
+
+  /**
+   * Create a new ProtoDynamicMessageSchema from a {@link ProtoDomain} and for 
a descriptor. The
+   * descriptor is only used for it's name, that name will be used for a 
search in the domain.
+   */
+  public static ProtoDynamicMessageSchema forDescriptor(
+  ProtoDomain domain, Descriptors.Descriptor descriptor) {
+return new ProtoDynamicMessageSchema<>(descriptor.getFullName(), 

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=392327=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392327
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 25/Feb/20 07:05
Start Date: 25/Feb/20 07:05
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10502: 
[BEAM-7274] Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#discussion_r383691619
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
 ##
 @@ -0,0 +1,852 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getFieldNumber;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMapKeyMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMapValueMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.withFieldNumber;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.withMessageName;
+
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.Descriptors.FieldDescriptor;
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import java.io.Serializable;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.schemas.logicaltypes.EnumerationType;
+import org.apache.beam.sdk.schemas.logicaltypes.NanosDuration;
+import org.apache.beam.sdk.schemas.logicaltypes.NanosInstant;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+@Experimental(Experimental.Kind.SCHEMAS)
+public class ProtoDynamicMessageSchema implements Serializable {
+  public static final long serialVersionUID = 1L;
+
+  /**
+   * Context of the schema, the context can be generated from a source schema 
or descriptors. The
+   * ability of converting back from Row to proto depends on the type of 
context.
+   */
+  private final Context context;
+
+  /** THe SchemaCoder holds the resolved schema and to/fromRow functions. */
+  private transient SchemaCoder schemaCoder;
+
+  /** List of field converters for each field in the row. */
+  private transient List converters;
+
+  private ProtoDynamicMessageSchema(String messageName, ProtoDomain domain) {
+this.context = new DescriptorContext(messageName, domain);
+readResolve();
+  }
+
+  private ProtoDynamicMessageSchema(Context context) {
+this.context = context;
+readResolve();
+  }
+
+  /**
+   * Create a new ProtoDynamicMessageSchema from a {@link ProtoDomain} and for 
a message. The
+   * message need to be in the domain and needs to be the fully qualified name.
+   */
+  public static ProtoDynamicMessageSchema forDescriptor(ProtoDomain domain, 
String messageName) {
+return new ProtoDynamicMessageSchema(messageName, domain);
+  }
+
+  /**
+   * Create a new ProtoDynamicMessageSchema from a {@link ProtoDomain} and for 
a descriptor. The
+   * descriptor is only used for it's name, that name will be used for a 
search in the domain.
+   */
+  public static ProtoDynamicMessageSchema forDescriptor(
+  ProtoDomain domain, Descriptors.Descriptor descriptor) {
+return new ProtoDynamicMessageSchema<>(descriptor.getFullName(), domain);
+  }
+
+  static ProtoDynamicMessageSchema forContext(Context context, Schema.Field 
field) {
+return new 

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=392266=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392266
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 25/Feb/20 03:15
Start Date: 25/Feb/20 03:15
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10502: [BEAM-7274] 
Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#discussion_r383637559
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
 ##
 @@ -0,0 +1,852 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getFieldNumber;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMapKeyMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMapValueMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.withFieldNumber;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.withMessageName;
+
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.Descriptors.FieldDescriptor;
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import java.io.Serializable;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.schemas.logicaltypes.EnumerationType;
+import org.apache.beam.sdk.schemas.logicaltypes.NanosDuration;
+import org.apache.beam.sdk.schemas.logicaltypes.NanosInstant;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+@Experimental(Experimental.Kind.SCHEMAS)
+public class ProtoDynamicMessageSchema implements Serializable {
+  public static final long serialVersionUID = 1L;
+
+  /**
+   * Context of the schema, the context can be generated from a source schema 
or descriptors. The
+   * ability of converting back from Row to proto depends on the type of 
context.
+   */
+  private final Context context;
+
+  /** THe SchemaCoder holds the resolved schema and to/fromRow functions. */
+  private transient SchemaCoder schemaCoder;
+
+  /** List of field converters for each field in the row. */
+  private transient List converters;
+
+  private ProtoDynamicMessageSchema(String messageName, ProtoDomain domain) {
+this.context = new DescriptorContext(messageName, domain);
+readResolve();
+  }
+
+  private ProtoDynamicMessageSchema(Context context) {
+this.context = context;
+readResolve();
+  }
+
+  /**
+   * Create a new ProtoDynamicMessageSchema from a {@link ProtoDomain} and for 
a message. The
+   * message need to be in the domain and needs to be the fully qualified name.
+   */
+  public static ProtoDynamicMessageSchema forDescriptor(ProtoDomain domain, 
String messageName) {
+return new ProtoDynamicMessageSchema(messageName, domain);
+  }
+
+  /**
+   * Create a new ProtoDynamicMessageSchema from a {@link ProtoDomain} and for 
a descriptor. The
+   * descriptor is only used for it's name, that name will be used for a 
search in the domain.
+   */
+  public static ProtoDynamicMessageSchema forDescriptor(
+  ProtoDomain domain, Descriptors.Descriptor descriptor) {
+return new ProtoDynamicMessageSchema<>(descriptor.getFullName(), domain);
+  }
+
+  static ProtoDynamicMessageSchema forContext(Context context, Schema.Field 
field) {
+return new 

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=392086=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392086
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 24/Feb/20 20:57
Start Date: 24/Feb/20 20:57
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10502: 
[BEAM-7274] Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#discussion_r383509654
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
 ##
 @@ -0,0 +1,852 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getFieldNumber;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMapKeyMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMapValueMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.withFieldNumber;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.withMessageName;
+
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.Descriptors.FieldDescriptor;
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import java.io.Serializable;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.schemas.logicaltypes.EnumerationType;
+import org.apache.beam.sdk.schemas.logicaltypes.NanosDuration;
+import org.apache.beam.sdk.schemas.logicaltypes.NanosInstant;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+@Experimental(Experimental.Kind.SCHEMAS)
+public class ProtoDynamicMessageSchema implements Serializable {
+  public static final long serialVersionUID = 1L;
+
+  /**
+   * Context of the schema, the context can be generated from a source schema 
or descriptors. The
+   * ability of converting back from Row to proto depends on the type of 
context.
+   */
+  private final Context context;
+
+  /** THe SchemaCoder holds the resolved schema and to/fromRow functions. */
+  private transient SchemaCoder schemaCoder;
+
+  /** List of field converters for each field in the row. */
+  private transient List converters;
+
+  private ProtoDynamicMessageSchema(String messageName, ProtoDomain domain) {
+this.context = new DescriptorContext(messageName, domain);
+readResolve();
+  }
+
+  private ProtoDynamicMessageSchema(Context context) {
+this.context = context;
+readResolve();
+  }
+
+  /**
+   * Create a new ProtoDynamicMessageSchema from a {@link ProtoDomain} and for 
a message. The
+   * message need to be in the domain and needs to be the fully qualified name.
+   */
+  public static ProtoDynamicMessageSchema forDescriptor(ProtoDomain domain, 
String messageName) {
+return new ProtoDynamicMessageSchema(messageName, domain);
+  }
+
+  /**
+   * Create a new ProtoDynamicMessageSchema from a {@link ProtoDomain} and for 
a descriptor. The
+   * descriptor is only used for it's name, that name will be used for a 
search in the domain.
+   */
+  public static ProtoDynamicMessageSchema forDescriptor(
+  ProtoDomain domain, Descriptors.Descriptor descriptor) {
+return new ProtoDynamicMessageSchema<>(descriptor.getFullName(), domain);
+  }
+
+  static ProtoDynamicMessageSchema forContext(Context context, Schema.Field 
field) {
+return new 

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=392069=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392069
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 24/Feb/20 20:28
Start Date: 24/Feb/20 20:28
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10502: 
[BEAM-7274] Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#discussion_r383496192
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
 ##
 @@ -163,15 +163,15 @@ private void indexDescriptorByName() {
 .values()
 .forEach(
 fileDescriptor -> {
-  fileDescriptor
-  .getMessageTypes()
-  .forEach(
-  descriptor -> {
-descriptorMap.put(descriptor.getFullName(), 
descriptor);
-  });
+  fileDescriptor.getMessageTypes().forEach(descriptor -> 
indexDescriptor(descriptor));
 
 Review comment:
   fixed
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 392069)
Time Spent: 25.5h  (was: 25h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Fix For: 2.20.0
>
>  Time Spent: 25.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=392070=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392070
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 24/Feb/20 20:28
Start Date: 24/Feb/20 20:28
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10502: 
[BEAM-7274] Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#discussion_r383496260
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
 ##
 @@ -163,15 +163,15 @@ private void indexDescriptorByName() {
 .values()
 .forEach(
 fileDescriptor -> {
-  fileDescriptor
-  .getMessageTypes()
-  .forEach(
-  descriptor -> {
-descriptorMap.put(descriptor.getFullName(), 
descriptor);
-  });
+  fileDescriptor.getMessageTypes().forEach(descriptor -> 
indexDescriptor(descriptor));
 });
   }
 
+  private void indexDescriptor(Descriptors.Descriptor descriptor) {
+descriptorMap.put(descriptor.getFullName(), descriptor);
+descriptor.getNestedTypes().forEach(nested -> indexDescriptor(nested));
 
 Review comment:
   fixed
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 392070)
Time Spent: 25h 40m  (was: 25.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Fix For: 2.20.0
>
>  Time Spent: 25h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=391834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391834
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 24/Feb/20 16:47
Start Date: 24/Feb/20 16:47
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10944: [BEAM-7274] 
optimize oneOf handling
URL: https://github.com/apache/beam/pull/10944
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 391834)
Time Spent: 25h 20m  (was: 25h 10m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Fix For: 2.20.0
>
>  Time Spent: 25h 20m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=391486=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391486
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 24/Feb/20 03:02
Start Date: 24/Feb/20 03:02
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10502: [BEAM-7274] Add 
DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#issuecomment-590152824
 
 
   A few comments.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 391486)
Time Spent: 24h 40m  (was: 24.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Fix For: 2.20.0
>
>  Time Spent: 24h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=391487=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391487
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 24/Feb/20 03:02
Start Date: 24/Feb/20 03:02
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10502: [BEAM-7274] 
Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#discussion_r383070050
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
 ##
 @@ -163,15 +163,15 @@ private void indexDescriptorByName() {
 .values()
 .forEach(
 fileDescriptor -> {
-  fileDescriptor
-  .getMessageTypes()
-  .forEach(
-  descriptor -> {
-descriptorMap.put(descriptor.getFullName(), 
descriptor);
-  });
+  fileDescriptor.getMessageTypes().forEach(descriptor -> 
indexDescriptor(descriptor));
 
 Review comment:
   I think you can replace the lambda with this::indexDescriptor
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 391487)
Time Spent: 24h 50m  (was: 24h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Fix For: 2.20.0
>
>  Time Spent: 24h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=391488=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391488
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 24/Feb/20 03:02
Start Date: 24/Feb/20 03:02
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10502: [BEAM-7274] 
Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#discussion_r383070124
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
 ##
 @@ -163,15 +163,15 @@ private void indexDescriptorByName() {
 .values()
 .forEach(
 fileDescriptor -> {
-  fileDescriptor
-  .getMessageTypes()
-  .forEach(
-  descriptor -> {
-descriptorMap.put(descriptor.getFullName(), 
descriptor);
-  });
+  fileDescriptor.getMessageTypes().forEach(descriptor -> 
indexDescriptor(descriptor));
 });
   }
 
+  private void indexDescriptor(Descriptors.Descriptor descriptor) {
+descriptorMap.put(descriptor.getFullName(), descriptor);
+descriptor.getNestedTypes().forEach(nested -> indexDescriptor(nested));
 
 Review comment:
   you can do the same here - forEach(this::indexDescriptor)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 391488)
Time Spent: 25h  (was: 24h 50m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Fix For: 2.20.0
>
>  Time Spent: 25h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=391489=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391489
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 24/Feb/20 03:02
Start Date: 24/Feb/20 03:02
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10502: [BEAM-7274] 
Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#discussion_r383071152
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
 ##
 @@ -0,0 +1,852 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getFieldNumber;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMapKeyMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMapValueMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMessageName;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.withFieldNumber;
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.withMessageName;
+
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.Descriptors.FieldDescriptor;
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import java.io.Serializable;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.schemas.logicaltypes.EnumerationType;
+import org.apache.beam.sdk.schemas.logicaltypes.NanosDuration;
+import org.apache.beam.sdk.schemas.logicaltypes.NanosInstant;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+@Experimental(Experimental.Kind.SCHEMAS)
+public class ProtoDynamicMessageSchema implements Serializable {
+  public static final long serialVersionUID = 1L;
+
+  /**
+   * Context of the schema, the context can be generated from a source schema 
or descriptors. The
+   * ability of converting back from Row to proto depends on the type of 
context.
+   */
+  private final Context context;
+
+  /** THe SchemaCoder holds the resolved schema and to/fromRow functions. */
+  private transient SchemaCoder schemaCoder;
+
+  /** List of field converters for each field in the row. */
+  private transient List converters;
+
+  private ProtoDynamicMessageSchema(String messageName, ProtoDomain domain) {
+this.context = new DescriptorContext(messageName, domain);
+readResolve();
+  }
+
+  private ProtoDynamicMessageSchema(Context context) {
+this.context = context;
+readResolve();
+  }
+
+  /**
+   * Create a new ProtoDynamicMessageSchema from a {@link ProtoDomain} and for 
a message. The
+   * message need to be in the domain and needs to be the fully qualified name.
+   */
+  public static ProtoDynamicMessageSchema forDescriptor(ProtoDomain domain, 
String messageName) {
+return new ProtoDynamicMessageSchema(messageName, domain);
+  }
+
+  /**
+   * Create a new ProtoDynamicMessageSchema from a {@link ProtoDomain} and for 
a descriptor. The
+   * descriptor is only used for it's name, that name will be used for a 
search in the domain.
+   */
+  public static ProtoDynamicMessageSchema forDescriptor(
+  ProtoDomain domain, Descriptors.Descriptor descriptor) {
+return new ProtoDynamicMessageSchema<>(descriptor.getFullName(), domain);
+  }
+
+  static ProtoDynamicMessageSchema forContext(Context context, Schema.Field 
field) {
+return new 

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=391475=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391475
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 24/Feb/20 02:35
Start Date: 24/Feb/20 02:35
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10944: [BEAM-7274] 
optimize oneOf handling
URL: https://github.com/apache/beam/pull/10944
 
 
   When codegen support was added to enable protocol buffer schemas, oneof 
fields were left out and handled via map lookups. This was because ByteBuddy 
does not provide an easy way to generate conditional code.
   
   In this PR we use the underlying ASM framework to generate switch/case 
statements to handle oneofs, and remove the slower map-based code path.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 391475)
Time Spent: 24.5h  (was: 24h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Fix For: 2.20.0
>
>  Time Spent: 24.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=389826=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389826
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 20/Feb/20 07:12
Start Date: 20/Feb/20 07:12
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10502: [BEAM-7274] Add 
DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#issuecomment-588652660
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 389826)
Time Spent: 24h 20m  (was: 24h 10m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Fix For: 2.20.0
>
>  Time Spent: 24h 20m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=389008=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389008
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 18/Feb/20 19:13
Start Date: 18/Feb/20 19:13
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10502: [BEAM-7274] Add 
DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#issuecomment-587683339
 
 
   @reuvenlax @iemejia @TheNeuralBit I've tested this PR extensively on 
Dataflow. This is way more stable than the original DynamicMessage schema 
implementation.
   
   Can I get a LGTM, that I can focus on the options. Thanks.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 389008)
Time Spent: 24h  (was: 23h 50m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 24h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=389012=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389012
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 18/Feb/20 19:13
Start Date: 18/Feb/20 19:13
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10502: [BEAM-7274] Add 
DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#issuecomment-587683339
 
 
   @reuvenlax @iemejia @TheNeuralBit I've tested this PR extensively on 
Dataflow. This is way more stable than the original DynamicMessage schema 
implementation.
   
   Can I get a LGTM, that I can focus on the options. Thanks.
   
   (master is failing, it was green till rebase)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 389012)
Time Spent: 24h 10m  (was: 24h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 24h 10m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=388951=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-388951
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 18/Feb/20 17:58
Start Date: 18/Feb/20 17:58
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10502: [BEAM-7274] Add 
DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#issuecomment-587592815
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 388951)
Time Spent: 23h 50m  (was: 23h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 23h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=384131=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-384131
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 09/Feb/20 18:50
Start Date: 09/Feb/20 18:50
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10502: [BEAM-7274] Add 
DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#issuecomment-583879456
 
 
   This branch has been updated to take the new Logical Types into account.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 384131)
Time Spent: 23h 40m  (was: 23.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 23h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=369982=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-369982
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 10/Jan/20 18:39
Start Date: 10/Jan/20 18:39
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10502: [BEAM-7274] Add 
DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#issuecomment-573155873
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 369982)
Time Spent: 23.5h  (was: 23h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 23.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=368124=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368124
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 08/Jan/20 10:50
Start Date: 08/Jan/20 10:50
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10502: [BEAM-7274] Add 
DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#issuecomment-571995563
 
 
   @reuvenlax this is ready for review. It has feature parity with the static 
compiled proto, including all logical types.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 368124)
Time Spent: 23h 20m  (was: 23h 10m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 23h 20m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=368073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368073
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 08/Jan/20 08:59
Start Date: 08/Jan/20 08:59
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #8690: 
[BEAM-7274] Implement the Protobuf schema provider
URL: https://github.com/apache/beam/pull/8690
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 368073)
Time Spent: 23h 10m  (was: 23h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 23h 10m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=368072=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368072
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 08/Jan/20 08:59
Start Date: 08/Jan/20 08:59
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #8690: [BEAM-7274] 
Implement the Protobuf schema provider
URL: https://github.com/apache/beam/pull/8690#issuecomment-571953287
 
 
   Closing this PR as it's supersede by several other PR that are already on 
master: Last remaining PR for full proto support is: 
https://github.com/apache/beam/pull/10502
   
   Option/Metadata support has been split of and can be followed:
   - https://github.com/apache/beam/pull/10413 (options implementation)
   - https://github.com/apache/beam/pull/10529 (options support out of proto)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 368072)
Time Spent: 23h  (was: 22h 50m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 23h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=366317=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366317
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 05/Jan/20 11:43
Start Date: 05/Jan/20 11:43
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10502: [BEAM-7274] Add 
DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#issuecomment-570903828
 
 
   R: @reuvenlax this is an implementation for DynamicMessages. It takes into 
account the new Logical Types and also use RowWithStorage instead of going with 
the getters.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366317)
Time Spent: 22h 50m  (was: 22h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 22h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=366309=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366309
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 05/Jan/20 10:50
Start Date: 05/Jan/20 10:50
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10502: [BEAM-7274] Add 
DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#issuecomment-570899437
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366309)
Time Spent: 22h 40m  (was: 22.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 22h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=366308=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366308
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 05/Jan/20 10:49
Start Date: 05/Jan/20 10:49
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10502: [BEAM-7274] Add 
DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#issuecomment-570894333
 
 
   Run Java PreCommit again
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366308)
Time Spent: 22.5h  (was: 22h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 22.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=366307=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366307
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 05/Jan/20 10:49
Start Date: 05/Jan/20 10:49
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10502: [BEAM-7274] Add 
DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#issuecomment-570894333
 
 
   Run Java PreCommit again
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366307)
Time Spent: 22h 20m  (was: 22h 10m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 22h 20m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=366306=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366306
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 05/Jan/20 09:45
Start Date: 05/Jan/20 09:45
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10502: [BEAM-7274] Add 
DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#issuecomment-570894333
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366306)
Time Spent: 22h 10m  (was: 22h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 22h 10m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=366184=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366184
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 04/Jan/20 09:00
Start Date: 04/Jan/20 09:00
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10502: 
[BEAM-7274] Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502
 
 
   Add DynamicMessage schema support. This is different from
   generated classes as it uses the proto descriptors. It uses
   the ProtoDomain as an index for searching embedded messages.
   
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[![Build
 

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=363356=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-363356
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 25/Dec/19 18:20
Start Date: 25/Dec/19 18:20
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10449: [BEAM-7274] 
Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 363356)
Time Spent: 21h 50m  (was: 21h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 21h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=363119=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-363119
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 24/Dec/19 21:47
Start Date: 24/Dec/19 21:47
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10449: [BEAM-7274] 
Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#issuecomment-568805008
 
 
   @alexvanboxel  support for wrapper types added, and all tests are now green.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 363119)
Time Spent: 21h 40m  (was: 21.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 21h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362740=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362740
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 23/Dec/19 21:07
Start Date: 23/Dec/19 21:07
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10449: [BEAM-7274] 
Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#issuecomment-568581706
 
 
   > If I remove the OneOf then tests will start failing, because we won't be 
able to properly support the codegen for OneOf (basically we would then be 
removing OneOf support in protobuf as well).
   > […](#)
   
   OK, let's keep it in then. Better to tackle it later then. Spotless and Java 
tests are failing though.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362740)
Time Spent: 21.5h  (was: 21h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 21.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362615=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362615
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 23/Dec/19 18:20
Start Date: 23/Dec/19 18:20
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10449: [BEAM-7274] 
Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#issuecomment-568545736
 
 
   If I remove the OneOf then tests will start failing, because we won't be
   able to properly support the codegen for OneOf (basically we would then be
   removing OneOf support in protobuf as well).
   
   
   On Mon, Dec 23, 2019 at 2:39 AM Alex Van Boxel 
   wrote:
   
   > *@alexvanboxel* requested changes on this pull request.
   >
   > Maybe remove the OneOf in the Row (and let's mark it as a known issue). If
   > test are green + spotless I'll give a LGTM.
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362615)
Time Spent: 21h 20m  (was: 21h 10m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 21h 20m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362517=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362517
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 23/Dec/19 12:36
Start Date: 23/Dec/19 12:36
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10449: 
[BEAM-7274] Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360873653
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowWithGetters.java
 ##
 @@ -122,6 +123,15 @@ private Iterable getIterableValue(FieldType elementType, 
Object fieldValue) {
   cacheKey, i -> getMapValue(type.getMapKeyType(), 
type.getMapValueType(), map))
   : (T) getMapValue(type.getMapKeyType(), type.getMapValueType(), map);
 } else {
+  if (type.isLogicalType(OneOfType.IDENTIFIER)) {
 
 Review comment:
   My gut feeling says that we should take it separately. Ok, union types will 
fail, but for now it's only proto specific right? I've added this on my todo 
list when testing the implementation with our use cases.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362517)
Time Spent: 21h 10m  (was: 21h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 21h 10m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362370=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362370
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 23/Dec/19 01:12
Start Date: 23/Dec/19 01:12
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10449: [BEAM-7274] 
Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360741835
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FromRowUsingCreator.java
 ##
 @@ -127,6 +123,22 @@ public T apply(Row row) {
   valueType,
   typeFactory);
 } else {
+  if (type.getTypeName().isLogicalType()
+  && 
OneOfType.IDENTIFIER.equals(type.getLogicalType().getIdentifier())) {
 
 Review comment:
   Same response as before. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362370)
Time Spent: 20h 10m  (was: 20h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 20h 10m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362372=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362372
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 23/Dec/19 01:12
Start Date: 23/Dec/19 01:12
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10449: [BEAM-7274] 
Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360741844
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
 ##
 @@ -0,0 +1,55 @@
+package org.apache.beam.sdk.extensions.protobuf;
+
+import com.google.protobuf.DynamicMessage;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.FieldValueGetter;
+import org.apache.beam.sdk.schemas.FieldValueTypeInformation;
+import org.apache.beam.sdk.schemas.GetterBasedSchemaProvider;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaProvider;
+import org.apache.beam.sdk.schemas.SchemaUserTypeCreator;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+
+import javax.annotation.Nullable;
+import java.util.List;
+
+@Experimental(Kind.SCHEMAS)
+public class ProtoDynamicMessageSchema extends GetterBasedSchemaProvider {
+private static final TypeDescriptor 
DYNAMIC_MESSAGE_TYPE_DESCRIPTOR =
+TypeDescriptor.of(DynamicMessage.class);
+@Nullable
+@Override
+public  Schema schemaFor(TypeDescriptor typeDescriptor) {
+checkForDynamicType(typeDescriptor);
+return ProtoSchemaTranslator.getSchema((Class) 
typeDescriptor.getRawType());
+}
+
+@Override
+public List fieldValueGetters(Class targetClass, 
Schema schema) {
+return null;
 
 Review comment:
   Actually, I meant to remove this file from this PR - it belongs in the next 
PR. I'll remove it.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362372)
Time Spent: 20.5h  (was: 20h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 20.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362373=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362373
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 23/Dec/19 01:12
Start Date: 23/Dec/19 01:12
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10449: [BEAM-7274] 
Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360741852
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoRecordSchema.java
 ##
 @@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoByteBuddyUtils.getProtoGetter;
+
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import java.lang.reflect.Method;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import 
org.apache.beam.sdk.extensions.protobuf.ProtoByteBuddyUtils.ProtoTypeConversionsFactory;
+import org.apache.beam.sdk.schemas.FieldValueGetter;
+import org.apache.beam.sdk.schemas.FieldValueTypeInformation;
+import org.apache.beam.sdk.schemas.GetterBasedSchemaProvider;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.SchemaUserTypeCreator;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType;
+import org.apache.beam.sdk.schemas.utils.FieldValueTypeSupplier;
+import org.apache.beam.sdk.schemas.utils.JavaBeanUtils;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+
+@Experimental(Kind.SCHEMAS)
+public class ProtoRecordSchema extends GetterBasedSchemaProvider {
 
 Review comment:
   Changed.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362373)
Time Spent: 20h 40m  (was: 20.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 20h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362369=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362369
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 23/Dec/19 01:12
Start Date: 23/Dec/19 01:12
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10449: [BEAM-7274] 
Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360741829
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowWithGetters.java
 ##
 @@ -122,6 +123,15 @@ private Iterable getIterableValue(FieldType elementType, 
Object fieldValue) {
   cacheKey, i -> getMapValue(type.getMapKeyType(), 
type.getMapValueType(), map))
   : (T) getMapValue(type.getMapKeyType(), type.getMapValueType(), map);
 } else {
+  if (type.isLogicalType(OneOfType.IDENTIFIER)) {
+OneOfType oneOfType = type.getLogicalType(OneOfType.class);
+OneOfType.Value oneOfValue = (OneOfType.Value) fieldValue;
+Object convertedOneOfField =
+getValue(oneOfValue.getFieldType(), oneOfValue.getValue(), null);
+return (T)
 
 Review comment:
   This is because the current semantics of Row.getValue() is to return the 
underlying type and you're expected to call Row.getLogicalTypeValue() to get 
the LogicalType value. It's debatable whether that's the best solution for Row, 
but there are arguments both ways and I didn't want to change those semantics 
here.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362369)
Time Spent: 20h  (was: 19h 50m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 20h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362368=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362368
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 23/Dec/19 01:12
Start Date: 23/Dec/19 01:12
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10449: [BEAM-7274] 
Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360741828
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowWithGetters.java
 ##
 @@ -122,6 +123,15 @@ private Iterable getIterableValue(FieldType elementType, 
Object fieldValue) {
   cacheKey, i -> getMapValue(type.getMapKeyType(), 
type.getMapValueType(), map))
   : (T) getMapValue(type.getMapKeyType(), type.getMapValueType(), map);
 } else {
+  if (type.isLogicalType(OneOfType.IDENTIFIER)) {
 
 Review comment:
   I thought that originally, however it turns out that the only way to make 
OneOf work with the getters framework is to actually handle it explicitly, 
otherwise recursive translation fails to work. Happy to pull this out into a 
separate PR if you think that's better.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362368)
Time Spent: 19h 50m  (was: 19h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 19h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362374=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362374
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 23/Dec/19 01:12
Start Date: 23/Dec/19 01:12
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10449: [BEAM-7274] 
Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360741860
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
 ##
 @@ -0,0 +1,55 @@
+package org.apache.beam.sdk.extensions.protobuf;
+
+import com.google.protobuf.DynamicMessage;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.FieldValueGetter;
+import org.apache.beam.sdk.schemas.FieldValueTypeInformation;
+import org.apache.beam.sdk.schemas.GetterBasedSchemaProvider;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaProvider;
+import org.apache.beam.sdk.schemas.SchemaUserTypeCreator;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+
+import javax.annotation.Nullable;
+import java.util.List;
+
+@Experimental(Kind.SCHEMAS)
+public class ProtoDynamicMessageSchema extends GetterBasedSchemaProvider {
+private static final TypeDescriptor 
DYNAMIC_MESSAGE_TYPE_DESCRIPTOR =
+TypeDescriptor.of(DynamicMessage.class);
+@Nullable
+@Override
+public  Schema schemaFor(TypeDescriptor typeDescriptor) {
+checkForDynamicType(typeDescriptor);
+return ProtoSchemaTranslator.getSchema((Class) 
typeDescriptor.getRawType());
+}
+
+@Override
+public List fieldValueGetters(Class targetClass, 
Schema schema) {
+return null;
 
 Review comment:
   Actually, I meant to remove this file from this PR - it belongs in the next 
PR. I'll remove it.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362374)
Time Spent: 20h 50m  (was: 20h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 20h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362371=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362371
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 23/Dec/19 01:12
Start Date: 23/Dec/19 01:12
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10449: [BEAM-7274] 
Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360741842
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoByteBuddyUtils.java
 ##
 @@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getFieldNumber;
+
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Duration;
+import com.google.protobuf.Internal.EnumLite;
+import com.google.protobuf.MessageLite;
+import com.google.protobuf.ProtocolMessageEnum;
+import com.google.protobuf.Timestamp;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.lang.reflect.Type;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaLogicalTypes.DurationNanos;
+import 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaLogicalTypes.TimestampNanos;
+import org.apache.beam.sdk.schemas.FieldValueGetter;
+import org.apache.beam.sdk.schemas.FieldValueSetter;
+import org.apache.beam.sdk.schemas.FieldValueTypeInformation;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.schemas.SchemaUserTypeCreator;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType.Value;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertType;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForGetter;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForSetter;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.InjectPackageStrategy;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.TypeConversion;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.TypeConversionsFactory;
+import org.apache.beam.sdk.schemas.utils.FieldValueTypeSupplier;
+import org.apache.beam.sdk.schemas.utils.JavaBeanUtils;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
+import org.apache.beam.sdk.util.common.ReflectHelpers;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.DynamicType;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.Implementation;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.Assigner;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
+import 

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362375=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362375
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 23/Dec/19 01:12
Start Date: 23/Dec/19 01:12
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10449: [BEAM-7274] 
Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360741862
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoByteBuddyUtils.java
 ##
 @@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getFieldNumber;
+
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Duration;
+import com.google.protobuf.Internal.EnumLite;
+import com.google.protobuf.MessageLite;
+import com.google.protobuf.ProtocolMessageEnum;
+import com.google.protobuf.Timestamp;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.lang.reflect.Type;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaLogicalTypes.DurationNanos;
+import 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaLogicalTypes.TimestampNanos;
+import org.apache.beam.sdk.schemas.FieldValueGetter;
+import org.apache.beam.sdk.schemas.FieldValueSetter;
+import org.apache.beam.sdk.schemas.FieldValueTypeInformation;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.schemas.SchemaUserTypeCreator;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType.Value;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertType;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForGetter;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForSetter;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.InjectPackageStrategy;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.TypeConversion;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.TypeConversionsFactory;
+import org.apache.beam.sdk.schemas.utils.FieldValueTypeSupplier;
+import org.apache.beam.sdk.schemas.utils.JavaBeanUtils;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
+import org.apache.beam.sdk.util.common.ReflectHelpers;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.DynamicType;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.Implementation;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.Assigner;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
+import 

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362367=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362367
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 23/Dec/19 01:11
Start Date: 23/Dec/19 01:11
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10449: [BEAM-7274] 
Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360741774
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
 ##
 @@ -94,7 +94,6 @@
   new ForLoadedType(ReadableInstant.class);
   private static final ForLoadedType READABLE_PARTIAL_TYPE =
   new ForLoadedType(ReadablePartial.class);
-  private static final ForLoadedType OBJECT_TYPE = new 
ForLoadedType(Object.class);
 
 Review comment:
   It was unused.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362367)
Time Spent: 19h 40m  (was: 19.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 19h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362293
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 22/Dec/19 13:03
Start Date: 22/Dec/19 13:03
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10449: 
[BEAM-7274] Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360701462
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowWithGetters.java
 ##
 @@ -122,6 +123,15 @@ private Iterable getIterableValue(FieldType elementType, 
Object fieldValue) {
   cacheKey, i -> getMapValue(type.getMapKeyType(), 
type.getMapValueType(), map))
   : (T) getMapValue(type.getMapKeyType(), type.getMapValueType(), map);
 } else {
+  if (type.isLogicalType(OneOfType.IDENTIFIER)) {
+OneOfType oneOfType = type.getLogicalType(OneOfType.class);
+OneOfType.Value oneOfValue = (OneOfType.Value) fieldValue;
+Object convertedOneOfField =
+getValue(oneOfValue.getFieldType(), oneOfValue.getValue(), null);
+return (T)
 
 Review comment:
   Will this not generate a different type for each message. If conversion 
happens here as a user of the getValue you don't have a way to see what the 
effective type and field was. Why not return the LogicalTypes? 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362293)
Time Spent: 19h 10m  (was: 19h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 19h 10m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362290=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362290
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 22/Dec/19 13:03
Start Date: 22/Dec/19 13:03
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10449: 
[BEAM-7274] Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360702129
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoByteBuddyUtils.java
 ##
 @@ -0,0 +1,573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getFieldNumber;
+
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Duration;
+import com.google.protobuf.Internal.EnumLite;
+import com.google.protobuf.MessageLite;
+import com.google.protobuf.ProtocolMessageEnum;
+import com.google.protobuf.Timestamp;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.lang.reflect.Type;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaLogicalTypes.DurationNanos;
+import 
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaLogicalTypes.TimestampNanos;
+import org.apache.beam.sdk.schemas.FieldValueGetter;
+import org.apache.beam.sdk.schemas.FieldValueSetter;
+import org.apache.beam.sdk.schemas.FieldValueTypeInformation;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.schemas.SchemaUserTypeCreator;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType.Value;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertType;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForGetter;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForSetter;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.InjectPackageStrategy;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.TypeConversion;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.TypeConversionsFactory;
+import org.apache.beam.sdk.schemas.utils.FieldValueTypeSupplier;
+import org.apache.beam.sdk.schemas.utils.JavaBeanUtils;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
+import org.apache.beam.sdk.util.common.ReflectHelpers;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.DynamicType;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.Implementation;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.Assigner;
+import 
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
+import 

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362289=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362289
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 22/Dec/19 13:03
Start Date: 22/Dec/19 13:03
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10449: 
[BEAM-7274] Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360699411
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
 ##
 @@ -94,7 +94,6 @@
   new ForLoadedType(ReadableInstant.class);
   private static final ForLoadedType READABLE_PARTIAL_TYPE =
   new ForLoadedType(ReadablePartial.class);
-  private static final ForLoadedType OBJECT_TYPE = new 
ForLoadedType(Object.class);
 
 Review comment:
   This is probably intensional, but where is OBJECT_TYPE?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362289)
Time Spent: 18.5h  (was: 18h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 18.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362295=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362295
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 22/Dec/19 13:03
Start Date: 22/Dec/19 13:03
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10449: 
[BEAM-7274] Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360701604
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FromRowUsingCreator.java
 ##
 @@ -127,6 +123,22 @@ public T apply(Row row) {
   valueType,
   typeFactory);
 } else {
+  if (type.getTypeName().isLogicalType()
+  && 
OneOfType.IDENTIFIER.equals(type.getLogicalType().getIdentifier())) {
 
 Review comment:
   Same comment as with RowWithGetters... should we handle the OneOf logical 
type as a special case?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362295)
Time Spent: 19h 20m  (was: 19h 10m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 19h 20m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362292
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 22/Dec/19 13:03
Start Date: 22/Dec/19 13:03
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10449: 
[BEAM-7274] Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360700268
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoRecordSchema.java
 ##
 @@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoByteBuddyUtils.getProtoGetter;
+
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import java.lang.reflect.Method;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import 
org.apache.beam.sdk.extensions.protobuf.ProtoByteBuddyUtils.ProtoTypeConversionsFactory;
+import org.apache.beam.sdk.schemas.FieldValueGetter;
+import org.apache.beam.sdk.schemas.FieldValueTypeInformation;
+import org.apache.beam.sdk.schemas.GetterBasedSchemaProvider;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.SchemaUserTypeCreator;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType;
+import org.apache.beam.sdk.schemas.utils.FieldValueTypeSupplier;
+import org.apache.beam.sdk.schemas.utils.JavaBeanUtils;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+
+@Experimental(Kind.SCHEMAS)
+public class ProtoRecordSchema extends GetterBasedSchemaProvider {
 
 Review comment:
   Funky naming, looks like spilled over from Avro implementation. Wouldn't 
ProtoMessageSchema be better?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362292)
Time Spent: 19h  (was: 18h 50m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 19h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362296=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362296
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 22/Dec/19 13:03
Start Date: 22/Dec/19 13:03
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10449: 
[BEAM-7274] Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360702389
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoRecordSchema.java
 ##
 @@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static 
org.apache.beam.sdk.extensions.protobuf.ProtoByteBuddyUtils.getProtoGetter;
+
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import java.lang.reflect.Method;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import 
org.apache.beam.sdk.extensions.protobuf.ProtoByteBuddyUtils.ProtoTypeConversionsFactory;
+import org.apache.beam.sdk.schemas.FieldValueGetter;
+import org.apache.beam.sdk.schemas.FieldValueTypeInformation;
+import org.apache.beam.sdk.schemas.GetterBasedSchemaProvider;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.SchemaUserTypeCreator;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType;
+import org.apache.beam.sdk.schemas.utils.FieldValueTypeSupplier;
+import org.apache.beam.sdk.schemas.utils.JavaBeanUtils;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+
+@Experimental(Kind.SCHEMAS)
+public class ProtoRecordSchema extends GetterBasedSchemaProvider {
 
 Review comment:
   ProtoRecordSchema seems like a funky name, it's neither Beam nor Proto. 
Seems like Avro naming.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362296)
Time Spent: 19.5h  (was: 19h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 19.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362294
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 22/Dec/19 13:03
Start Date: 22/Dec/19 13:03
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10449: 
[BEAM-7274] Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360701094
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowWithGetters.java
 ##
 @@ -122,6 +123,15 @@ private Iterable getIterableValue(FieldType elementType, 
Object fieldValue) {
   cacheKey, i -> getMapValue(type.getMapKeyType(), 
type.getMapValueType(), map))
   : (T) getMapValue(type.getMapKeyType(), type.getMapValueType(), map);
 } else {
+  if (type.isLogicalType(OneOfType.IDENTIFIER)) {
 
 Review comment:
   Should we handle the oneOf (or UnionType) separately from all other 
LogicalTypes. I _think_ we decided to leave Enums also out for now. I think 
it's best that we handle this separate from the Proto PR.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362294)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 19h 10m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362291
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 22/Dec/19 13:03
Start Date: 22/Dec/19 13:03
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10449: 
[BEAM-7274] Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449#discussion_r360702210
 
 

 ##
 File path: 
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
 ##
 @@ -0,0 +1,55 @@
+package org.apache.beam.sdk.extensions.protobuf;
+
+import com.google.protobuf.DynamicMessage;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.FieldValueGetter;
+import org.apache.beam.sdk.schemas.FieldValueTypeInformation;
+import org.apache.beam.sdk.schemas.GetterBasedSchemaProvider;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaProvider;
+import org.apache.beam.sdk.schemas.SchemaUserTypeCreator;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+
+import javax.annotation.Nullable;
+import java.util.List;
+
+@Experimental(Kind.SCHEMAS)
+public class ProtoDynamicMessageSchema extends GetterBasedSchemaProvider {
+private static final TypeDescriptor 
DYNAMIC_MESSAGE_TYPE_DESCRIPTOR =
+TypeDescriptor.of(DynamicMessage.class);
+@Nullable
+@Override
+public  Schema schemaFor(TypeDescriptor typeDescriptor) {
+checkForDynamicType(typeDescriptor);
+return ProtoSchemaTranslator.getSchema((Class) 
typeDescriptor.getRawType());
+}
+
+@Override
+public List fieldValueGetters(Class targetClass, 
Schema schema) {
+return null;
 
 Review comment:
   I would throw RuntimeException with "Not Implemented" for now.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362291)
Time Spent: 18h 50m  (was: 18h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 18h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362185=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362185
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 21/Dec/19 19:45
Start Date: 21/Dec/19 19:45
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10449: [BEAM-7274] 
Implement the Protobuf schema provider for compiled protocol buffers
URL: https://github.com/apache/beam/pull/10449
 
 
   Since compiled protocol buffers have efficient generated getters and 
builders, we reuse the ByteBuddy framework to create lazy rows that delegate to 
those methods.
   
   Support for DynamicMessage is in a followon PR.
   
   R: @alexvanboxel 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362185)
Time Spent: 18h 20m  (was: 18h 10m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 18h 20m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362168=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362168
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 21/Dec/19 18:32
Start Date: 21/Dec/19 18:32
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10356: [BEAM-7274] 
Infer a Beam Schema from a protocol buffer class.
URL: https://github.com/apache/beam/pull/10356
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362168)
Time Spent: 18h 10m  (was: 18h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 18h 10m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362099=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362099
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 21/Dec/19 09:02
Start Date: 21/Dec/19 09:02
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10356: [BEAM-7274] 
Infer a Beam Schema from a protocol buffer class.
URL: https://github.com/apache/beam/pull/10356#issuecomment-568165271
 
 
   > @alexvanboxel let me know if you have more thoughts here or if this looks 
good.
   > 
   > One more comment - once your options work is in, we should switch my use 
of field metadata over to the structured options approach.
   
   I've done a second pass, to verify use of logicaltypes and schema. I've 
added a note in my todo list for the serializeLogicalType boolean for later so 
we don't forget.
   
   **LGTM**
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362099)
Time Spent: 18h  (was: 17h 50m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 18h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362009=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362009
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 21/Dec/19 01:44
Start Date: 21/Dec/19 01:44
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10356: [BEAM-7274] Infer 
a Beam Schema from a protocol buffer class.
URL: https://github.com/apache/beam/pull/10356#issuecomment-568141194
 
 
   @alexvanboxel let me know if you have more thoughts here or if this looks 
good.
   
   One more comment - once your options work is in, we should switch my use of 
field metadata over to the structured options approach.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362009)
Time Spent: 17h 50m  (was: 17h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 17h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=362005=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362005
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 21/Dec/19 01:41
Start Date: 21/Dec/19 01:41
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10356: [BEAM-7274] Infer 
a Beam Schema from a protocol buffer class.
URL: https://github.com/apache/beam/pull/10356#issuecomment-568140963
 
 
   You are correct, that this requires other language to implement this parsing 
as well. However I think the visibility advantage of having a fully-represented 
proto (v.s. just embedding a bytes field in a proto) is worth that tax - and it 
shouldn't be a huge tax on Beam SDKs (it only took me about 30-40 minutes to 
write the code here)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362005)
Time Spent: 17.5h  (was: 17h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 17.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=361178=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361178
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 17/Dec/19 22:49
Start Date: 17/Dec/19 22:49
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10356: [BEAM-7274] Infer 
a Beam Schema from a protocol buffer class.
URL: https://github.com/apache/beam/pull/10356#issuecomment-566784391
 
 
   Friendly ping. @alexvanboxel do you have any thoughts on this PR?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 361178)
Time Spent: 17h 20m  (was: 17h 10m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 17h 20m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=358235=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-358235
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 12/Dec/19 01:46
Start Date: 12/Dec/19 01:46
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10356: [BEAM-7274] Infer 
a Beam Schema from a protocol buffer class.
URL: https://github.com/apache/beam/pull/10356#issuecomment-564812504
 
 
   Run Python PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 358235)
Time Spent: 17h 10m  (was: 17h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 17h 10m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=358038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-358038
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 11/Dec/19 19:06
Start Date: 11/Dec/19 19:06
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10356: [BEAM-7274] Infer 
a Beam Schema from a protocol buffer class.
URL: https://github.com/apache/beam/pull/10356#issuecomment-564687496
 
 
   R: @alexvanboxel 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 358038)
Time Spent: 17h  (was: 16h 50m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 17h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=358037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-358037
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 11/Dec/19 19:06
Start Date: 11/Dec/19 19:06
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10356: [BEAM-7274] 
Infer a Beam Schema from a protocol buffer class.
URL: https://github.com/apache/beam/pull/10356
 
 
   This utility infers a Beam schema from a protocol buffer type. LogicalType 
is used for types that not natively represented in Beam schemas.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 358037)
Time Spent: 16h 50m  (was: 16h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 16h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=354701=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354701
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 05/Dec/19 21:24
Start Date: 05/Dec/19 21:24
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10278: [BEAM-7274] 
Support recursive type transformation in ByteBuddyUtils
URL: https://github.com/apache/beam/pull/10278
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 354701)
Time Spent: 16h 40m  (was: 16.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 16h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=354666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354666
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 05/Dec/19 20:48
Start Date: 05/Dec/19 20:48
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10278: [BEAM-7274] 
Support recursive type transformation in ByteBuddyUtils
URL: https://github.com/apache/beam/pull/10278#issuecomment-562311246
 
 
   run sql postcommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 354666)
Time Spent: 16.5h  (was: 16h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 16.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=354610=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354610
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 05/Dec/19 19:17
Start Date: 05/Dec/19 19:17
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10278: [BEAM-7274] 
Support recursive type transformation in ByteBuddyUtils
URL: https://github.com/apache/beam/pull/10278#discussion_r354498969
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ReflectUtils.java
 ##
 @@ -33,16 +36,19 @@
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.schemas.Schema;
 import org.apache.beam.sdk.schemas.annotations.SchemaCreate;
+import org.apache.beam.sdk.values.TypeDescriptor;
 import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
 import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Primitives;
 
 /** A set of reflection helper methods. */
 public class ReflectUtils {
-  static class ClassWithSchema {
+  /** Represents a class and a schema. */
+  public static class ClassWithSchema {
 
 Review comment:
   It's actually public because we need to use this in the protobuf schema 
provider which is in a different package (and in general, you should be able to 
put schema providers in other packages).
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 354610)
Time Spent: 16h 20m  (was: 16h 10m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 16h 20m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=354607=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354607
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 05/Dec/19 19:08
Start Date: 05/Dec/19 19:08
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10278: [BEAM-7274] 
Support recursive type transformation in ByteBuddyUtils
URL: https://github.com/apache/beam/pull/10278#issuecomment-562270149
 
 
   In general this LGTM. I see optimizations for the List vs Collection as List 
will be the most common use case. If I remember correctly the instanceof has 
minimal impact.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 354607)
Time Spent: 16h 10m  (was: 16h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 16h 10m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=354587=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354587
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10278: 
[BEAM-7274] Support recursive type transformation in ByteBuddyUtils
URL: https://github.com/apache/beam/pull/10278#discussion_r354493498
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ReflectUtils.java
 ##
 @@ -33,16 +36,19 @@
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.schemas.Schema;
 import org.apache.beam.sdk.schemas.annotations.SchemaCreate;
+import org.apache.beam.sdk.values.TypeDescriptor;
 import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
 import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Primitives;
 
 /** A set of reflection helper methods. */
 public class ReflectUtils {
-  static class ClassWithSchema {
+  /** Represents a class and a schema. */
+  public static class ClassWithSchema {
 
 Review comment:
   I assume it has a reason that they are now public? (was package protected)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 354587)
Time Spent: 16h  (was: 15h 50m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 16h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=354044=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354044
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 05/Dec/19 05:15
Start Date: 05/Dec/19 05:15
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10247: [BEAM-7274] 
In preparation for protocol-buffer schemas, add OneOf and Enumeration logical 
types
URL: https://github.com/apache/beam/pull/10247
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 354044)
Time Spent: 15h 50m  (was: 15h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 15h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=353954=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353954
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 05/Dec/19 00:42
Start Date: 05/Dec/19 00:42
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-561913183
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 353954)
Time Spent: 15h 40m  (was: 15.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 15h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=353836=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353836
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 04/Dec/19 21:56
Start Date: 04/Dec/19 21:56
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-561857396
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 353836)
Time Spent: 15.5h  (was: 15h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 15.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=353764=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353764
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 04/Dec/19 20:48
Start Date: 04/Dec/19 20:48
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-561832168
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 353764)
Time Spent: 15h 20m  (was: 15h 10m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 15h 20m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=353664=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353664
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 04/Dec/19 19:02
Start Date: 04/Dec/19 19:02
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-561791532
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 353664)
Time Spent: 15h 10m  (was: 15h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 15h 10m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=353567=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353567
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 04/Dec/19 16:38
Start Date: 04/Dec/19 16:38
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-561727277
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 353567)
Time Spent: 15h  (was: 14h 50m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 15h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=352977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352977
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 03/Dec/19 21:10
Start Date: 03/Dec/19 21:10
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-561358176
 
 
   > Right now we don't support evolving schemas. There is a plan (that needs 
to be discussed on the dev list for this): but essentially it would require the 
new pipeline to preserve the old job's ordinal values as much as possible, 
instead of simply taking them in order.
   
   OK, then it's not an issue for now.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 352977)
Time Spent: 14h 50m  (was: 14h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 14h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=352970=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352970
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 03/Dec/19 20:56
Start Date: 03/Dec/19 20:56
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10248: [BEAM-7274] 
Add type conversions factory
URL: https://github.com/apache/beam/pull/10248
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 352970)
Time Spent: 14h 40m  (was: 14.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 14h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=352969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352969
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 03/Dec/19 20:55
Start Date: 03/Dec/19 20:55
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10248: [BEAM-7274] Add 
type conversions factory
URL: https://github.com/apache/beam/pull/10248#issuecomment-561352764
 
 
   LGTM.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 352969)
Time Spent: 14.5h  (was: 14h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 14.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=352940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352940
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 03/Dec/19 19:50
Start Date: 03/Dec/19 19:50
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-561328331
 
 
   Right now we don't support evolving schemas. There is a plan (that needs to 
be discussed on the dev list for this): but essentially it would require the 
new pipeline to preserve the old job's ordinal values as much as possible, 
instead of simply taking them in order.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 352940)
Time Spent: 14h 20m  (was: 14h 10m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 14h 20m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=352877=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352877
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 03/Dec/19 18:47
Start Date: 03/Dec/19 18:47
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-561303499
 
 
   OK, it took a while... but I've wanted to verify something. I've been 
experimenting with how Avro compiles to Java code and how the ordinal value is 
created. I just wanted to check out the following scenario:
   
   ` { "name": "foo", "type": { "name": "FooEnum", "type": "enum", "symbols": 
["x","y"] }},`
   
   we have a schema evolution to:
   
   ` { "name": "foo", "type": { "name": "FooEnum", "type": "enum", "symbols": 
["a", "x", "y"] }},`
   
   we **upgrade** the pipeline, so the current state is stored. As the ordinal 
value is stored in the intermediate state and the schema is evolved by 
**pre**-pending a enum value, that means the ordinal values changed (as it's 
sequential as defined in the source file: from `x=0 , y=1` > `a=0, x=1, y=2` ).
   
   I do notice the same holds through to the Java Enum type. Not to **Proto** 
as it has an explicit number value assigned to an Enum string.
   
   _I don't know if it's an issue... but I wanted to raise it so you're aware 
of the side effect_. Is a pipeline even upgradable when the schema changes?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 352877)
Time Spent: 14h 10m  (was: 14h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 14h 10m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=352167=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352167
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 02/Dec/19 19:59
Start Date: 02/Dec/19 19:59
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10248: [BEAM-7274] Add 
type conversions factory
URL: https://github.com/apache/beam/pull/10248#issuecomment-560556486
 
 
   @alexvanboxel Thanks! This is the first of two changes to the byte-buddy 
codegen necessary to support protos (the second one was a design flaw/bug that 
was not noticed until I tried to hook up protos).
   
   The byte-buddy codegen is a bit much to absorb in a single review. If the 
review is taking more time than you want to spend, I'd also be happy if you 
simply review for obvious coding errors - no need to try and fully learn the 
whole system to review this PR.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 352167)
Time Spent: 14h  (was: 13h 50m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 14h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=352128=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352128
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 02/Dec/19 19:37
Start Date: 02/Dec/19 19:37
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-560547987
 
 
   @alexvanboxel I added Avro support for enums, and have now fixed CheckStyle 
errors. Let me know how this looks to you.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 352128)
Time Spent: 13h 50m  (was: 13h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 13h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=352096=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352096
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 02/Dec/19 18:44
Start Date: 02/Dec/19 18:44
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10248: [BEAM-7274] Add 
type conversions factory
URL: https://github.com/apache/beam/pull/10248#issuecomment-560526797
 
 
   Still reviewing... big commit... First look it seems to make the system more 
flexible and extensible.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 352096)
Time Spent: 13h 40m  (was: 13.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 13h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351750=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351750
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 02/Dec/19 05:46
Start Date: 02/Dec/19 05:46
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-560241889
 
 
   Added Enum support to Avro. Adding support for generic Avro unions proved 
complicated (mostly because Avro compiler represents them using an Object 
type), so best left for another PR; also, it appears that most usages of Avro 
unions are to represent nullable types, which we already handle.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351750)
Time Spent: 13.5h  (was: 13h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 13.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351665=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351665
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 01/Dec/19 21:12
Start Date: 01/Dec/19 21:12
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-560159655
 
 
   > Note: fixing the Avro issue requires refactoring a bit. This refactor I 
already have in another PR as protos require it, so I'll go send that out now.
   
   I was indeed wondering how this would Avro would be handled. For me it looks 
all good.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351665)
Time Spent: 13h 20m  (was: 13h 10m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 13h 20m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351655=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351655
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 01/Dec/19 19:58
Start Date: 01/Dec/19 19:58
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-560151944
 
 
   Note: fixing the Avro issue requires refactoring a bit. This refactor I 
already have in another PR as protos require it, so I'll go send that out now.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351655)
Time Spent: 13h 10m  (was: 13h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 13h 10m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351603=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351603
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 01/Dec/19 00:28
Start Date: 01/Dec/19 00:28
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-560033450
 
 
   Run Python2_PVR_Flink PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351603)
Time Spent: 13h  (was: 12h 50m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 13h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351599=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351599
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 01/Dec/19 00:00
Start Date: 01/Dec/19 00:00
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-560031651
 
 
   Also added support for enums in the java type mapping. This means that POJOs 
or JavaBeans (or AutoValues, or any similar type) with Java enums will map to 
the new EnumerationType.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351599)
Time Spent: 12h 50m  (was: 12h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 12h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351598=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351598
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 30/Nov/19 23:58
Start Date: 30/Nov/19 23:58
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10247: [BEAM-7274] 
In preparation for protocol-buffer schemas, add OneOf and Enumeration logical 
types
URL: https://github.com/apache/beam/pull/10247#discussion_r352310964
 
 

 ##
 File path: sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
 ##
 @@ -208,6 +210,26 @@ public Boolean getBoolean(String fieldName) {
 return getMap(getSchema().indexOf(fieldName));
   }
 
+  /** Helper method to get an enum field value. */
+  public EnumerationValue getEnumValue(String fieldValue) {
 
 Review comment:
   IMO unions are pretty basic. However removed these methods for now, as it's 
not difficult to simply access as a logical type.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351598)
Time Spent: 12h 40m  (was: 12.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 12h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351591=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351591
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 30/Nov/19 21:42
Start Date: 30/Nov/19 21:42
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10247: 
[BEAM-7274] In preparation for protocol-buffer schemas, add OneOf and 
Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#discussion_r352305959
 
 

 ##
 File path: sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
 ##
 @@ -208,6 +210,26 @@ public Boolean getBoolean(String fieldName) {
 return getMap(getSchema().indexOf(fieldName));
   }
 
+  /** Helper method to get an enum field value. */
+  public EnumerationValue getEnumValue(String fieldValue) {
 
 Review comment:
   Mmm.. yes. But we should keep it to the basic well known types then. Enum, 
DateTime they make sense. But the OneOf, is not so common.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351591)
Time Spent: 12.5h  (was: 12h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 12.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351580=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351580
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 30/Nov/19 20:50
Start Date: 30/Nov/19 20:50
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10247: [BEAM-7274] 
In preparation for protocol-buffer schemas, add OneOf and Enumeration logical 
types
URL: https://github.com/apache/beam/pull/10247#discussion_r352303826
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/LogicalTypes.java
 ##
 @@ -111,4 +134,247 @@ public FieldType getBaseType() {
   }
 }
   }
+
+  /**
+   * This class represents a single enum value. It can be referenced as a 
String or as an integer.
+   */
+  public static class EnumerationValue {
+private final String stringValue;
+private final int value;
+
+EnumerationValue(String stringValue, int value) {
+  this.stringValue = stringValue;
+  this.value = value;
+}
+
+/** Return the integer enum value. */
+int getValue() {
+  return value;
+}
+
+/** Return the String enum value. */
+@Override
+public String toString() {
+  return stringValue;
+}
+
+@Override
+public boolean equals(Object o) {
+  if (this == o) {
+return true;
+  }
+  if (o == null || getClass() != o.getClass()) {
+return false;
+  }
+  EnumerationValue enumValue = (EnumerationValue) o;
+  return value == enumValue.value && Objects.equals(stringValue, 
enumValue.stringValue);
+}
+
+@Override
+public int hashCode() {
+  return Objects.hash(stringValue, value);
+}
+  }
+
+  /** This {@link LogicalType} represent an enumeration over a fixed set of 
values. */
+  public static class EnumerationType implements LogicalType {
+public static final String IDENTIFIER = "Enum";
+final BiMap enumValues = HashBiMap.create();
+
+private EnumerationType(Map enumValues) {
+  this.enumValues.putAll(enumValues);
+}
+
+/** Create an enumeration type over a set of String->Integer values. */
+public static EnumerationType create(Map enumValues) {
+  return new EnumerationType(enumValues);
+}
+
+/**
+ * Create an enumeration type from a fixed set of String values; integer 
values will be
+ * automatically chosen.
+ */
+public static EnumerationType create(List enumValues) {
+  return new EnumerationType(
+  IntStream.range(0, enumValues.size())
+  .boxed()
+  .collect(Collectors.toMap(i -> enumValues.get(i), i -> i)));
+}
+
+/** Return an {@link EnumerationValue} corresponding to one of the 
enumeration strings. */
+public EnumerationValue valueOf(String stringValue) {
+  return new EnumerationValue(stringValue, enumValues.get(stringValue));
+}
+
+/**
+ * Return an {@link EnumerationValue} corresponding to one of the 
enumeration integer values.
+ */
+public EnumerationValue valueOf(int value) {
+  return new EnumerationValue(enumValues.inverse().get(value), value);
+}
+
+@Override
+public String getIdentifier() {
+  return IDENTIFIER;
+}
+
+@Override
+public FieldType getArgumentType() {
+  return FieldType.map(FieldType.STRING, FieldType.INT32);
+}
+
+@Override
+public Map getArgument() {
+  return enumValues;
+}
+
+@Override
+public FieldType getBaseType() {
+  return FieldType.INT32;
+}
+
+@Override
+public Integer toBaseType(EnumerationValue input) {
+  return input.value;
+}
+
+@Override
+public EnumerationValue toInputType(Integer base) {
+  return valueOf(base);
+}
+  }
+
+  /**
+   * Represents a single OneOf value. Each object contains an {@link 
EnumerationValue} specifying
+   * which field is set along with the value of that field.
+   */
+  public static class OneOfValue {
+private EnumerationValue caseType;
+private Object value;
+
+OneOfValue(EnumerationValue caseType, Object value) {
+  this.caseType = caseType;
+  this.value = value;
+}
+
+/** Returns the enumeration that specified which OneOf field is set. */
+public EnumerationValue getCaseType() {
+  return caseType;
+}
+
+/** Returns the current value of the OneOf. */
+@SuppressWarnings("TypeParameterUnusedInFormals")
+public  T getValue() {
+  return (T) value;
+}
+  }
+  /**
+   * A logical type representing a union of fields. This logical type is 
initialized with a set of
+   * field and represents a union of those fields. This logical type is backed 
by a Row object
+   * containing one nullable field matching each input field, and one 
additional {@link
+   * EnumerationType} logical type field 

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351579=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351579
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 30/Nov/19 20:50
Start Date: 30/Nov/19 20:50
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10247: [BEAM-7274] 
In preparation for protocol-buffer schemas, add OneOf and Enumeration logical 
types
URL: https://github.com/apache/beam/pull/10247#discussion_r352303817
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/LogicalTypes.java
 ##
 @@ -111,4 +134,247 @@ public FieldType getBaseType() {
   }
 }
   }
+
+  /**
+   * This class represents a single enum value. It can be referenced as a 
String or as an integer.
+   */
+  public static class EnumerationValue {
+private final String stringValue;
+private final int value;
+
+EnumerationValue(String stringValue, int value) {
+  this.stringValue = stringValue;
+  this.value = value;
+}
+
+/** Return the integer enum value. */
+int getValue() {
+  return value;
+}
+
+/** Return the String enum value. */
+@Override
+public String toString() {
+  return stringValue;
+}
+
+@Override
+public boolean equals(Object o) {
+  if (this == o) {
+return true;
+  }
+  if (o == null || getClass() != o.getClass()) {
+return false;
+  }
+  EnumerationValue enumValue = (EnumerationValue) o;
+  return value == enumValue.value && Objects.equals(stringValue, 
enumValue.stringValue);
+}
+
+@Override
+public int hashCode() {
+  return Objects.hash(stringValue, value);
+}
+  }
+
+  /** This {@link LogicalType} represent an enumeration over a fixed set of 
values. */
+  public static class EnumerationType implements LogicalType {
 
 Review comment:
   good idea - done
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351579)
Time Spent: 12h 10m  (was: 12h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 12h 10m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351561=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351561
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 30/Nov/19 19:52
Start Date: 30/Nov/19 19:52
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-560015890
 
 
   run java precommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351561)
Time Spent: 12h  (was: 11h 50m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 12h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351560=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351560
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 30/Nov/19 19:51
Start Date: 30/Nov/19 19:51
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10247: [BEAM-7274] 
In preparation for protocol-buffer schemas, add OneOf and Enumeration logical 
types
URL: https://github.com/apache/beam/pull/10247#discussion_r352301326
 
 

 ##
 File path: sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
 ##
 @@ -208,6 +210,26 @@ public Boolean getBoolean(String fieldName) {
 return getMap(getSchema().indexOf(fieldName));
   }
 
+  /** Helper method to get an enum field value. */
+  public EnumerationValue getEnumValue(String fieldValue) {
 
 Review comment:
   See above (also keep in mind that we want to make DATETIME a logical type as 
well). However I don't feel strongly one way or the other about this.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351560)
Time Spent: 11h 50m  (was: 11h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 11h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351559=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351559
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 30/Nov/19 19:51
Start Date: 30/Nov/19 19:51
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10247: [BEAM-7274] 
In preparation for protocol-buffer schemas, add OneOf and Enumeration logical 
types
URL: https://github.com/apache/beam/pull/10247#discussion_r352301313
 
 

 ##
 File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
 ##
 @@ -181,6 +183,14 @@ public Builder addBooleanField(String name) {
   return this;
 }
 
+public Builder addEnumerationField(String name, EnumerationType 
enumerationType) {
 
 Review comment:
   Originally I left them out. But then decided that these are fairly "basic" 
logical types (e.g. they exist in proto, avro, etc.) so it was worth adding 
this method as syntactic sugar. I don't feel very strongly about it either way 
though.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351559)
Time Spent: 11h 40m  (was: 11.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 11h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351548=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351548
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 30/Nov/19 18:52
Start Date: 30/Nov/19 18:52
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10247: 
[BEAM-7274] In preparation for protocol-buffer schemas, add OneOf and 
Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#discussion_r352298646
 
 

 ##
 File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
 ##
 @@ -181,6 +183,14 @@ public Builder addBooleanField(String name) {
   return this;
 }
 
+public Builder addEnumerationField(String name, EnumerationType 
enumerationType) {
 
 Review comment:
   Does it make sense to add these methods as first class add'ers to Schema? 
This feels like breaking consistency... as they are logical types.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351548)
Time Spent: 11.5h  (was: 11h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 11.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351546=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351546
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 30/Nov/19 18:52
Start Date: 30/Nov/19 18:52
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10247: 
[BEAM-7274] In preparation for protocol-buffer schemas, add OneOf and 
Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#discussion_r352298426
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/LogicalTypes.java
 ##
 @@ -111,4 +134,247 @@ public FieldType getBaseType() {
   }
 }
   }
+
+  /**
+   * This class represents a single enum value. It can be referenced as a 
String or as an integer.
+   */
+  public static class EnumerationValue {
+private final String stringValue;
+private final int value;
+
+EnumerationValue(String stringValue, int value) {
+  this.stringValue = stringValue;
+  this.value = value;
+}
+
+/** Return the integer enum value. */
+int getValue() {
+  return value;
+}
+
+/** Return the String enum value. */
+@Override
+public String toString() {
+  return stringValue;
+}
+
+@Override
+public boolean equals(Object o) {
+  if (this == o) {
+return true;
+  }
+  if (o == null || getClass() != o.getClass()) {
+return false;
+  }
+  EnumerationValue enumValue = (EnumerationValue) o;
+  return value == enumValue.value && Objects.equals(stringValue, 
enumValue.stringValue);
+}
+
+@Override
+public int hashCode() {
+  return Objects.hash(stringValue, value);
+}
+  }
+
+  /** This {@link LogicalType} represent an enumeration over a fixed set of 
values. */
+  public static class EnumerationType implements LogicalType {
+public static final String IDENTIFIER = "Enum";
+final BiMap enumValues = HashBiMap.create();
+
+private EnumerationType(Map enumValues) {
+  this.enumValues.putAll(enumValues);
+}
+
+/** Create an enumeration type over a set of String->Integer values. */
+public static EnumerationType create(Map enumValues) {
+  return new EnumerationType(enumValues);
+}
+
+/**
+ * Create an enumeration type from a fixed set of String values; integer 
values will be
+ * automatically chosen.
+ */
+public static EnumerationType create(List enumValues) {
+  return new EnumerationType(
+  IntStream.range(0, enumValues.size())
+  .boxed()
+  .collect(Collectors.toMap(i -> enumValues.get(i), i -> i)));
+}
+
+/** Return an {@link EnumerationValue} corresponding to one of the 
enumeration strings. */
+public EnumerationValue valueOf(String stringValue) {
+  return new EnumerationValue(stringValue, enumValues.get(stringValue));
+}
+
+/**
+ * Return an {@link EnumerationValue} corresponding to one of the 
enumeration integer values.
+ */
+public EnumerationValue valueOf(int value) {
+  return new EnumerationValue(enumValues.inverse().get(value), value);
+}
+
+@Override
+public String getIdentifier() {
+  return IDENTIFIER;
+}
+
+@Override
+public FieldType getArgumentType() {
+  return FieldType.map(FieldType.STRING, FieldType.INT32);
+}
+
+@Override
+public Map getArgument() {
+  return enumValues;
+}
+
+@Override
+public FieldType getBaseType() {
+  return FieldType.INT32;
+}
+
+@Override
+public Integer toBaseType(EnumerationValue input) {
+  return input.value;
+}
+
+@Override
+public EnumerationValue toInputType(Integer base) {
+  return valueOf(base);
+}
+  }
+
+  /**
+   * Represents a single OneOf value. Each object contains an {@link 
EnumerationValue} specifying
+   * which field is set along with the value of that field.
+   */
+  public static class OneOfValue {
+private EnumerationValue caseType;
+private Object value;
+
+OneOfValue(EnumerationValue caseType, Object value) {
+  this.caseType = caseType;
+  this.value = value;
+}
+
+/** Returns the enumeration that specified which OneOf field is set. */
+public EnumerationValue getCaseType() {
+  return caseType;
+}
+
+/** Returns the current value of the OneOf. */
+@SuppressWarnings("TypeParameterUnusedInFormals")
+public  T getValue() {
+  return (T) value;
+}
+  }
+  /**
+   * A logical type representing a union of fields. This logical type is 
initialized with a set of
+   * field and represents a union of those fields. This logical type is backed 
by a Row object
+   * containing one nullable field matching each input field, and one 
additional {@link
+   * EnumerationType} logical type 

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351547=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351547
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 30/Nov/19 18:52
Start Date: 30/Nov/19 18:52
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10247: 
[BEAM-7274] In preparation for protocol-buffer schemas, add OneOf and 
Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#discussion_r352279249
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/LogicalTypes.java
 ##
 @@ -111,4 +134,247 @@ public FieldType getBaseType() {
   }
 }
   }
+
+  /**
+   * This class represents a single enum value. It can be referenced as a 
String or as an integer.
+   */
+  public static class EnumerationValue {
+private final String stringValue;
+private final int value;
+
+EnumerationValue(String stringValue, int value) {
+  this.stringValue = stringValue;
+  this.value = value;
+}
+
+/** Return the integer enum value. */
+int getValue() {
+  return value;
+}
+
+/** Return the String enum value. */
+@Override
+public String toString() {
+  return stringValue;
+}
+
+@Override
+public boolean equals(Object o) {
+  if (this == o) {
+return true;
+  }
+  if (o == null || getClass() != o.getClass()) {
+return false;
+  }
+  EnumerationValue enumValue = (EnumerationValue) o;
+  return value == enumValue.value && Objects.equals(stringValue, 
enumValue.stringValue);
+}
+
+@Override
+public int hashCode() {
+  return Objects.hash(stringValue, value);
+}
+  }
+
+  /** This {@link LogicalType} represent an enumeration over a fixed set of 
values. */
+  public static class EnumerationType implements LogicalType {
 
 Review comment:
   I see the collection of LogicalTypes growing. Wouldn't it start to make 
sense to have them in there own package, eg 
`org.apache.beam.sdk.schema.logicaltypes`?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351547)
Time Spent: 11h 20m  (was: 11h 10m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 11h 20m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351549=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351549
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 30/Nov/19 18:52
Start Date: 30/Nov/19 18:52
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #10247: 
[BEAM-7274] In preparation for protocol-buffer schemas, add OneOf and 
Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#discussion_r352298953
 
 

 ##
 File path: sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
 ##
 @@ -208,6 +210,26 @@ public Boolean getBoolean(String fieldName) {
 return getMap(getSchema().indexOf(fieldName));
   }
 
+  /** Helper method to get an enum field value. */
+  public EnumerationValue getEnumValue(String fieldValue) {
 
 Review comment:
   Same remark. Are they first class citizens to merit extra methods?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351549)
Time Spent: 11.5h  (was: 11h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 11.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351492=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351492
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 30/Nov/19 05:19
Start Date: 30/Nov/19 05:19
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-559913841
 
 
   run java precommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351492)
Time Spent: 11h  (was: 10h 50m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 11h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351453=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351453
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 29/Nov/19 21:22
Start Date: 29/Nov/19 21:22
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10247: [BEAM-7274] In 
preparation for protocol-buffer schemas, add OneOf and Enumeration logical types
URL: https://github.com/apache/beam/pull/10247#issuecomment-559882384
 
 
   run sql postcommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351453)
Time Spent: 10h 50m  (was: 10h 40m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 10h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=351441=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351441
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 29/Nov/19 19:25
Start Date: 29/Nov/19 19:25
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10247: [BEAM-7274] 
In preparation for protocol-buffer schemas, add OneOf and Enumeration logical 
types
URL: https://github.com/apache/beam/pull/10247
 
 
   Add new LogicalTypes to represent enumerations and OneOf types. In order to 
implement this, we extended the LogicalType argument to be able to have a 
Schema so that we don't have to encode complicated arguments inside of strings.
   
   R:@alexvanboxel 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 351441)
Time Spent: 10h 40m  (was: 10.5h)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 10h 40m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=346165=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-346165
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 19/Nov/19 18:28
Start Date: 19/Nov/19 18:28
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #8690: [BEAM-7274] 
Implement the Protobuf schema provider
URL: https://github.com/apache/beam/pull/8690#issuecomment-555643921
 
 
   > BTW I've checked out the PR to play with it a bit more. My worry is that 
in its current form "optimizing later" will not be so simple, since it doesn't 
currently fit in with the framework used by AvroSchema, AutoValueSchema, 
JavaBeanSchema, etc.. If you don't mind, I'll try and play with it a bit to see 
if I can get it to fit into that framework a bit better.
   
   I don't mind, the thing that worries me is that it will miss the 2.18 as 
well (as it missed the 2.17 already). As we are depending in our business on 
this feature we are currently building our own version of Beam and that's a 
pain. For me the DynamicMessage is very important... I was thinking for the 
normal use-case going to a complete other codepath.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 346165)
Time Spent: 10.5h  (was: 10h 20m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 10.5h
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=346156=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-346156
 ]

ASF GitHub Bot logged work on BEAM-7274:


Author: ASF GitHub Bot
Created on: 19/Nov/19 18:15
Start Date: 19/Nov/19 18:15
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on pull request #8690: 
[BEAM-7274] Implement the Protobuf schema provider
URL: https://github.com/apache/beam/pull/8690#discussion_r348088141
 
 

 ##
 File path: sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
 ##
 @@ -554,6 +555,12 @@ public Builder withFieldValueGetters(
   return this;
 }
 
+/** The FieldValueGetters will handle the conversion for Arrays, Maps and 
Rows. */
+public Builder withFieldValueGettersHandleCollections(boolean 
collectionHandledByGetter) {
+  this.collectionHandledByGetter = collectionHandledByGetter;
+  return this;
+}
 
 Review comment:
   Would we make a ticket out of this or is this blocking?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 346156)
Time Spent: 10h 20m  (was: 10h 10m)

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
>  Time Spent: 10h 20m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)