ahmedabu98 commented on code in PR #25927:
URL: https://github.com/apache/beam/pull/25927#discussion_r1147523858
##########
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/SchemaAwareJavaBeans.java:
##########
@@ -270,6 +400,55 @@ public static SerializableFunction<Row,
DoublyNestedDataTypes> doublyNestedDataT
return
DEFAULT_SCHEMA_PROVIDER.fromRowFunction(DOUBLY_NESTED_DATA_TYPES_TYPE_DESCRIPTOR);
}
+ /**
+ * Contains all primitive Java types supported by Avro. The purpose of this
class is to test
+ * schema-aware PTransforms with flat {@link Schema} {@link Row}s.
+ */
+ @DefaultSchema(AutoValueSchema.class)
+ @AutoValue
+ public abstract static class AvroPrimitiveDataTypes implements Serializable {
Review Comment:
Yeah the problem is Short (int16) and Byte (a single Byte) are not types
that are supported by Avro (see [Avro primitive
types](https://docs.oracle.com/cd/E26161_02/html/GettingStartedGuide/avroschemas.html#:~:text=in%20this%20chapter.-,Primitive%20Data%20Types,-In%20the%20previous))
or Parquet (see [primitive
types](https://parquet.apache.org/docs/file-format/types/)). So actually in
AvroUtils we turn Short and Byte types into Beam Schema types to an Integer
(int32).
Avro and Parquet do support a sequence of bytes though, so I've replaced
Byte with ByteBuffer in `AvroPrimitiveDataTypes`.
>Another alternative would be just to get rid of the Byte field in
AllPrimitiveDataTypes
If you agree, I'd be happy to remove Short and replace Byte with ByteBuffer
in `AllPrimitiveDataTypes` and use that for everything instead (for both read
and write).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
