robertwb commented on code in PR #32512:
URL: https://github.com/apache/beam/pull/32512#discussion_r1801485731
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BeamRowToStorageApiProto.java:
##########
@@ -272,6 +286,8 @@ private static Object messageValueFromRowValue(
if (value == null) {
if (fieldDescriptor.isOptional()) {
return null;
+ } else if (fieldDescriptor.isRepeated()) {
Review Comment:
Currently we distinguish between the empty list and a missing value. I think
we want to keep that distinction.
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BeamRowToStorageApiProto.java:
##########
@@ -291,9 +307,17 @@ private static Object toProtoValue(
if (arrayElementType == null) {
throw new RuntimeException("Unexpected null element type!");
}
- return list.stream()
- .map(v -> toProtoValue(fieldDescriptor, arrayElementType, v))
- .collect(Collectors.toList());
+ boolean shouldFlatMap =
+ arrayElementType.getTypeName().isCollectionType()
+ || arrayElementType.getTypeName().isMapType();
+
+ Stream<Object> valueStream =
+ list.stream().map(v -> toProtoValue(fieldDescriptor,
arrayElementType, v));
+
+ if (shouldFlatMap) {
+ valueStream = valueStream.flatMap(vs -> ((List) vs).stream());
Review Comment:
Why are we introducing this flattening here?
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BeamRowToStorageApiProto.java:
##########
@@ -243,7 +245,19 @@ private static TableFieldSchema
fieldDescriptorFromBeamField(Field field) {
builder = builder.setType(type);
break;
case MAP:
- throw new RuntimeException("Map types not supported by BigQuery.");
+ @Nullable FieldType keyType = field.getType().getMapKeyType();
+ @Nullable FieldType valueType = field.getType().getMapValueType();
+ if (keyType == null || valueType == null) {
+ throw new RuntimeException("Unexpected null element type!");
Review Comment:
It'd be more informative if the key and value raised distinct errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
