prodriguezdefino commented on code in PR #22179:
URL: https://github.com/apache/beam/pull/22179#discussion_r1037400402
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BeamRowToStorageApiProto.java:
##########
@@ -248,9 +264,20 @@ private static Object toProtoValue(
if (arrayElementType == null) {
throw new RuntimeException("Unexpected null element type!");
}
- return list.stream()
- .map(v -> toProtoValue(fieldDescriptor, arrayElementType, v))
- .collect(Collectors.toList());
+ Boolean shouldFlatMap =
+ arrayElementType.getTypeName().isCollectionType()
+ || arrayElementType.getTypeName().isMapType()
+ ? true
+ : false;
+
+ Stream<Object> valueStream =
+ list.stream().map(v -> toProtoValue(fieldDescriptor,
arrayElementType, v));
+
+ if (shouldFlatMap) {
+ valueStream = valueStream.flatMap(vs -> ((List) vs).stream());
+ }
Review Comment:
Because BQ does not support arrays of maps, this code is making the decision
of flattening those structures (the if condition is computed based on that
particular scenario).
[
map1 [k1,v2] [k2,v2] ,
map2 [k3,v3],
map3 [k4,v4] [k5,v5],
]
------------------------- to
[
record {key:k1, value:v1},
record {key:k2, value:v2},
record {key:k3, value:v3},
record {key:k4, value:v4},
record {key:k5, value:v5}
]
It respects the order in the array and the inherent order of iteration in
the maps, but it won't check for repeated keys across the maps in the original
array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
