stankiewicz commented on PR #34102: URL: https://github.com/apache/beam/pull/34102#issuecomment-2694187730
issue I see additional problem here: [StorageApiWriteUnshardedRecords](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java#L587) If we have a row with known fields (name:string, surname:string) and one unknown (foo:string) , this line will concat and not cause schema issues. but if unknown field is part of repeated structure, e.g. (name:string, phone:(repeated struct type:string, num:string) ) and we have unknown in repeated struct (phone: [ null, {favourite:yes} ]) concat doesn't make any sense - should be mix of merge & concat at each level. For current bug, it is resolved - it won't add empty records so we can merge it. Added issue for concatenating unknown nested fields which may require bigger changes #34145 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
