[
https://issues.apache.org/jira/browse/BEAM-10176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367056#comment-17367056
]
Tobias Hermann commented on BEAM-10176:
---------------------------------------
Just ran into the same issue ("Expected Avro schema types [] for BigQuery
STRUCT field metadata, but received RECORD") when using not Python but Java.
In BigQueryAvroUtils.java, there only is
{{.put("RECORD", Type.RECORD)}}
but
{{.put("STRUCT", Type.RECORD)}}
is missing:
https://github.com/apache/beam/blob/cf3f2a4287a7636954286c6c17e8d0823e4c04ee/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java#L84
GoogleCloudPlatform/bigquery-ingest-avro-dataflow-sample
(https://github.com/GoogleCloudPlatform/bigquery-ingest-avro-dataflow-sample/blob/42c32a5bd6084412d2c7abb24e9eaa273f81b321/BeamAvro/src/main/java/com/google/cloud/solutions/beamavro/BigQueryAvroUtils.java#L90)
shows how it could look like.
Could we add it to Apache beam too? That would be awesome! :)
> Support BigQuery type aliases in WriteToBigQuery with Avro format
> -----------------------------------------------------------------
>
> Key: BEAM-10176
> URL: https://issues.apache.org/jira/browse/BEAM-10176
> Project: Beam
> Issue Type: Improvement
> Components: io-py-gcp
> Reporter: Chun Yang
> Assignee: Chun Yang
> Priority: P3
> Fix For: 2.23.0
>
> Original Estimate: 3h
> Time Spent: 1h 40m
> Remaining Estimate: 1h 20m
>
> Support BigQuery type aliases in WriteToBigQuery with avro temp file format.
> The following aliases are missing:
> * {{STRUCT}} ({{==RECORD}})
> * {{FLOAT64}} ({{==FLOAT}})
> * {{INT64}} ({{==INTEGER}})
> Reference:
> https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types
> https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/TableFieldSchema.html#setType-java.lang.String-
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
