Minbo Bae created BEAM-13753:
--------------------------------
Summary: BigQuery Storage Write API fails with type mismatch error
for DATE type
Key: BEAM-13753
URL: https://issues.apache.org/jira/browse/BEAM-13753
Project: Beam
Issue Type: Bug
Components: io-java-gcp
Reporter: Minbo Bae
If {{DATE}} type is specified in schema, {{BigQueryIO}} with Storage Write API
may fail with the following error:
{quote}Got error com.google.api.gax.rpc.InvalidArgumentException:
io.grpc.StatusRuntimeException: INVALID_ARGUMENT: The proto field mismatched
with BigQuery field at <table>, the proto field type string, BigQuery field
type DATE Entity: <write_stream_name>
...
{quote}
{{DATE}} type in BEAM for BigQuery is converted to {{string}} in proto request
[1], while BigQuery Storage Write API requires an integer type for {{DATE}}
type in BigQuery [2].
A workaround is to use {{INTEGER}} in BEAM and manually compute the number of
days since epoch.
Actually, a feature request to BigQuery is filed at [3]. If it's implemented,
we can use {{DATE}} type successfully.
[1]
[https://github.com/apache/beam/blob/v2.35.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java#L55-L73]
[2] [https://cloud.google.com/bigquery/docs/write-api#data_type_conversions]
[3] [https://issuetracker.google.com/issues/205174128]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
