JoeCMoore commented on code in PR #26889:
URL: https://github.com/apache/beam/pull/26889#discussion_r1305718914
##########
sdks/python/apache_beam/examples/snippets/snippets.py:
##########
@@ -1181,6 +1189,32 @@ def model_bigqueryio(
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED)
# [END model_bigqueryio_write]
+ # [START model_bigqueryio_write_with_storage_write_api]
+ quotes | beam.io.WriteToBigQuery(
+ table_spec,
+ schema=table_schema,
+ method=beam.io.WriteToBigQuery.Method.STORAGE_WRITE_API)
+ # [END model_bigqueryio_write_with_storage_write_api]
Review Comment:
> I looked into the code again and found
[this](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/typehints/schemas.py#L787~L799).
It seems that `beam:logical_type:millis_instant:v1` is translated as
`FieldType.DATETIME` in
[SchemaTranslation](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaTranslation.java#L427),
which is matched with the [output
mapping](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L324).
To use `MillisInstant`, `LogicalType.register_logical_type(MillisInstant)`
should be added to the pipeline code.
>
> **Example**
>
> ```
> LogicalType.register_logical_type(MillisInstant)
> quotes | beam.io.WriteToBigQuery(
> table_spec,
> schema=table_schema,
> write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE,
> create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
> method=beam.io.WriteToBigQuery.Method.STORAGE_WRITE_API,
> )
> ```
Where does `LogicalType.register_logical_type(MillisInstant)` reference
from? I'm trying to find a workaround that's usable for now since this isn't
fixed in v2.49.0. My JSON schema is correctly converted to a
`bigquery.TableSchema` instance but when passing it to apache beam it gives the
error you mentioned above.
In my case the timestamp type is incorrectly translated to datetime
(`LOGICAL_TYPE<beam:logical_type:micros_instant:v1>` ➔ `DATETIME NOT NULL`).
I'd love to help solve this, please let me know of anything I can help with
for testing etc.
##########
sdks/python/apache_beam/examples/snippets/snippets.py:
##########
@@ -1181,6 +1189,32 @@ def model_bigqueryio(
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED)
# [END model_bigqueryio_write]
+ # [START model_bigqueryio_write_with_storage_write_api]
+ quotes | beam.io.WriteToBigQuery(
+ table_spec,
+ schema=table_schema,
+ method=beam.io.WriteToBigQuery.Method.STORAGE_WRITE_API)
+ # [END model_bigqueryio_write_with_storage_write_api]
Review Comment:
> I looked into the code again and found
[this](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/typehints/schemas.py#L787~L799).
It seems that `beam:logical_type:millis_instant:v1` is translated as
`FieldType.DATETIME` in
[SchemaTranslation](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaTranslation.java#L427),
which is matched with the [output
mapping](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L324).
To use `MillisInstant`, `LogicalType.register_logical_type(MillisInstant)`
should be added to the pipeline code.
>
> **Example**
>
> ```
> LogicalType.register_logical_type(MillisInstant)
> quotes | beam.io.WriteToBigQuery(
> table_spec,
> schema=table_schema,
> write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE,
> create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
> method=beam.io.WriteToBigQuery.Method.STORAGE_WRITE_API,
> )
> ```
Where does `LogicalType.register_logical_type(MillisInstant)` reference
from? I'm trying to find a workaround that's usable for now since this isn't
fixed in v2.49.0. My JSON schema is correctly converted to a
`bigquery.TableSchema` instance but when passing it to apache beam it gives the
error you mentioned above.
In my case the timestamp type is incorrectly translated to datetime
(`LOGICAL_TYPE<beam:logical_type:micros_instant:v1>` ➔ `DATETIME NOT NULL`).
I'd love to help solve this, please let me know of anything I can help with
for testing etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
