ahmedabu98 opened a new pull request, #34707: URL: https://github.com/apache/beam/pull/34707
tl;dr just adding some missing logic to support converting BQ timestamp type --> Java logical timestamp type --> Python timestamp type We have some existing Schema translation between Python and Java SDKs for Timestamps. A Python Timestamp [maps](https://github.com/apache/beam/blob/73739725633f76e1a960e61171e498df1be9012f/sdks/python/apache_beam/typehints/schemas.py#L36) to a Java MicrosInstant (aka [SqlTypes.TIMESTAMP](https://github.com/apache/beam/blob/73739725633f76e1a960e61171e498df1be9012f/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/SqlTypes.java#L42)) Writing timestamps to BQ Storage Write API with the Python SDK works fine, but when a row fails and we try converting the BQ type back to a MicrosInstant, we get the following error: ``` java.lang.UnsupportedOperationException: Converting BigQuery type 'class java.lang.String' to 'LOGICAL_TYPE<beam:logical_type:micros_instant:v1> NOT NULL' is not supported at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamValue(BigQueryUtils.java:802) at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.lambda$toBeamRow$6(BigQueryUtils.java:693) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.Collections$2.tryAdvance(Collections.java:5073) at java.base/java.util.Collections$2.forEachRemaining(Collections.java:5081) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRow(BigQueryUtils.java:694) ``` We're just missing some logic that accounts for when we want to convert a BQ timestamp to a MicrosInstant. I believe this was neglected because MicrosInstant isn't meant to be used as a Java native type. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
