[
https://issues.apache.org/jira/browse/BEAM-9613?focusedWorklogId=575822&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-575822
]
ASF GitHub Bot logged work on BEAM-9613:
----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Apr/21 23:35
Start Date: 01/Apr/21 23:35
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on pull request #14350:
URL: https://github.com/apache/beam/pull/14350#issuecomment-812232598
I took a look at the PostCommit failure
(`BigQueryNestedRecordsIT.testNestedRecords`), it's another case where the
expected result contains doubles, and now we're producing strings. However in
this case we're using `BigqueryClient` to read the expected value directly, and
it's producing the value that contains doubles.
It looks like BigqueryClient actually already has logic to convert the
Strings we get from BQ into booleans and doubles:
https://github.com/apache/beam/blob/5ddb69f4bd9fc281c3f401fee87232163ee09ba9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java#L258-L259
(CC: @pabloem since it looks like you added this logic)
So maybe your [original
approach](https://github.com/mouyang/beam/commit/326b291ab333c719a9f54446c34611581ea696eb)
is actually correct?
If that's the case I would expect reading beam rows to fail with JSON as
well as Avro, is there a test for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 575822)
Time Spent: 3h 20m (was: 3h 10m)
> BigQuery IO not support convert double type for beam row
> --------------------------------------------------------
>
> Key: BEAM-9613
> URL: https://issues.apache.org/jira/browse/BEAM-9613
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.28.0
> Reporter: MAKSIM TSYGAN
> Assignee: Matthew Ouyang
> Priority: P3
> Fix For: 2.30.0
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> If execute query with double column via BigQueryIO.readFrom(), I get
> exception:
> Caused by: java.lang.UnsupportedOperationException: Converting BigQuery type
> 'class java.lang.Double' to 'FieldType\{typeName=DOUBLE, nullable=true,
> logicalType=null, collectionElementType=null, mapKeyType=null,
> mapValueType=null, rowSchema=null, metadata={}}' is not supported
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamValue(BigQueryUtils.java:532)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRowFieldValue(BigQueryUtils.java:483)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.lambda$toBeamRow$6(BigQueryUtils.java:469)
> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRow(BigQueryUtils.java:470)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
