[
https://issues.apache.org/jira/browse/BEAM-13959?focusedWorklogId=729354&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729354
]
ASF GitHub Bot logged work on BEAM-13959:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Feb/22 00:43
Start Date: 18/Feb/22 00:43
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on pull request #16886:
URL: https://github.com/apache/beam/pull/16886#issuecomment-1043680718
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 729354)
Time Spent: 2h (was: 1h 50m)
> Unable to write to BigQuery tables with column named 'f'
> --------------------------------------------------------
>
> Key: BEAM-13959
> URL: https://issues.apache.org/jira/browse/BEAM-13959
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.36.0
> Reporter: Joel Weierman
> Assignee: Reuven Lax
> Priority: P1
> Time Spent: 2h
> Remaining Estimate: 0h
>
> When using the BigQuery Storage Write API through the Java Beam SDK (both the
> latest release 2.35.0 and 2.36.0-SNAPSHOT), there seems to be an issue when
> converting field Storage API Proto to columns named 'f'.
> Reproduction Steps: The "field" named 'f' is unable to be written to BigQuery
> with the error referenced below.
> [1]
> "name": "item3",
> "type": "RECORD",
> "mode": "NULLABLE",
> "fields": [
> {
> "name": "data",
> "mode": "NULLABLE",
> "type": "RECORD",
> "fields": [
> {
> "mode": "NULLABLE",
> "name": "a",
> "type": "FLOAT"
> },
> {
> "mode": "NULLABLE",
> "name": "b",
> "type": "FLOAT"
> },
> {
> "mode": "NULLABLE",
> "name": "c",
> "type": "FLOAT"
> },
> {
> "mode": "NULLABLE",
> "name": "d",
> "type": "FLOAT"
> },
> {
> "mode": "NULLABLE",
> "name": "e",
> "type": "FLOAT"
> },
> {
> "mode": "NULLABLE",
> "name": "f",
> "type": "FLOAT"
> }
> ]
> }
> ]
> [2]
> {
> ...
> "item3": {
> "data": {
> "a": 1.627424812511E12,
> "b": 3.0,
> "c": 3.0,
> "d": 530.0,
> "e": 675.0
> }
> },
> ...
> }
> The following error occurs: Exception in thread "main"
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.IllegalArgumentException: Can not set java.util.List field
> com.google.api.services.bigquery.model.TableRow.f to java.lang.Double at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:373)
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:341)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:218) at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67) at
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:323) at
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:309) at
> com.google.cloud.teleport.templates.PubSubToBigQuery.run(PubSubToBigQuery.java:342)
> at
> com.google.cloud.teleport.templates.PubSubToBigQuery.main(PubSubToBigQuery.java:223)
> Caused by: java.lang.IllegalArgumentException: Can not set java.util.List
> field com.google.api.services.bigquery.model.TableRow.f to java.lang.Double
> at
> sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
> at
> sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
> at
> sun.reflect.UnsafeObjectFieldAccessorImpl.set(UnsafeObjectFieldAccessorImpl.java:81)
> at java.lang.reflect.Field.set(Field.java:764) at
> com.google.api.client.util.FieldInfo.setFieldValue(FieldInfo.java:275) at
> com.google.api.client.util.FieldInfo.setValue(FieldInfo.java:231) at
> com.google.api.client.util.GenericData.set(GenericData.java:118) at
> com.google.api.client.json.GenericJson.set(GenericJson.java:91) at
> com.google.api.services.bigquery.model.TableRow.set(TableRow.java:64) at
> com.google.api.services.bigquery.model.TableRow.set(TableRow.java:29) at
> com.google.api.client.util.GenericData.putAll(GenericData.java:131) at
> org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto.toProtoValue(TableRowToStorageApiProto.java:206)
> at
> org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto.messageValueFromFieldValue(TableRowToStorageApiProto.java:175)
> at
> org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto.messageFromTableRow(TableRowToStorageApiProto.java:103)
> at
> org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto.toProtoValue(TableRowToStorageApiProto.java:207)
> at
> org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto.messageValueFromFieldValue(TableRowToStorageApiProto.java:175)
> at
> org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto.messageFromTableRow(TableRowToStorageApiProto.java:103)
> at
> org.apache.beam.sdk.io.gcp.bigquery.StorageApiDynamicDestinationsTableRow$1.toMessage(StorageApiDynamicDestinationsTableRow.java:95)
> at
> org.apache.beam.sdk.io.gcp.bigquery.StorageApiConvertMessages$ConvertMessagesDoFn.processElement(StorageApiConvertMessages.java:106)
> This error does not show up if I leave the write method to use Streaming
> Inserts.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
