[
https://issues.apache.org/jira/browse/BEAM-12479?focusedWorklogId=637882&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-637882
]
ASF GitHub Bot logged work on BEAM-12479:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Aug/21 20:28
Start Date: 13/Aug/21 20:28
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #14997:
URL: https://github.com/apache/beam/pull/14997#issuecomment-898703812
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 637882)
Time Spent: 50m (was: 40m)
> UnsupportedOperationException when reading from BigQuery tables and
> converting TableRows to Beam Rows
> -----------------------------------------------------------------------------------------------------
>
> Key: BEAM-12479
> URL: https://issues.apache.org/jira/browse/BEAM-12479
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.29.0
> Environment: Beam Java SDK
> Reporter: Pascal GILLET
> Priority: P2
> Labels: pull-request-available
> Fix For: 2.33.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {{UnsupportedOperationExceptions}} are thrown in
> {{org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils#toBeamValue(FieldType,
> Object)}}
> when reading from BigQuery tables with
> {{org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO#readTableRowsWithSchema()}}
> and converting the returned {{TableRows}} to Beam {{Rows}}
> Example:
> {code:java}
> PCollection<Row> rows =
> pipeline
> .apply(
> "Read from BigQuery table",
> BigQueryIO.readTableRowsWithSchema().from(String.format("%s:%s.%s", project,
> dataset, table)))
> .apply(Convert.toRows());{code}
>
> {{UnsupportedOperationException}} messages that I have encountered are of the
> type:
> {{Converting BigQuery type "java.lang.Boolean" to "BOOLEAN" is not supported}}
> {{Converting BigQuery type "java.lang.Double" to "DOUBLE" is not supported}}
> ...While the conversion of these Java types should be straightforward.
> Indeed, the method {{BigQueryUtils#toBeamValue(FieldType, Object)}} expects
> only {{String}} objects or {{Collections}} of {{Strings}}.
> I had to upgrade {{com.google.cloud:google-cloud-bigquery}} from 1.108.0 to
> 1.132.0 in my project. So my guess is this latest version is now able to map
> BiqQuery (SQL) types to Java types instead of raw Strings, in particular BOOL
> to Boolean, INT64 to Long and FLOAT64 to Double.
> In my understanding though, from BigQuery to Beam, there would be no need to
> manage Java Byte, Short, Integer and Float since BigQuery types are "limited"
> to standard SQL types INT64, and FLOAT64 that encompass them all (on the
> other hand, BQ NUMERIC type is mapped to Java BigDecimal).
> I propose a pull request to also support {{Number}} and {{Boolean}} objects
> in method {{BigQueryUtils#toBeamValue(FieldType, Object)}}. It is only added
> behavior and the updated method is still compatible with the current
> functioning.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
