[
https://issues.apache.org/jira/browse/BEAM-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841822#comment-16841822
]
Robert Burke commented on BEAM-7326:
------------------------------------
The BigQuery Go package (not Beam's IO) doesn't mention base64 at all. I
believe that it handles that by itself usually, and treats them as opaque
blobs. In particular, it's handled by the JSON encoding of the values, which
automatically base64 encodes bytes.
See [https://godoc.org/cloud.google.com/go/bigquery] and
[https://godoc.org/encoding/json#Marshal]
In other words, in Go, its a BiqQuery implementation detail that is hidden from
users, unless they configure things to change it.
> Document that Beam BigQuery IO expects users to pass base64-encoded bytes,
> and BQ IO serves base64-encoded bytes to the user.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-7326
> URL: https://issues.apache.org/jira/browse/BEAM-7326
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, io-python-gcp
> Reporter: Valentyn Tymofieiev
> Priority: Major
>
> BYTES is one of the Datatypes supported by Google Cloud BigQuery, and Apache
> Beam BigQuery IO connector.
> Current implementation of BigQuery connector in Java and Python SDKs expects
> that users base64-encode bytes before passing them to BigQuery IO, see
> discussion on dev: [1]
> This needs to be reflected in public documentation, see [2-4]
> cc: [~juta] [~chamikara] [~pabloem]
> cc: [~lostluck] [~kedin] FYI and to advise whether similar action needs to be
> done for Go SDK and/or Beam SQL.
> [1]
> https://lists.apache.org/thread.html/f35c836887014e059527ed1a806e730321e2f9726164a3030575f455@%3Cdev.beam.apache.org%3E
> [2] https://beam.apache.org/documentation/io/built-in/google-bigquery/
> [3]
> https://beam.apache.org/releases/pydoc/2.12.0/apache_beam.io.gcp.bigquery.html
> [4]
> https://beam.apache.org/releases/javadoc/2.12.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
