[
https://issues.apache.org/jira/browse/BEAM-14273?focusedWorklogId=763873&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-763873
]
ASF GitHub Bot logged work on BEAM-14273:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 28/Apr/22 20:33
Start Date: 28/Apr/22 20:33
Worklog Time Spent: 10m
Work Description: ahmedabu98 commented on code in PR #17431:
URL: https://github.com/apache/beam/pull/17431#discussion_r861288700
##########
sdks/python/apache_beam/io/gcp/bigquery.py:
##########
@@ -2190,6 +2190,20 @@ def expand(self, pcoll):
'A schema must be provided when writing to BigQuery using '
'Avro based file loads')
+ if self.schema and type(self.schema) is dict:
+
+ def find_in_nested_dict(schema):
+ for field in schema['fields']:
+ if field['type'] == 'JSON':
+ raise ValueError(
+ 'Found JSON type in table schema. JSON data '
+ 'insertion is currently not supported with '
+ 'FILE_LOADS write method.')
+ elif field['type'] == 'STRUCT':
+ find_in_nested_dict(field)
Review Comment:
that is correct. I can add the [bq json type
doc#batchloads](https://cloud.google.com/bigquery/docs/reference/standard-sql/json-data#use_a_batch_load_job)
to the message. Should i also mention `"because Beam doesn't support CSV
format"`?
Issue Time Tracking
-------------------
Worklog Id: (was: 763873)
Time Spent: 4h 10m (was: 4h)
> Update BQ connector to support new JSON type (Python)
> -----------------------------------------------------
>
> Key: BEAM-14273
> URL: https://issues.apache.org/jira/browse/BEAM-14273
> Project: Beam
> Issue Type: New Feature
> Components: io-py-gcp
> Reporter: Ahmed Abualsaud
> Assignee: Ahmed Abualsaud
> Priority: P2
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> BQ has a new JSON type that is defined here:
> [https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#json_type]
> We should update Beam BQ Java and Python connectors to support that for
> various read methods (export jobs, storage API) and write methods (load jobs,
> streaming inserts, storage API).
> We should also add integration tests that exercise reading from /writing to
> BQ tables with columns that has JSON type.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
