[
https://issues.apache.org/jira/browse/BEAM-13945?focusedWorklogId=765557&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-765557
]
ASF GitHub Bot logged work on BEAM-13945:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 03/May/22 17:21
Start Date: 03/May/22 17:21
Worklog Time Spent: 10m
Work Description: johnjcasey commented on code in PR #17492:
URL: https://github.com/apache/beam/pull/17492#discussion_r864009059
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -3002,6 +3004,25 @@ private <DestinationT> WriteResult continueExpandTyped(
}
}
+ private void validateNoJsonTypeInSchema(JSONObject schema) {
+ JSONArray fields = schema.getJSONArray("fields");
+
+ for (int i = 0; i < fields.length(); i++) {
+ JSONObject field = fields.getJSONObject(i);
+ checkArgument(
+ !field.getString("type").equals("JSON"),
+ "Found JSON type in TableSchema. JSON data insertion is currently "
+ + "not supported with 'FILE_LOADS' write method. This is
supported with the "
+ + "other write methods, however. For more information, visit: "
+ +
"https://cloud.google.com/bigquery/docs/reference/standard-sql/";
+ + "json-data#ingest_json_data");
+
+ if (field.getString("type").equals("STRUCT")) {
+ validateNoJsonTypeInSchema(field);
Review Comment:
Probably not, if the bq limit is 15. LGTM
Issue Time Tracking
-------------------
Worklog Id: (was: 765557)
Time Spent: 10h 20m (was: 10h 10m)
> Update BQ connector to support new JSON type
> --------------------------------------------
>
> Key: BEAM-13945
> URL: https://issues.apache.org/jira/browse/BEAM-13945
> Project: Beam
> Issue Type: New Feature
> Components: io-java-gcp
> Reporter: Chamikara Madhusanka Jayalath
> Assignee: Ahmed Abualsaud
> Priority: P2
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> BQ has a new JSON type that is defined here:
> https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#json_type
> We should update Beam BQ Java and Python connectors to support that for
> various read methods (export jobs, storage API) and write methods (load jobs,
> streaming inserts, storage API).
> We should also add integration tests that exercise reading from /writing to
> BQ tables with columns that has JSON type.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
