[
https://issues.apache.org/jira/browse/BEAM-9674?focusedWorklogId=423043&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-423043
]
ASF GitHub Bot logged work on BEAM-9674:
----------------------------------------
Author: ASF GitHub Bot
Created on: 15/Apr/20 20:38
Start Date: 15/Apr/20 20:38
Worklog Time Spent: 10m
Work Description: kmjung commented on pull request #11292: [BEAM-9674]
Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#discussion_r409117833
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageSourceBase.java
##########
@@ -149,11 +151,14 @@
return ImmutableList.of();
}
+ Schema sessionSchema = new
Schema.Parser().parse(readSession.getAvroSchema().getSchema());
Review comment:
With this change, we're no longer specifying the list of selected fields to
the tables.get call from which the BigQuery schema is taken; as a result, we
get the entire table schema back, so we have to trim it on the client side in
the case where the client has specified selected fields. The Avro schema is
returned as part of the read session and contains only the selected fields, so
we use it as the basis for trimming the BigQuery schema.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 423043)
Time Spent: 4h 20m (was: 4h 10m)
> "Selected fields list too long" error when calling tables.get in
> BigQueryStorageTableSource
> -------------------------------------------------------------------------------------------
>
> Key: BEAM-9674
> URL: https://issues.apache.org/jira/browse/BEAM-9674
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.19.0
> Reporter: Kenneth Jung
> Assignee: Kenneth Jung
> Priority: Minor
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Customers experience errors similar to the following:
> Caused by:
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad
> Request { "code" : 400, "errors" : [
> { "domain" : "global", "message" : "Selected fields too long: <fields> must
> be less than 16384 characters.", "reason" : "invalid" }
> ], "message" : "Selected fields too long: <fields> must be less than 16384
> characters.", "status" : "INVALID_ARGUMENT" }
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
>
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
>
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
>
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
> com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1097)
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
>
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
>
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
>
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.executeWithRetries(BigQueryServicesImpl.java:938)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
