kennknowles opened a new issue, #19120:
URL: https://github.com/apache/beam/issues/19120
When many machines, or too big machines are used, BigQuery returns rateLimit
exceeded errors in response to the GET requests from
BigQueryWriteFn.start_bundle.
Unsure if this causes performance issues, or is just noise. (Produced with a
pipeline reading strings from Pub/Sub that are ~1000 characters long, and
writing directly to BigQuery)
{{... File
"/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py", line
1087, in get_or_create_table found_table = self._get_table(project_id,
dataset_id, table_id) File
"/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line 184,
in wrapper return fun(*args, **kwargs) File
"/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py", line
925, in _get_table response = self.client.tables.Get(request) File
"/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
line 611, in Get config, request, global_params=global_params) File
"/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
722, in _RunMethod return self.ProcessHttpResponse(method_config,
http_response, request) File
"/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
728, in ProcessHttpResponse self.__ProcessHttpResponse(method_config,
http_response, request)) File "/usr/local/lib/
python2.7/dist-packages/apitools/base/py/base_api.py", line 599, in
__ProcessHttpResponse http_response, method_config=method_config,
request=request) HttpForbiddenError: HttpError accessing
<[https://www.googleapis.com/bigquery/v2/projects/<xxxxxx>/datasets/](https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json)
[<xxxxxx>](https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json)
[/tables/](https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json)
[<xxxxxx>](https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json)
[?alt=json](https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json)\>:
response: <
{'status': '403', 'content-length': '577', 'x-xss-protection': '1;
mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding':
'chunked', 'expires': 'Sun, 25 Nov 2018 14:36:24 GMT', 'vary': 'Origin,
X-Origin', 'server': 'GSE', '-content-encoding': 'gzip', 'cache-control':
'private, max-age=0', 'date': 'Sun, 25 Nov 2018 14:36:24 GMT',
'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json;
charset=UTF-8'}
\>, content <{ "error": { "errors": [
{ "domain": "global", "reason": "rateLimitExceeded", "message": "Exceeded
rate limits: Your user_method exceeded quota for api requests per user per
method. For more information, see
https://cloud.google.com/bigquery/troubleshooting-errors";, "locationType":
"other", "location": "helix_api.method_request" }
], "code": 403, "message": "Exceeded rate limits: Your user_method exceeded
quota for api requests per user per method. For more information, see
[https://cloud.google.com/bigquery/troubleshooting-errors](https://cloud.google.com/bigquery/troubleshooting-errors)"}}
Imported from Jira
[BEAM-6169](https://issues.apache.org/jira/browse/BEAM-6169). Original Jira may
contain additional context.
Reported by: tpilewicz.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
