[
https://issues.apache.org/jira/browse/BEAM-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Pilewicz updated BEAM-6169:
----------------------------------
Description:
When many machines, or too big machines are used, BigQuery returns rateLimit
exceeded errors in response to the GET requests from
BigQueryWriteFn.start_bundle.
Unsure if this causes performance issues, or is just noise. (Produced with a
pipeline reading strings from Pub/Sub that are ~1000 characters long, and
writing directly to BigQuery)
{{... File
"/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py", line
1087, in get_or_create_table found_table = self._get_table(project_id,
dataset_id, table_id) File
"/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line 184,
in wrapper return fun(*args, **kwargs) File
"/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py", line
925, in _get_table response = self.client.tables.Get(request) File
"/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
line 611, in Get config, request, global_params=global_params) File
"/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
722, in _RunMethod return self.ProcessHttpResponse(method_config,
http_response, request) File
"/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
728, in ProcessHttpResponse self.__ProcessHttpResponse(method_config,
http_response, request)) File
"/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
599, in __ProcessHttpResponse http_response, method_config=method_config,
request=request) HttpForbiddenError: HttpError accessing
<[https://www.googleapis.com/bigquery/v2/projects/<xxxxxx>/datasets/|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
[<xxxxxx>|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
[/tables/|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
[<xxxxxx>|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
[?alt=json|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]>:
response: <
{'status': '403', 'content-length': '577', 'x-xss-protection': '1; mode=block',
'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'expires':
'Sun, 25 Nov 2018 14:36:24 GMT', 'vary': 'Origin, X-Origin', 'server': 'GSE',
'-content-encoding': 'gzip', 'cache-control': 'private, max-age=0', 'date':
'Sun, 25 Nov 2018 14:36:24 GMT', 'x-frame-options': 'SAMEORIGIN',
'content-type': 'application/json; charset=UTF-8'}
>, content <{ "error": { "errors": [
{ "domain": "global", "reason": "rateLimitExceeded", "message": "Exceeded rate
limits: Your user_method exceeded quota for api requests per user per method.
For more information, see
https://cloud.google.com/bigquery/troubleshooting-errors";, "locationType":
"other", "location": "helix_api.method_request" }
], "code": 403, "message": "Exceeded rate limits: Your user_method exceeded
quota for api requests per user per method. For more information, see
[https://cloud.google.com/bigquery/troubleshooting-errors]"}}
was:
When many machines, or too big machines are used, BigQuery returns rateLimit
exceeded errors in response to the GET requests from
BigQueryWriteFn.start_bundle.
Unsure if this causes performance issues, or is just noise. (Produced with a
pipeline reading strings from Pub/Sub that are ~1000 characters long, and
writing directly to BigQuery)
{{... File
"/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py", line
1087, in get_or_create_table found_table = self._get_table(project_id,
dataset_id, table_id) File
"/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line 184,
in wrapper return fun(*args, **kwargs) File
"/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py", line
925, in _get_table response = self.client.tables.Get(request) File
"/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
line 611, in Get config, request, global_params=global_params) File
"/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
722, in _RunMethod return self.ProcessHttpResponse(method_config,
http_response, request) File
"/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
728, in ProcessHttpResponse self.__ProcessHttpResponse(method_config,
http_response, request)) File
"/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
599, in __ProcessHttpResponse http_response, method_config=method_config,
request=request) HttpForbiddenError: HttpError accessing
<https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json>:
response: <{'status': '403', 'content-length': '577', 'x-xss-protection': '1;
mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding':
'chunked', 'expires': 'Sun, 25 Nov 2018 14:36:24 GMT', 'vary': 'Origin,
X-Origin', 'server': 'GSE', '-content-encoding': 'gzip', 'cache-control':
'private, max-age=0', 'date': 'Sun, 25 Nov 2018 14:36:24 GMT',
'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json;
charset=UTF-8'}>, content <{ "error": { "errors": [ { "domain": "global",
"reason": "rateLimitExceeded", "message": "Exceeded rate limits: Your
user_method exceeded quota for api requests per user per method. For more
information, see https://cloud.google.com/bigquery/troubleshooting-errors";,
"locationType": "other", "location": "helix_api.method_request" } ], "code":
403, "message": "Exceeded rate limits: Your user_method exceeded quota for api
requests per user per method. For more information, see
https://cloud.google.com/bigquery/troubleshooting-errors"}}
> Dataflow runner: rateLimit exceeded from BigQuery API in start_bundle
> ---------------------------------------------------------------------
>
> Key: BEAM-6169
> URL: https://issues.apache.org/jira/browse/BEAM-6169
> Project: Beam
> Issue Type: Bug
> Components: gcp-quota
> Affects Versions: 2.8.0
> Environment: Python SDK 2.8, Dataflow runner
> Reporter: Thomas Pilewicz
> Assignee: Rafael Fernandez
> Priority: Minor
>
> When many machines, or too big machines are used, BigQuery returns rateLimit
> exceeded errors in response to the GET requests from
> BigQueryWriteFn.start_bundle.
> Unsure if this causes performance issues, or is just noise. (Produced with a
> pipeline reading strings from Pub/Sub that are ~1000 characters long, and
> writing directly to BigQuery)
> {{... File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py", line
> 1087, in get_or_create_table found_table = self._get_table(project_id,
> dataset_id, table_id) File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line
> 184, in wrapper return fun(*args, **kwargs) File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py", line
> 925, in _get_table response = self.client.tables.Get(request) File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
> line 611, in Get config, request, global_params=global_params) File
> "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
> 722, in _RunMethod return self.ProcessHttpResponse(method_config,
> http_response, request) File
> "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
> 728, in ProcessHttpResponse self.__ProcessHttpResponse(method_config,
> http_response, request)) File
> "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
> 599, in __ProcessHttpResponse http_response, method_config=method_config,
> request=request) HttpForbiddenError: HttpError accessing
> <[https://www.googleapis.com/bigquery/v2/projects/<xxxxxx>/datasets/|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
>
> [<xxxxxx>|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
>
> [/tables/|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
>
> [<xxxxxx>|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
>
> [?alt=json|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]>:
> response: <
> {'status': '403', 'content-length': '577', 'x-xss-protection': '1;
> mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding':
> 'chunked', 'expires': 'Sun, 25 Nov 2018 14:36:24 GMT', 'vary': 'Origin,
> X-Origin', 'server': 'GSE', '-content-encoding': 'gzip', 'cache-control':
> 'private, max-age=0', 'date': 'Sun, 25 Nov 2018 14:36:24 GMT',
> 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json;
> charset=UTF-8'}
> >, content <{ "error": { "errors": [
> { "domain": "global", "reason": "rateLimitExceeded", "message": "Exceeded
> rate limits: Your user_method exceeded quota for api requests per user per
> method. For more information, see
> https://cloud.google.com/bigquery/troubleshooting-errors";, "locationType":
> "other", "location": "helix_api.method_request" }
> ], "code": 403, "message": "Exceeded rate limits: Your user_method exceeded
> quota for api requests per user per method. For more information, see
> [https://cloud.google.com/bigquery/troubleshooting-errors]"}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
