Daniel Oliveira created BEAM-7643:
-------------------------------------
Summary: Nearly all PostCommits failing due to Google Cloud issues
Key: BEAM-7643
URL: https://issues.apache.org/jira/browse/BEAM-7643
Project: Beam
Issue Type: Bug
Components: test-failures
Reporter: Daniel Oliveira
Assignee: Mark Liu
Multiple Beam PostCommits seem to be failing due to a variety of errors, such
as "503 Service unavailable", "429 Too Many Requests", or "404 Not Found".
It's hard to add a complete description of the issue here since there are too
many tests failing. Also made worse because some other flakes and issues seem
to be masking these failures. But I'll try to find a few examples:
[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3661/]
{noformat}
11:17:01 BadStatusCodeError: HttpError accessing
<https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>:
response: <{'status': '429', 'content-length': '598', 'x-xss-protection': '0',
'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary':
'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip',
'cache-control': 'private', 'date': 'Wed, 26 Jun 2019 18:17:01 GMT',
'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json;
charset=UTF-8'}>, content <{
11:17:01 "error": {
11:17:01 "code": 429,
11:17:01 "message": "Quota exceeded for quota metric
'dataflow.googleapis.com/create_requests' and limit
'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for
consumer 'project_number:844138762903'.",
11:17:01 "status": "RESOURCE_EXHAUSTED",
11:17:01 "details": [
11:17:01 {
11:17:01 "@type": "type.googleapis.com/google.rpc.Help",
11:17:01 "links": [
11:17:01 {
11:17:01 "description": "Google developer console API key",
11:17:01 "url":
"https://console.developers.google.com/project/844138762903/apiui/credential";
11:17:01 }
11:17:01 ]
11:17:01 }
11:17:01 ]
11:17:01 }
11:17:01 }{noformat}
[https://builds.apache.org/job/beam_PostCommit_Python_Verify/8598/]
{noformat}
BadStatusCodeError: HttpError accessing
<https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>:
response: <{'status': '429', 'content-length': '598', 'x-xss-protection': '0',
'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary':
'Origin, X-Origin, Referer', 'se
rver': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date':
'Wed, 26 Jun 2019 18:10:42 GMT', 'x-frame-options': 'SAMEORIGIN',
'content-type': 'application/json; charset=UTF-8'}>, content <{
"error": {
"code": 429,
"message": "Quota exceeded for quota metric
'dataflow.googleapis.com/create_requests' and limit
'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for
consumer 'project_number:844138762903'.",
"status": "RESOURCE_EXHAUSTED",
"details": [
{
"@type": "type.googleapis.com/google.rpc.Help",
"links": [
{
"description": "Google developer console API key",
"url":
"https://console.developers.google.com/project/844138762903/apiui/credential";
}
{noformat}
[https://builds.apache.org/job/beam_PostCommit_Java/3617/]
{noformat}
java.lang.RuntimeException : Failed to create a workflow job: The service is
currently unavailable.
Caused by:
com.google.api.client.googleapis.json.GoogleJsonResponseException:
503 Service Unavailable
{
"code" : 503,
"errors" : [ {
"domain" : "global",
"message" : "The service is currently unavailable.",
"reason" : "backendError"
} ],
"message" : "The service is currently unavailable.",
"status" : "UNAVAILABLE"
}
Dataflow SDK version: 2.15.0-SNAPSHOT{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
