Daniel Oliveira created BEAM-7643: ------------------------------------- Summary: Nearly all PostCommits failing due to Google Cloud issues Key: BEAM-7643 URL: https://issues.apache.org/jira/browse/BEAM-7643 Project: Beam Issue Type: Bug Components: test-failures Reporter: Daniel Oliveira Assignee: Mark Liu
Multiple Beam PostCommits seem to be failing due to a variety of errors, such as "503 Service unavailable", "429 Too Many Requests", or "404 Not Found". It's hard to add a complete description of the issue here since there are too many tests failing. Also made worse because some other flakes and issues seem to be masking these failures. But I'll try to find a few examples: [https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3661/] {noformat} 11:17:01 BadStatusCodeError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>: response: <{'status': '429', 'content-length': '598', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Wed, 26 Jun 2019 18:17:01 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json; charset=UTF-8'}>, content <{ 11:17:01 "error": { 11:17:01 "code": 429, 11:17:01 "message": "Quota exceeded for quota metric 'dataflow.googleapis.com/create_requests' and limit 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for consumer 'project_number:844138762903'.", 11:17:01 "status": "RESOURCE_EXHAUSTED", 11:17:01 "details": [ 11:17:01 { 11:17:01 "@type": "type.googleapis.com/google.rpc.Help", 11:17:01 "links": [ 11:17:01 { 11:17:01 "description": "Google developer console API key", 11:17:01 "url": "https://console.developers.google.com/project/844138762903/apiui/credential" 11:17:01 } 11:17:01 ] 11:17:01 } 11:17:01 ] 11:17:01 } 11:17:01 }{noformat} [https://builds.apache.org/job/beam_PostCommit_Python_Verify/8598/] {noformat} BadStatusCodeError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>: response: <{'status': '429', 'content-length': '598', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'se rver': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Wed, 26 Jun 2019 18:10:42 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json; charset=UTF-8'}>, content <{ "error": { "code": 429, "message": "Quota exceeded for quota metric 'dataflow.googleapis.com/create_requests' and limit 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for consumer 'project_number:844138762903'.", "status": "RESOURCE_EXHAUSTED", "details": [ { "@type": "type.googleapis.com/google.rpc.Help", "links": [ { "description": "Google developer console API key", "url": "https://console.developers.google.com/project/844138762903/apiui/credential" } {noformat} [https://builds.apache.org/job/beam_PostCommit_Java/3617/] {noformat} java.lang.RuntimeException : Failed to create a workflow job: The service is currently unavailable. Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 503 Service Unavailable { "code" : 503, "errors" : [ { "domain" : "global", "message" : "The service is currently unavailable.", "reason" : "backendError" } ], "message" : "The service is currently unavailable.", "status" : "UNAVAILABLE" } Dataflow SDK version: 2.15.0-SNAPSHOT{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)