[
https://issues.apache.org/jira/browse/BEAM-7643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873802#comment-16873802
]
Mark Liu commented on BEAM-7643:
--------------------------------
Google Dataflow service is down currently.
> Nearly all PostCommits failing due to Google Cloud issues
> ---------------------------------------------------------
>
> Key: BEAM-7643
> URL: https://issues.apache.org/jira/browse/BEAM-7643
> Project: Beam
> Issue Type: Bug
> Components: test-failures
> Reporter: Daniel Oliveira
> Assignee: Mark Liu
> Priority: Major
> Labels: currently-failing
>
> Multiple Beam PostCommits seem to be failing due to a variety of errors, such
> as "503 Service unavailable", "429 Too Many Requests", or "404 Not Found".
> It's hard to add a complete description of the issue here since there are too
> many tests failing. Also made worse because some other flakes and issues seem
> to be masking these failures. But I'll try to find a few examples:
> [https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3661/]
>
> {noformat}
> 11:17:01 BadStatusCodeError: HttpError accessing
> <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>:
> response: <{'status': '429', 'content-length': '598', 'x-xss-protection':
> '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked',
> 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding':
> 'gzip', 'cache-control': 'private', 'date': 'Wed, 26 Jun 2019 18:17:01 GMT',
> 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json;
> charset=UTF-8'}>, content <{
> 11:17:01 "error": {
> 11:17:01 "code": 429,
> 11:17:01 "message": "Quota exceeded for quota metric
> 'dataflow.googleapis.com/create_requests' and limit
> 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for
> consumer 'project_number:844138762903'.",
> 11:17:01 "status": "RESOURCE_EXHAUSTED",
> 11:17:01 "details": [
> 11:17:01 {
> 11:17:01 "@type": "type.googleapis.com/google.rpc.Help",
> 11:17:01 "links": [
> 11:17:01 {
> 11:17:01 "description": "Google developer console API key",
> 11:17:01 "url":
> "https://console.developers.google.com/project/844138762903/apiui/credential";
> 11:17:01 }
> 11:17:01 ]
> 11:17:01 }
> 11:17:01 ]
> 11:17:01 }
> 11:17:01 }{noformat}
>
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/8598/]
>
> {noformat}
> BadStatusCodeError: HttpError accessing
> <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>:
> response: <{'status': '429', 'content-length': '598', 'x-xss-protection':
> '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked',
> 'vary': 'Origin, X-Origin, Referer', 'se
> rver': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private',
> 'date': 'Wed, 26 Jun 2019 18:10:42 GMT', 'x-frame-options': 'SAMEORIGIN',
> 'content-type': 'application/json; charset=UTF-8'}>, content <{
> "error": {
> "code": 429,
> "message": "Quota exceeded for quota metric
> 'dataflow.googleapis.com/create_requests' and limit
> 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for
> consumer 'project_number:844138762903'.",
> "status": "RESOURCE_EXHAUSTED",
> "details": [
> {
> "@type": "type.googleapis.com/google.rpc.Help",
> "links": [
> {
> "description": "Google developer console API key",
> "url":
> "https://console.developers.google.com/project/844138762903/apiui/credential";
> }
> {noformat}
>
> [https://builds.apache.org/job/beam_PostCommit_Java/3617/]
> {noformat}
> java.lang.RuntimeException : Failed to create a workflow job: The service is
> currently unavailable.
> Caused by:
> com.google.api.client.googleapis.json.GoogleJsonResponseException:
> 503 Service Unavailable
> {
> "code" : 503,
> "errors" : [ {
> "domain" : "global",
> "message" : "The service is currently unavailable.",
> "reason" : "backendError"
> } ],
> "message" : "The service is currently unavailable.",
> "status" : "UNAVAILABLE"
> }
> Dataflow SDK version: 2.15.0-SNAPSHOT{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
