[ https://issues.apache.org/jira/browse/BEAM-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930869#comment-16930869 ]
Valentyn Tymofieiev commented on BEAM-8215: ------------------------------------------- Seems like a recent change in GCP? Can be reproducible with Direct runner, using a released SDK: python -m apache_beam.examples.wordcount --input=gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.0000999 --output=~/tmp/out {noformat} Traceback (most recent call last): File "/usr/local/google/home/valentyn/tmp/py3env/lib/python3.6/site-packages/apache_beam/utils/retry.py", line 206, in wrapper return fun(*args, **kwargs) File "/usr/local/google/home/valentyn/tmp/py3env/lib/python3.6/site-packages/apache_beam/io/gcp/gcsio.py", line 374, in exists self.client.objects.Get(request) # metadata File "/usr/local/google/home/valentyn/tmp/py3env/lib/python3.6/site-packages/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py", line 1098, in Get download=download) File "/usr/local/google/home/valentyn/tmp/py3env/lib/python3.6/site-packages/apitools/base/py/base_api.py", line 729, in _RunMethod http, http_request, **opts) File "/usr/local/google/home/valentyn/tmp/py3env/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py", line 356, in MakeRequest max_retry_wait, total_wait_sec)) File "/usr/local/google/home/valentyn/tmp/py3env/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py", line 304, in HandleExceptionsAndRebuildHttpConnections raise retry_args.exc File "/usr/local/google/home/valentyn/tmp/py3env/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py", line 346, in MakeRequest check_response_func=check_response_func) File "/usr/local/google/home/valentyn/tmp/py3env/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py", line 396, in _MakeRequestNoRetry redirections=redirections, connection_type=connection_type) File "/usr/local/google/home/valentyn/tmp/py3env/lib/python3.6/site-packages/oauth2client/transport.py", line 153, in new_request credentials._refresh(orig_request_method) File "/usr/local/google/home/valentyn/tmp/py3env/lib/python3.6/site-packages/oauth2client/client.py", line 765, in _refresh self._do_refresh_request(http_request) File "/usr/local/google/home/valentyn/tmp/py3env/lib/python3.6/site-packages/oauth2client/client.py", line 834, in _do_refresh_request raise HttpAccessTokenRefreshError(error_msg, status=resp.status) oauth2client.client.HttpAccessTokenRefreshError: invalid_grant: reauth related error (invalid_rapt) {noformat} > Wordcount 1GB Python PKB benchmarks sometimes fail with uninformative error, > ----------------------------------------------------------------------------- > > Key: BEAM-8215 > URL: https://issues.apache.org/jira/browse/BEAM-8215 > Project: Beam > Issue Type: Bug > Components: testing > Reporter: Valentyn Tymofieiev > Assignee: Chamikara Jayalath > Priority: Major > > Example: > https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/452/console > {noformat} > 12:09:27 2019-09-11 19:09:27,655 a47400ce MainThread > beam_integration_benchmark(1/1) ERROR Error during benchmark > beam_integration_benchmark > 12:09:27 Traceback (most recent call last): > 12:09:27 File > "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", > line 841, in RunBenchmark > 12:09:27 DoRunPhase(spec, collector, detailed_timer) > 12:09:27 File > "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", > line 687, in DoRunPhase > 12:09:27 samples = spec.BenchmarkRun(spec) > 12:09:27 File > "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py", > line 160, in Run > 12:09:27 job_type=job_type) > 12:09:27 File > "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py", > line 91, in SubmitJob > 12:09:27 assert retcode == 0, "Integration Test Failed." > 12:09:27 AssertionError: Integration Test Failed. > {noformat} > It seems like Job submission failed, but there are no details. I talked with > [~markflyhigh], and sounds like we plan to stop using PKB in favor of another > framework. > Assigning to Mark for now to triage follow up or reassign as appropriate. -- This message was sent by Atlassian Jira (v8.3.2#803003)