See <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/8171/display/redirect>
Changes: ------------------------------------------ [...truncated 1.18 MB...] [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124123-125752.1626957683.125908/dataflow_python_sdk.tar in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124123-125752.1626957683.125908/dataflow-worker.jar... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124123-125752.1626957683.125908/dataflow-worker.jar in 6 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124123-125752.1626957683.125908/pipeline.pb... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124123-125752.1626957683.125908/pipeline.pb in 0 seconds. [33mWARNING [0m apache_beam.options.pipeline_options:pipeline_options.py:309 Discarding unparseable args: ['--sleep_secs=20', '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test'] [33mWARNING [0m apache_beam.options.pipeline_options:pipeline_options.py:309 Discarding unparseable args: ['--sleep_secs=20', '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test'] [31m[1m______________ ReshuffleTest.test_reshuffle_preserves_timestamps _______________[0m [gw4] linux -- Python 3.8.5 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/bin/python3.8> self = <apache_beam.transforms.util_test.ReshuffleTest testMethod=test_reshuffle_preserves_timestamps> [1m @pytest.mark.it_validatesrunner[0m [1m def test_reshuffle_preserves_timestamps(self):[0m [1m with TestPipeline() as pipeline:[0m [1m [0m [1m # Create a PCollection and assign each element with a different timestamp.[0m [1m before_reshuffle = ([0m [1m pipeline[0m [1m | beam.Create([[0m [1m {[0m [1m 'name': 'foo', 'timestamp': MIN_TIMESTAMP[0m [1m },[0m [1m {[0m [1m 'name': 'foo', 'timestamp': 0[0m [1m },[0m [1m {[0m [1m 'name': 'bar', 'timestamp': 33[0m [1m },[0m [1m {[0m [1m 'name': 'bar', 'timestamp': 0[0m [1m },[0m [1m ])[0m [1m | beam.Map([0m [1m lambda element: beam.window.TimestampedValue([0m [1m element, element['timestamp'])))[0m [1m [0m [1m # Reshuffle the PCollection above and assign the timestamp of an element[0m [1m # to that element again.[0m [1m after_reshuffle = before_reshuffle | beam.Reshuffle()[0m [1m [0m [1m # Given an element, emits a string which contains the timestamp and the[0m [1m # name field of the element.[0m [1m def format_with_timestamp(element, timestamp=beam.DoFn.TimestampParam):[0m [1m t = str(timestamp)[0m [1m if timestamp == MIN_TIMESTAMP:[0m [1m t = 'MIN_TIMESTAMP'[0m [1m elif timestamp == MAX_TIMESTAMP:[0m [1m t = 'MAX_TIMESTAMP'[0m [1m return '{} - {}'.format(t, element['name'])[0m [1m [0m [1m # Combine each element in before_reshuffle with its timestamp.[0m [1m formatted_before_reshuffle = ([0m [1m before_reshuffle[0m [1m | "Get before_reshuffle timestamp" >> beam.Map(format_with_timestamp))[0m [1m [0m [1m # Combine each element in after_reshuffle with its timestamp.[0m [1m formatted_after_reshuffle = ([0m [1m after_reshuffle[0m [1m | "Get after_reshuffle timestamp" >> beam.Map(format_with_timestamp))[0m [1m [0m [1m expected_data = [[0m [1m 'MIN_TIMESTAMP - foo',[0m [1m 'Timestamp(0) - foo',[0m [1m 'Timestamp(33) - bar',[0m [1m 'Timestamp(0) - bar'[0m [1m ][0m [1m [0m [1m # Can't compare formatted_before_reshuffle and formatted_after_reshuffle[0m [1m # directly, because they are deferred PCollections while equal_to only[0m [1m # takes a concrete argument.[0m [1m assert_that([0m [1m formatted_before_reshuffle,[0m [1m equal_to(expected_data),[0m [1m label="formatted_before_reshuffle")[0m [1m> assert_that([0m [1m formatted_after_reshuffle,[0m [1m equal_to(expected_data),[0m [1m label="formatted_after_reshuffle")[0m [1m[31mapache_beam/transforms/util_test.py[0m:611: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [1m[31mapache_beam/pipeline.py[0m:585: in __exit__ [1m self.result = self.run()[0m [1m[31mapache_beam/testing/test_pipeline.py[0m:112: in run [1m result = super(TestPipeline, self).run([0m [1m[31mapache_beam/pipeline.py[0m:537: in run [1m return Pipeline.from_runner_api([0m [1m[31mapache_beam/pipeline.py[0m:564: in run [1m return self.runner.run_pipeline(self, self._options)[0m [1m[31mapache_beam/runners/dataflow/test_dataflow_runner.py[0m:53: in run_pipeline [1m self.result = super(TestDataflowRunner,[0m [1m[31mapache_beam/runners/dataflow/dataflow_runner.py[0m:588: in run_pipeline [1m self.dataflow_client.create_job(self.job), self)[0m [1m[31mapache_beam/utils/retry.py[0m:253: in wrapper [1m return fun(*args, **kwargs)[0m [1m[31mapache_beam/runners/dataflow/internal/apiclient.py[0m:688: in create_job [1m return self.submit_job_description(job)[0m [1m[31mapache_beam/utils/retry.py[0m:253: in wrapper [1m return fun(*args, **kwargs)[0m [1m[31mapache_beam/runners/dataflow/internal/apiclient.py[0m:789: in submit_job_description [1m response = self._client.projects_locations_jobs.Create(request)[0m [1m[31mapache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py[0m:902: in Create [1m return self._RunMethod([0m [1m[31m../../build/gradleenv/-1734967051/lib/python3.8/site-packages/apitools/base/py/base_api.py[0m:731: in _RunMethod [1m return self.ProcessHttpResponse(method_config, http_response, request)[0m [1m[31m../../build/gradleenv/-1734967051/lib/python3.8/site-packages/apitools/base/py/base_api.py[0m:737: in ProcessHttpResponse [1m self.__ProcessHttpResponse(method_config, http_response, request))[0m _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3.ProjectsLocationsJobsService object at 0x7fd3ba9ee130> method_config = <ApiMethodInfo relative_path: 'v1b3/projects/{projectId}/locations/{location}/jobs' method_id: 'dataflow.projects.lo...DataflowProjectsLocationsJobsCreateRequest' response_type_name: 'Job' request_field: 'job' supports_download: False> http_response = Response(info={'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Thu, 2...', request_url='https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json') request = <DataflowProjectsLocationsJobsCreateRequest job: <Job environment: <Environment clusterManagerApiService: 'compute....iles: [] type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)> location: 'us-central1' projectId: 'apache-beam-testing'> [1m def __ProcessHttpResponse(self, method_config, http_response, request):[0m [1m """Process the given http response."""[0m [1m if http_response.status_code not in (http_client.OK,[0m [1m http_client.CREATED,[0m [1m http_client.NO_CONTENT):[0m [1m> raise exceptions.HttpError.FromResponse([0m [1m http_response, method_config=method_config, request=request)[0m [1m[31mE apitools.base.py.exceptions.HttpBadRequestError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Thu, 22 Jul 2021 12:41:32 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'status': '400', 'content-length': '237', '-content-encoding': 'gzip'}>, content <{[0m [1m[31mE "error": {[0m [1m[31mE "code": 400,[0m [1m[31mE "message": "(8e359bb26683af7a): The workflow could not be created. Causes: (8e359bb26683a751): A Cloud KMS key cannot be used with Streaming Engine enabled.",[0m [1m[31mE "status": "FAILED_PRECONDITION"[0m [1m[31mE }[0m [1m[31mE }[0m [1m[31mE >[0m [1m[31m../../build/gradleenv/-1734967051/lib/python3.8/site-packages/apitools/base/py/base_api.py[0m:603: HttpBadRequestError ------------------------------ Captured log call ------------------------------- [32mINFO [0m apache_beam.runners.portability.stager:stager.py:644 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/bin/python3.8',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', 'postcommit_requirements.txt', '--exists-action', 'i', '--no-binary', ':all:'] [32mINFO [0m apache_beam.runners.portability.stager:stager.py:300 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz";> to staging location. [33mWARNING [0m root:environments.py:371 Make sure that locally built Python SDK docker image has Python 3.8 interpreter. [32mINFO [0m root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.8_sdk:2.33.0.dev [32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python38-fnapi:beam-master-20210720 [32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python38-fnapi:beam-master-20210720" for Docker environment [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/requirements.txt... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/requirements.txt in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/pbr-5.5.1.tar.gz... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/pbr-5.5.1.tar.gz in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/pbr-5.6.0.tar.gz... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/pbr-5.6.0.tar.gz in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/mock-2.0.0.tar.gz... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/mock-2.0.0.tar.gz in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/six-1.16.0.tar.gz... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/six-1.16.0.tar.gz in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/soupsieve-2.1.tar.gz... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/soupsieve-2.1.tar.gz in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/soupsieve-2.2.1.tar.gz... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/soupsieve-2.2.1.tar.gz in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/PyHamcrest-1.10.1.tar.gz... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/PyHamcrest-1.10.1.tar.gz in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/parameterized-0.7.5.tar.gz... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/parameterized-0.7.5.tar.gz in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/beautifulsoup4-4.9.3.tar.gz... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/beautifulsoup4-4.9.3.tar.gz in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/dataflow_python_sdk.tar... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/dataflow_python_sdk.tar in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/dataflow-worker.jar... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/dataflow-worker.jar in 5 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/pipeline.pb... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0722124124-559729.1626957684.559885/pipeline.pb in 0 seconds. [33mWARNING [0m apache_beam.options.pipeline_options:pipeline_options.py:309 Discarding unparseable args: ['--sleep_secs=20', '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test'] [33mWARNING [0m apache_beam.options.pipeline_options:pipeline_options.py:309 Discarding unparseable args: ['--sleep_secs=20', '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test'] [33m=============================== warnings summary ===============================[0m <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead def call(self, fn, *args, **kwargs): -- Docs: https://docs.pytest.org/en/latest/warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/pytest_validatesRunnerStreamingTests-df-py38-xdist.xml> - [31m[1m========= 26 failed, 3 passed, 3 skipped, 8 warnings in 80.40 seconds ==========[0m FAILURE: Build completed with 3 failures. 1: Task failed with an exception. ----------- * Where: Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 189 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py37:validatesRunnerStreamingTests'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * Where: Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 189 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py36:validatesRunnerStreamingTests'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 3: Task failed with an exception. ----------- * Where: Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 189 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py38:validatesRunnerStreamingTests'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 41m 5s 92 actionable tasks: 62 executed, 30 from cache Publishing build scan... https://gradle.com/s/jatucj77kkryo Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
