See <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/1001/display/redirect?page=changes>
Changes: [noreply] [BEAM-12368] Updated doc nav to match wireframe (#15229) [noreply] [BEAM-3713] Move PostCommit_Python from nosetest to pytest (#14859) [noreply] [BEAM-12653] Fix container cleanup for Java Dataflow tests (#15213) ------------------------------------------ [...truncated 456.75 KB...] pytz==2020.1 PyYAML==5.4 requests==2.24.0 requests-oauthlib==1.3.0 rsa==4.7.2 scikit-learn==0.24.1 scipy==1.4.1 setuptools==57.4.0 six==1.15.0 soupsieve==2.2.1 tenacity==8.0.1 tensorboard==2.5.0 tensorboard-data-server==0.6.1 tensorboard-plugin-wit==1.8.0 tensorflow==2.5.0 tensorflow-estimator==2.5.0 termcolor==1.1.0 threadpoolctl==2.2.0 typing-extensions==3.7.4.3 uritemplate==3.0.1 urllib3==1.25.11 wcwidth==0.2.5 Werkzeug==2.0.1 wheel==0.36.2 wrapt==1.12.1 zipp==3.5.0 Removing intermediate container 6efaf369254e ---> 8178deb48122 Step 22/27 : RUN pip check ---> Running in aa4ca8e31dc8 No broken requirements found. Removing intermediate container aa4ca8e31dc8 ---> b9895a52fd7c Step 23/27 : COPY target/LICENSE /opt/apache/beam/ ---> a31650036401 Step 24/27 : COPY target/LICENSE.python /opt/apache/beam/ ---> 8cef7280cb23 Step 25/27 : COPY target/NOTICE /opt/apache/beam/ ---> e4eb5579ba1e Step 26/27 : ADD target/launcher/linux_amd64/boot /opt/apache/beam/ ---> 6e230d962ad4 Step 27/27 : ENTRYPOINT ["/opt/apache/beam/boot"] ---> Running in 0d41dbf3edd4 Removing intermediate container 0d41dbf3edd4 ---> 9883bdd87165 Successfully built 9883bdd87165 Successfully tagged apache/beam_python3.6_sdk:2.33.0.dev > Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerSetup Launching Java expansion service @ 36655 Launching Python expansion service @ 44341 > Task > :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava >>> RUNNING integration tests with pipeline options: >>> --runner=TestDataflowRunner --project=apache-beam-testing >>> --region=us-central1 >>> --sdk_harness_container_image_overrides=.*java.*,us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210728000117 >>> --experiments=use_runner_v2 --experiments=shuffle_mode=appliance >>> --sdk_location=<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz> >>> pytest options: --capture=no --numprocesses=8 --timeout=4500 >>> --log-cli-level=INFO >>> collect markers: -m=xlang_transforms ============================= test session starts ============================== platform linux -- Python 3.6.8, pytest-4.6.11, py-1.10.0, pluggy-0.13.1 rootdir: <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python,> inifile: pytest.ini plugins: xdist-1.34.0, timeout-1.4.2, forked-1.3.0, requests-mock-1.9.3 timeout: 4500.0s timeout method: signal timeout func_only: False gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I [gw0] Python 3.6.8 (default, Dec 24 2018, 19:24:27) -- [GCC 5.4.0 20160609] [gw1] Python 3.6.8 (default, Dec 24 2018, 19:24:27) -- [GCC 5.4.0 20160609] [gw2] Python 3.6.8 (default, Dec 24 2018, 19:24:27) -- [GCC 5.4.0 20160609] [gw6] Python 3.6.8 (default, Dec 24 2018, 19:24:27) -- [GCC 5.4.0 20160609] [gw4] Python 3.6.8 (default, Dec 24 2018, 19:24:27) -- [GCC 5.4.0 20160609] [gw3] Python 3.6.8 (default, Dec 24 2018, 19:24:27) -- [GCC 5.4.0 20160609] [gw5] Python 3.6.8 (default, Dec 24 2018, 19:24:27) -- [GCC 5.4.0 20160609] [gw7] Python 3.6.8 (default, Dec 24 2018, 19:24:27) -- [GCC 5.4.0 20160609] gw0 [8] / gw1 [8] / gw2 [8] / gw3 [8] / gw4 [8] / gw5 [8] / gw6 [8] / gw7 [8] scheduling tests via LoadScheduling apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_group_by_key apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_cogroup_by_key apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_combine_per_key apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_prefix apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_combine_globally apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_multi_input_output_with_sideinput apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_partition apache_beam/io/external/generate_sequence_test.py::XlangGenerateSequenceTest::test_generate_sequence [gw4] FAILED apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_group_by_key [gw7] PASSED apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_prefix [gw6] PASSED apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_combine_per_key [gw3] PASSED apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_multi_input_output_with_sideinput [gw0] PASSED apache_beam/io/external/generate_sequence_test.py::XlangGenerateSequenceTest::test_generate_sequence [gw1] PASSED apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_cogroup_by_key [gw2] PASSED apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_combine_globally [gw5] PASSED apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_partition =================================== FAILURES =================================== __________________ ValidateRunnerXlangTest.test_group_by_key ___________________ [gw4] linux -- Python 3.6.8 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/bin/python3.6> self = <apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest testMethod=test_group_by_key> test_pipeline = None @pytest.mark.xlang_transforms def test_group_by_key(self, test_pipeline=None): CrossLanguageTestPipelines().run_group_by_key( > test_pipeline or self.create_pipeline()) apache_beam/transforms/validate_runner_xlang_test.py:264: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ apache_beam/transforms/validate_runner_xlang_test.py:141: in run_group_by_key assert_that(res, equal_to(['0:1,2', '1:3'])) apache_beam/pipeline.py:585: in __exit__ self.result = self.run() apache_beam/testing/test_pipeline.py:114: in run False if self.not_use_test_runner_api else test_runner_api)) apache_beam/pipeline.py:564: in run return self.runner.run_pipeline(self, self._options) apache_beam/runners/dataflow/test_dataflow_runner.py:54: in run_pipeline self).run_pipeline(pipeline, options) apache_beam/runners/dataflow/dataflow_runner.py:588: in run_pipeline self.dataflow_client.create_job(self.job), self) apache_beam/utils/retry.py:253: in wrapper return fun(*args, **kwargs) apache_beam/runners/dataflow/internal/apiclient.py:688: in create_job return self.submit_job_description(job) apache_beam/utils/retry.py:253: in wrapper return fun(*args, **kwargs) apache_beam/runners/dataflow/internal/apiclient.py:789: in submit_job_description response = self._client.projects_locations_jobs.Create(request) apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py:903: in Create config, request, global_params=global_params) ../../build/gradleenv/1922375555/lib/python3.6/site-packages/apitools/base/py/base_api.py:731: in _RunMethod return self.ProcessHttpResponse(method_config, http_response, request) ../../build/gradleenv/1922375555/lib/python3.6/site-packages/apitools/base/py/base_api.py:737: in ProcessHttpResponse self.__ProcessHttpResponse(method_config, http_response, request)) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3.ProjectsLocationsJobsService object at 0x7fa3099510f0> method_config = <ApiMethodInfo relative_path: 'v1b3/projects/{projectId}/locations/{location}/jobs' method_id: 'dataflow.projects.lo...DataflowProjectsLocationsJobsCreateRequest' response_type_name: 'Job' request_field: 'job' supports_download: False> http_response = Response(info={'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Wed, 2...', request_url='https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json') request = <DataflowProjectsLocationsJobsCreateRequest job: <Job environment: <Environment clusterManagerApiService: 'compute....empFiles: [] type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> location: 'us-central1' projectId: 'apache-beam-testing'> def __ProcessHttpResponse(self, method_config, http_response, request): """Process the given http response.""" if http_response.status_code not in (http_client.OK, http_client.CREATED, http_client.NO_CONTENT): raise exceptions.HttpError.FromResponse( > http_response, method_config=method_config, request=request) E apitools.base.py.exceptions.HttpConflictError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Wed, 28 Jul 2021 00:19:02 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'status': '409', 'content-length': '318', '-content-encoding': 'gzip'}>, content <{ E "error": { E "code": 409, E "message": "(e51c5ef0a5c38e2a): The workflow could not be created. Causes: (fe27f2e38dbb06ec): There is already an active job named beamapp-jenkins-0728001854-453977. If you want to submit a second job, try again by setting a different name.", E "status": "ALREADY_EXISTS" E } E } E > ../../build/gradleenv/1922375555/lib/python3.6/site-packages/apitools/base/py/base_api.py:604: HttpConflictError ------------------------------ Captured log call ------------------------------- INFO apache_beam.runners.portability.stager:stager.py:300 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz";> to staging location. WARNING root:environments.py:374 Make sure that locally built Python SDK docker image has Python 3.6 interpreter. INFO root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.6_sdk:2.33.0.dev INFO root:environments.py:296 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python36-fnapi:beam-master-20210720 INFO root:environments.py:304 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python36-fnapi:beam-master-20210720" for Docker environment INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:650 ==================== <function pack_combiners at 0x7fa30b9a8840> ==================== INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:650 ==================== <function sort_stages at 0x7fa30b9a9048> ==================== INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:452 Defaulting to the temp_location as staging_location: gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494 INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/beam-sdks-java-testing-expansion-service-testExpansionService-2.33.0-SNAPSHOT-IUJJ3amQP6N2TQt1oDYEJZgtvEfgfR1qoX-LcVQA2dE.jar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/beam-sdks-java-testing-expansion-service-testExpansionService-2.33.0-SNAPSHOT-IUJJ3amQP6N2TQt1oDYEJZgtvEfgfR1qoX-LcVQA2dE.jar in 5 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/dataflow_python_sdk.tar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/dataflow_python_sdk.tar in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/pipeline.pb... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0728001854-453977.1627431534.454836/pipeline.pb in 0 seconds. =============================== warnings summary =============================== apache_beam/io/filesystems_test.py:54 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string apache_beam/io/filesystems_test.py:62 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string -- Docs: https://docs.pytest.org/en/latest/warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/pytest_xlangValidateRunner.xml> - =============== 1 failed, 7 passed, 2 warnings in 684.03 seconds =============== > Task > :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava > FAILED > Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerCleanup Stopping expansion service pid: 18329. Stopping expansion service pid: 18332. > Task :runners:google-cloud-dataflow-java:cleanUpDockerImages Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210728000117 Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:d1fbc30f935c2cb81542b5e204d1df76a8e77f675e39fa7e5df7b099ee73d34f Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210728000117] - referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:d1fbc30f935c2cb81542b5e204d1df76a8e77f675e39fa7e5df7b099ee73d34f] Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210728000117] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:d1fbc30f935c2cb81542b5e204d1df76a8e77f675e39fa7e5df7b099ee73d34f])]. Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:d1fbc30f935c2cb81542b5e204d1df76a8e77f675e39fa7e5df7b099ee73d34f Digests: - us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:d1fbc30f935c2cb81542b5e204d1df76a8e77f675e39fa7e5df7b099ee73d34f Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:d1fbc30f935c2cb81542b5e204d1df76a8e77f675e39fa7e5df7b099ee73d34f]. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 29m 14s 113 actionable tasks: 84 executed, 25 from cache, 4 up-to-date Publishing build scan... https://gradle.com/s/4d4hcbzs7zix6 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
