See <https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/1035/display/redirect?page=changes>
Changes: [Moritz Mack] Minor improvements to the tpcds gradle build for Spark [Moritz Mack] Fix SparkSessionFactory to not fail when using Spark master local[*] [Moritz Mack] [Spark dataset runner] Add direct translation of Reshuffle and ------------------------------------------ [...truncated 480.11 KB...] > Task :runners:google-cloud-dataflow-java:buildAndPushDockerPythonContainer WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03. As an alternative, use `gcloud auth configure-docker` to configure `docker` to use `gcloud` as a credential helper, then use `docker` as you would for non-GCR registries, e.g. `docker pull gcr.io/project-id/my-image`. Add `--verbosity=error` to silence this warning: `gcloud docker --verbosity=error -- pull gcr.io/project-id/my-image`. See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker The push refers to repository [us.gcr.io/apache-beam-testing/java-postcommit-it/python] 17d87bc60c0a: Preparing 90809ae6a4ce: Preparing 958969ae6b9a: Preparing 5d7b806c04cc: Preparing 42b9b4d26ada: Preparing 6d4f60f58979: Preparing 1fba3b0c4557: Preparing 8bdb0d1e903f: Preparing cf6bca345794: Preparing a82e5008ff97: Preparing e70b634c9fe3: Preparing 216b7f66ff33: Preparing 4d656736c957: Preparing b1e4e5983221: Preparing 4cfc86a86dc9: Preparing 634bb2b2ca8e: Preparing 80bc379fe03f: Preparing 0c7daf9a72c8: Preparing 216b7f66ff33: Waiting 75ba02937496: Preparing 4d656736c957: Waiting 288cf3a46e32: Preparing 186da837555d: Preparing a82e5008ff97: Waiting 955c9335e041: Preparing e70b634c9fe3: Waiting 8e079fee2186: Preparing 634bb2b2ca8e: Waiting 80bc379fe03f: Waiting 0c7daf9a72c8: Waiting 288cf3a46e32: Waiting 75ba02937496: Waiting b1e4e5983221: Waiting 186da837555d: Waiting 6d4f60f58979: Waiting 4cfc86a86dc9: Waiting 8e079fee2186: Waiting 1fba3b0c4557: Waiting 955c9335e041: Waiting 8bdb0d1e903f: Waiting cf6bca345794: Waiting 42b9b4d26ada: Pushed 5d7b806c04cc: Pushed 958969ae6b9a: Pushed 17d87bc60c0a: Pushed 90809ae6a4ce: Pushed 1fba3b0c4557: Pushed 8bdb0d1e903f: Pushed cf6bca345794: Pushed 4d656736c957: Pushed b1e4e5983221: Pushed 4cfc86a86dc9: Layer already exists e70b634c9fe3: Pushed 634bb2b2ca8e: Layer already exists 0c7daf9a72c8: Layer already exists 80bc379fe03f: Layer already exists 288cf3a46e32: Layer already exists 75ba02937496: Layer already exists 186da837555d: Layer already exists 955c9335e041: Layer already exists 8e079fee2186: Layer already exists 6d4f60f58979: Pushed a82e5008ff97: Pushed 216b7f66ff33: Pushed 20221014165337: digest: sha256:f696c36db89380dee5a33a651c94bb61836e8e1a63b6630044ef06b17e59702b size: 5156 > Task :runners:google-cloud-dataflow-java:setupXVR > Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerSetup Launching Java expansion service @ 44029 Launching Python expansion service @ 39121 > Task > :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava >>> RUNNING integration tests with pipeline options: >>> --runner=TestDataflowRunner --project=apache-beam-testing >>> --region=us-central1 >>> --sdk_harness_container_image_overrides=.*java.*,us.gcr.io/apache-beam-testing/java-postcommit-it/java:20221014165337 >>> --experiments=shuffle_mode=appliance >>> --sdk_location=<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz> >>> pytest options: --capture=no --numprocesses=8 --timeout=4500 >>> --log-cli-level=INFO >>> collect markers: -m=uses_java_expansion_service ============================= test session starts ============================== platform linux -- Python 3.7.12, pytest-7.1.3, pluggy-1.0.0 rootdir: <https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/python,> configfile: pytest.ini plugins: xdist-2.5.0, timeout-2.1.0, hypothesis-6.56.2, forked-1.4.0, requests-mock-1.10.0 timeout: 4500.0s timeout method: signal timeout func_only: False gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I [gw0] Python 3.7.12 (default, Jan 15 2022, 18:42:10) -- [GCC 9.3.0] [gw2] Python 3.7.12 (default, Jan 15 2022, 18:42:10) -- [GCC 9.3.0] [gw7] Python 3.7.12 (default, Jan 15 2022, 18:42:10) -- [GCC 9.3.0] [gw1] Python 3.7.12 (default, Jan 15 2022, 18:42:10) -- [GCC 9.3.0] [gw3] Python 3.7.12 (default, Jan 15 2022, 18:42:10) -- [GCC 9.3.0] [gw6] Python 3.7.12 (default, Jan 15 2022, 18:42:10) -- [GCC 9.3.0] [gw4] Python 3.7.12 (default, Jan 15 2022, 18:42:10) -- [GCC 9.3.0] [gw5] Python 3.7.12 (default, Jan 15 2022, 18:42:10) -- [GCC 9.3.0] gw0 [10] / gw1 [10] / gw2 [10] / gw3 [10] / gw4 [10] / gw5 [10] / gw6 [10] / gw7 [10] scheduling tests via LoadScheduling apache_beam/io/external/generate_sequence_test.py::XlangGenerateSequenceTest::test_generate_sequence_java_class_lookup_payload_builder apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_cogroup_by_key apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_multi_input_output_with_sideinput apache_beam/io/external/generate_sequence_test.py::XlangGenerateSequenceTest::test_generate_sequence_java_external_transform apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_group_by_key apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_combine_globally apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_combine_per_key apache_beam/io/external/generate_sequence_test.py::XlangGenerateSequenceTest::test_generate_sequence [gw5] PASSED apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_multi_input_output_with_sideinput [gw4] PASSED apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_group_by_key [gw7] PASSED apache_beam/io/external/generate_sequence_test.py::XlangGenerateSequenceTest::test_generate_sequence_java_external_transform [gw0] PASSED apache_beam/io/external/generate_sequence_test.py::XlangGenerateSequenceTest::test_generate_sequence apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_partition [gw3] PASSED apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_combine_globally [gw6] PASSED apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_combine_per_key [gw2] PASSED apache_beam/io/external/generate_sequence_test.py::XlangGenerateSequenceTest::test_generate_sequence_java_class_lookup_payload_builder apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_prefix [gw1] PASSED apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_cogroup_by_key [gw0] PASSED apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_partition [gw2] PASSED apache_beam/transforms/validate_runner_xlang_test.py::ValidateRunnerXlangTest::test_prefix =============================== warnings summary =============================== ../../build/gradleenv/1922375555/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/1922375555/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/1922375555/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/1922375555/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/1922375555/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/1922375555/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/1922375555/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/1922375555/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/1922375555/lib/python3.7/site-packages/hdfs/config.py:15 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.7/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses from imp import load_source -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/python/pytest_xlangValidateRunner.xml> - ============ 10 passed, 3 skipped, 9 warnings in 1612.93s (0:26:52) ============ > Task :runners:google-cloud-dataflow-java:cleanupXVR UP-TO-DATE > Task :runners:google-cloud-dataflow-java:cleanUpDockerPythonImages Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/python:20221014165337 Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/python@sha256:f696c36db89380dee5a33a651c94bb61836e8e1a63b6630044ef06b17e59702b Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/python:20221014165337] - referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/python@sha256:f696c36db89380dee5a33a651c94bb61836e8e1a63b6630044ef06b17e59702b] Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/python:20221014165337] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/python@sha256:f696c36db89380dee5a33a651c94bb61836e8e1a63b6630044ef06b17e59702b])]. Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/python@sha256:f696c36db89380dee5a33a651c94bb61836e8e1a63b6630044ef06b17e59702b Digests: - us.gcr.io/apache-beam-testing/java-postcommit-it/python@sha256:f696c36db89380dee5a33a651c94bb61836e8e1a63b6630044ef06b17e59702b Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/python@sha256:f696c36db89380dee5a33a651c94bb61836e8e1a63b6630044ef06b17e59702b]. > Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20221014165337 Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:50e190e5e34a7f48a9fdb799ca740fd9e1f326daa8d0cb83d92d0653f76b809b Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20221014165337] - referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:50e190e5e34a7f48a9fdb799ca740fd9e1f326daa8d0cb83d92d0653f76b809b] Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20221014165337] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:50e190e5e34a7f48a9fdb799ca740fd9e1f326daa8d0cb83d92d0653f76b809b])]. Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:50e190e5e34a7f48a9fdb799ca740fd9e1f326daa8d0cb83d92d0653f76b809b Digests: - us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:50e190e5e34a7f48a9fdb799ca740fd9e1f326daa8d0cb83d92d0653f76b809b Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:50e190e5e34a7f48a9fdb799ca740fd9e1f326daa8d0cb83d92d0653f76b809b]. > Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerCleanup Stopping expansion service pid: 1304013. Stopping expansion service pid: 1304014. Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness. Please consult deprecation warnings for more details. BUILD SUCCESSFUL in 42m 8s 139 actionable tasks: 95 executed, 38 from cache, 6 up-to-date Publishing build scan... https://gradle.com/s/nsdd64prgjqrs Build step 'Invoke Gradle script' changed build result to SUCCESS [beam_PostCommit_XVR_PythonUsingJava_Dataflow] $ /bin/bash -xe /tmp/jenkins4778971110522086329.sh + echo '*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA EXTERNAL TRANSFORMS USING PYTHON 3.10 ***' *** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA EXTERNAL TRANSFORMS USING PYTHON 3.10 *** [Gradle] - Launching build. [src] $ <https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/gradlew> --continue --max-workers=8 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx6g -Dorg.gradle.vfs.watch=false -Pdocker-pull-licenses -PpythonVersion=3.10 :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava Configuration on demand is an incubating feature. > Task :buildSrc:compileJava NO-SOURCE > Task :buildSrc:compileGroovy UP-TO-DATE > Task :buildSrc:pluginDescriptors UP-TO-DATE > Task :buildSrc:processResources UP-TO-DATE > Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE > Task :buildSrc:assemble UP-TO-DATE > Task :buildSrc:spotlessGroovy UP-TO-DATE > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE > Task :buildSrc:spotlessGroovyGradle UP-TO-DATE > Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE > Task :buildSrc:spotlessCheck UP-TO-DATE > Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE > Task :buildSrc:compileTestJava NO-SOURCE > Task :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test NO-SOURCE > Task :buildSrc:validatePlugins UP-TO-DATE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build UP-TO-DATE System Go installation: /snap/bin/go is go version go1.16.15 linux/amd64; Preparing to use /home/jenkins/go/bin/go1.18.1 go1.18.1: already downloaded in /home/jenkins/sdk/go1.18.1 GOCMD=/home/jenkins/go/bin/go1.18.1 FAILURE: Build failed with an exception. * What went wrong: Could not determine the dependencies of task ':runners:google-cloud-dataflow-java:buildAndPushDockerPythonContainer'. > Task with path ':sdks:python:container:py310:docker' not found in project > ':runners:google-cloud-dataflow-java'. * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 3s 10 actionable tasks: 10 up-to-date Publishing build scan... https://gradle.com/s/53o72ssqe56f2 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
