See <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/9814/display/redirect?page=changes>
Changes: [Moritz Mack] Minor improvements to the tpcds gradle build for Spark [Moritz Mack] Fix SparkSessionFactory to not fail when using Spark master local[*] [Moritz Mack] [Spark dataset runner] Add direct translation of Reshuffle and ------------------------------------------ [...truncated 40.53 KB...] --verbosity=error -- pull gcr.io/project-id/my-image`. See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker > Task :sdks:python:test-suites:dataflow:py37:validatesContainer function cleanup_container { # Delete the container locally and remotely docker rmi $CONTAINER:$TAG || echo "Failed to remove container image" for image in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH) do docker rmi $image || echo "Failed to remove prebuilt sdk container image" done gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to delete container" for digest in $(gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk --format="get(digest)") do gcloud container images delete $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk@$digest --force-delete-tags --quiet || echo "Failed to remove prebuilt sdk container image" done echo "Removed the container" } trap cleanup_container EXIT echo ">>> Successfully built and push container $CONTAINER" cd sdks/python SDK_LOCATION=$2 # Run ValidatesRunner tests on Google Cloud Dataflow service echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST" pytest -o junit_suite_name=$IMAGE_NAME \ -m="it_validatescontainer" \ --show-capture=no \ --numprocesses=1 \ --timeout=900 \ --junitxml=$XUNIT_FILE \ --ignore-glob '.*py3\d?\.py$' \ --log-cli-level=INFO \ --test-pipeline-options=" \ --runner=TestDataflowRunner \ --project=$PROJECT \ --region=$REGION \ --sdk_container_image=$CONTAINER:$TAG \ --staging_location=$GCS_LOCATION/staging-validatesrunner-test \ --temp_location=$GCS_LOCATION/temp-validatesrunner-test \ --output=$GCS_LOCATION/output \ --sdk_location=$SDK_LOCATION \ --num_workers=1 \ --docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH" > Task :sdks:python:test-suites:dataflow:py38:validatesContainer function cleanup_container { # Delete the container locally and remotely docker rmi $CONTAINER:$TAG || echo "Failed to remove container image" for image in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH) do docker rmi $image || echo "Failed to remove prebuilt sdk container image" done gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to delete container" for digest in $(gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk --format="get(digest)") do gcloud container images delete $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk@$digest --force-delete-tags --quiet || echo "Failed to remove prebuilt sdk container image" done echo "Removed the container" } trap cleanup_container EXIT echo ">>> Successfully built and push container $CONTAINER" cd sdks/python SDK_LOCATION=$2 # Run ValidatesRunner tests on Google Cloud Dataflow service echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST" pytest -o junit_suite_name=$IMAGE_NAME \ -m="it_validatescontainer" \ --show-capture=no \ --numprocesses=1 \ --timeout=900 \ --junitxml=$XUNIT_FILE \ --ignore-glob '.*py3\d?\.py$' \ --log-cli-level=INFO \ --test-pipeline-options=" \ --runner=TestDataflowRunner \ --project=$PROJECT \ --region=$REGION \ --sdk_container_image=$CONTAINER:$TAG \ --staging_location=$GCS_LOCATION/staging-validatesrunner-test \ --temp_location=$GCS_LOCATION/temp-validatesrunner-test \ --output=$GCS_LOCATION/output \ --sdk_location=$SDK_LOCATION \ --num_workers=1 \ --docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH" > Task :sdks:python:test-suites:dataflow:py39:validatesContainer function cleanup_container { # Delete the container locally and remotely docker rmi $CONTAINER:$TAG || echo "Failed to remove container image" for image in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH) do docker rmi $image || echo "Failed to remove prebuilt sdk container image" done gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to delete container" for digest in $(gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk --format="get(digest)") do gcloud container images delete $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk@$digest --force-delete-tags --quiet || echo "Failed to remove prebuilt sdk container image" done echo "Removed the container" } trap cleanup_container EXIT echo ">>> Successfully built and push container $CONTAINER" cd sdks/python SDK_LOCATION=$2 # Run ValidatesRunner tests on Google Cloud Dataflow service echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST" pytest -o junit_suite_name=$IMAGE_NAME \ -m="it_validatescontainer" \ --show-capture=no \ --numprocesses=1 \ --timeout=900 \ --junitxml=$XUNIT_FILE \ --ignore-glob '.*py3\d?\.py$' \ --log-cli-level=INFO \ --test-pipeline-options=" \ --runner=TestDataflowRunner \ --project=$PROJECT \ --region=$REGION \ --sdk_container_image=$CONTAINER:$TAG \ --staging_location=$GCS_LOCATION/staging-validatesrunner-test \ --temp_location=$GCS_LOCATION/temp-validatesrunner-test \ --output=$GCS_LOCATION/output \ --sdk_location=$SDK_LOCATION \ --num_workers=1 \ --docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH" > Task :sdks:python:test-suites:dataflow:py37:validatesContainer echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST" cleanup_container Digests: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk@sha256:45189ea4c773e4ad610cf0fe10d802e417245e29af99cf48ac63622c063cd3b0 Associated tags: - 20221014-165213477084889 Tags: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk:20221014-165213477084889 Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk:20221014-165213477084889]. Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk@sha256:45189ea4c773e4ad610cf0fe10d802e417245e29af99cf48ac63622c063cd3b0]. > Task :sdks:python:test-suites:dataflow:py38:validatesContainer [gw0] [ 75%] FAILED apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_local_docker apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py::ExerciseMetricsPipelineTest::test_metrics_fnapi_it > Task :sdks:python:test-suites:dataflow:py39:validatesContainer [gw0] [100%] PASSED apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py::ExerciseMetricsPipelineTest::test_metrics_fnapi_it =============================== warnings summary =============================== ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses from imp import load_source ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42 <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead def call(self, fn, *args, **kwargs): apache_beam/typehints/pandas_type_compatibility_test.py:66 <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. }).set_index(pd.Int64Index(range(123, 223), name='an_index')), apache_beam/typehints/pandas_type_compatibility_test.py:89 <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. pd.Int64Index(range(123, 223), name='an_index'), apache_beam/typehints/pandas_type_compatibility_test.py:90 <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. pd.Int64Index(range(475, 575), name='another_index'), -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/pytest-beam_python3.9_sdk.xml> - ============ 4 passed, 3 skipped, 6 warnings in 2478.17s (0:41:18) ============= >>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST" cleanup_container Untagged: us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk:20221014-165213730375998 Untagged: us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk@sha256:21a781c4efd16216b9cf25b20c9bb80d3fab37f541dfdbf188d3c06113707b03 Untagged: us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk:3a4564e4-d065-411c-967b-a7804324902f Untagged: us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:b74379cd12b2ebc74cee70c76b261972840c029aaf6560d5e5533bf6285d9d54 Deleted: sha256:8f02b5f58eed017019c502f3032056700f5964cc939c3212cc42be14c5781faa Deleted: sha256:0031ba1d3ad5a6410d915a2a89ac6b0b6f082e3aad725213aa1a5b31c776c72f Deleted: sha256:b38cdeade07d04ffb0f3c62420ace34b8d9979dcbf2b6b9a8600b7af8baaba3d Deleted: sha256:185c7085663307dac4c3a414682cc56ae471d0594d4c7683bc9f253c5bc227b7 Deleted: sha256:1955b8decd26442b495193e643b4a28f997c774a850774db29015a37f2873073 Deleted: sha256:85da377b44941917c2709c7288b6f448cf766762e8c8aee259109ecc44b25398 Digests: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk@sha256:21a781c4efd16216b9cf25b20c9bb80d3fab37f541dfdbf188d3c06113707b03 Associated tags: - 20221014-165213730375998 Tags: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk:20221014-165213730375998 Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk:20221014-165213730375998]. Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk@sha256:21a781c4efd16216b9cf25b20c9bb80d3fab37f541dfdbf188d3c06113707b03]. Removed the container > Task :sdks:python:test-suites:dataflow:py38:validatesContainer cleanup_container Digests: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:e22d07672c6e6848bd19986327aed0cf10fd5e7923cc7c1120ae9a07b8f85f2a Associated tags: - 20221014-165213611408270 Tags: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20221014-165213611408270 Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20221014-165213611408270]. Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:e22d07672c6e6848bd19986327aed0cf10fd5e7923cc7c1120ae9a07b8f85f2a]. > Task :sdks:python:test-suites:dataflow:py38:validatesContainer FAILED FAILURE: Build failed with an exception. * Where: Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 359 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py38:validatesContainer'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 1h 3m 49s 51 actionable tasks: 41 executed, 4 from cache, 6 up-to-date Publishing build scan... https://gradle.com/s/j7ifsao4tc6by Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
