See
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/9782/display/redirect?page=changes>
Changes:
[noreply] Fix broken link in online clustering documentation (#23516)
[noreply] Grant actions using GITHUB_TOKEN the appropriate permission set
(#23521)
[noreply] Fix failing Py37 BQ file loads test (#23334)
[noreply] [Website] update links to https (#23523)
------------------------------------------
[...truncated 1.33 MB...]
echo ">>> Successfully built and push container $CONTAINER"
cd sdks/python
SDK_LOCATION=$2
# Run ValidatesRunner tests on Google Cloud Dataflow service
echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST"
pytest -o junit_suite_name=$IMAGE_NAME \
-m="it_validatescontainer" \
--show-capture=no \
--numprocesses=1 \
--timeout=900 \
--junitxml=$XUNIT_FILE \
--ignore-glob '.*py3\d?\.py$' \
--log-cli-level=INFO \
--test-pipeline-options=" \
--runner=TestDataflowRunner \
--project=$PROJECT \
--region=$REGION \
--sdk_container_image=$CONTAINER:$TAG \
--staging_location=$GCS_LOCATION/staging-validatesrunner-test \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--output=$GCS_LOCATION/output \
--sdk_location=$SDK_LOCATION \
--num_workers=1 \
--docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH"
> Task :sdks:python:test-suites:dataflow:py38:validatesContainer
echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST"
cleanup_container
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:ab21f64df44d6dc26c4324069f8c707b6c57120c5dd917982651ab2ffe850cbf
Associated tags:
- 20221006-165358134538652
Tags:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20221006-165358134538652
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20221006-165358134538652].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:ab21f64df44d6dc26c4324069f8c707b6c57120c5dd917982651ab2ffe850cbf].
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:c6c13611483cc713b6313aaa3227dbaa955b498a3ae64bc4ad9bb392bf4e7ee1
Associated tags:
- 87cc568a-bdfd-4bbf-8b92-59593311f12b
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk:87cc568a-bdfd-4bbf-8b92-59593311f12b].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:c6c13611483cc713b6313aaa3227dbaa955b498a3ae64bc4ad9bb392bf4e7ee1].
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:5eadb7052f4835237557400a2ad2936f7a93aea77335f2d18d6d8562cf3fa9d4
Associated tags:
- 9675bc68-d85d-43fc-af4b-f8e7d361985a
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk:9675bc68-d85d-43fc-af4b-f8e7d361985a].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:5eadb7052f4835237557400a2ad2936f7a93aea77335f2d18d6d8562cf3fa9d4].
> Task :sdks:python:test-suites:dataflow:py39:validatesContainer
[gw0] [100%] PASSED
apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py::ExerciseMetricsPipelineTest::test_metrics_fnapi_it
=================================== FAILURES ===================================
____ WordCountIT.test_wordcount_it_with_prebuilt_sdk_container_cloud_build _____
[gw0] linux -- Python 3.9.10
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/build/gradleenv/-1734967050/bin/python3.9>
self = <apache_beam.examples.wordcount_it_test.WordCountIT
testMethod=test_wordcount_it_with_prebuilt_sdk_container_cloud_build>
@pytest.mark.it_validatescontainer
def test_wordcount_it_with_prebuilt_sdk_container_cloud_build(self):
> self._run_wordcount_it(
wordcount.run,
experiment='beam_fn_api',
prebuild_sdk_container_engine='cloud_build')
apache_beam/examples/wordcount_it_test.py:103:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam/examples/wordcount_it_test.py:146: in _run_wordcount_it
run_wordcount(
apache_beam/examples/wordcount.py:106: in run
output | 'Write' >> WriteToText(known_args.output)
apache_beam/pipeline.py:597: in __exit__
self.result = self.run()
apache_beam/pipeline.py:547: in run
return Pipeline.from_runner_api(
apache_beam/pipeline.py:574: in run
return self.runner.run_pipeline(self, self._options)
apache_beam/runners/dataflow/test_dataflow_runner.py:66: in run_pipeline
self.result.wait_until_finish(duration=wait_duration)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <DataflowPipelineResult <Job
clientRequestId: '20221006171416823258-9986'
createTime: '2022-10-06T17:14:18.067216Z'
...022-10-06T17:14:18.067216Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> at 0x7f18e1222d60>
duration = None
def wait_until_finish(self, duration=None):
if not self.is_in_terminal_state():
if not self.has_job:
raise IOError('Failed to get the Dataflow job id.')
consoleUrl = (
"Console URL: https://console.cloud.google.com/";
f"dataflow/jobs/<RegionId>/{self.job_id()}"
"?project=<ProjectId>")
thread = threading.Thread(
target=DataflowRunner.poll_for_job_completion,
args=(self._runner, self, duration))
# Mark the thread as a daemon thread so a keyboard interrupt on the main
# thread will terminate everything. This is also the reason we will not
# use thread.join() to wait for the polling thread.
thread.daemon = True
thread.start()
while thread.is_alive():
> time.sleep(5.0)
E Failed: Timeout >900.0s
apache_beam/runners/dataflow/dataflow_runner.py:1660: Failed
=============================== warnings summary ===============================
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15:
DeprecationWarning: the imp module is deprecated in favour of importlib; see
the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42:
DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use
"async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66:
FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas
in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89:
FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas
in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90:
FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas
in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/pytest-beam_python3.9_sdk.xml>
-
=========================== short test summary info ============================
FAILED
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_cloud_build
======= 1 failed, 3 passed, 3 skipped, 6 warnings in 2403.24s (0:40:03) ========
cleanup_container
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk:20221006-165358278525054
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk@sha256:a036ec935b44ff5bfd230b33f288b956360e688c3cf1baf8cf8e56fd34563f2e
Untagged:
us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk:78f946e2-18ef-4918-a370-770bd0f97237
Untagged:
us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:22f8b8c529f2bc68163cbce61bed75aa41aa1a2ed08ad34588fae07e2319fe49
Deleted: sha256:10d604be50ad174be40525e47247988f2af9dc3193f4c28f6159bda16a76d052
Deleted: sha256:7d25b588e75b49261dd73d40147951752a7a65af1541d99977fedd83ec5e439c
Deleted: sha256:2c82ed924e61a665464b361dd103101279a798d41af5772982b60dab15d122cd
Deleted: sha256:225a9d4e3231c8d093ceb1ba12eea35d457124d55b8cdc45c28b2d0881c0ec44
Deleted: sha256:c94299225f9bf0790df5f5f7f6cf8cce69f86195b62fda917a0613882ebc97ee
Deleted: sha256:bf732a3b856625c2dd67ff2892902ef39038e18183d8fc26d6bdaf10677419e8
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk@sha256:a036ec935b44ff5bfd230b33f288b956360e688c3cf1baf8cf8e56fd34563f2e
Associated tags:
- 20221006-165358278525054
Tags:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk:20221006-165358278525054
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk:20221006-165358278525054].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk@sha256:a036ec935b44ff5bfd230b33f288b956360e688c3cf1baf8cf8e56fd34563f2e].
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:22f8b8c529f2bc68163cbce61bed75aa41aa1a2ed08ad34588fae07e2319fe49
Associated tags:
- 78f946e2-18ef-4918-a370-770bd0f97237
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk:78f946e2-18ef-4918-a370-770bd0f97237].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:22f8b8c529f2bc68163cbce61bed75aa41aa1a2ed08ad34588fae07e2319fe49].
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:9b7ebcb8dedd7146e31b26b7bc7557c035bb0c73818ee9fd7b5b291e90fc8081
Associated tags:
- 270b3b9e-cdf3-4ed7-8230-fab9c52e4644
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk:270b3b9e-cdf3-4ed7-8230-fab9c52e4644].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:9b7ebcb8dedd7146e31b26b7bc7557c035bb0c73818ee9fd7b5b291e90fc8081].
Removed the container
> Task :sdks:python:test-suites:dataflow:py39:validatesContainer FAILED
> Task :sdks:python:test-suites:dataflow:py37:validatesContainer
cleanup_container
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk@sha256:608bb40a527dcfab19fea18873873829bd59f378617792003c6c279904a17c00
Associated tags:
- 20221006-165358152136630
Tags:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk:20221006-165358152136630
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk:20221006-165358152136630].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk@sha256:608bb40a527dcfab19fea18873873829bd59f378617792003c6c279904a17c00].
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:2cdd252d620d973fa104f9b6535f892c8b0d1ab00a7be7ee4941166999dd8c29
Associated tags:
- c5eb415f-f165-438d-8f08-ef25000f64c9
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk:c5eb415f-f165-438d-8f08-ef25000f64c9].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:2cdd252d620d973fa104f9b6535f892c8b0d1ab00a7be7ee4941166999dd8c29].
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:d9fbb623502630780f55a25570548b069f3b0be2507931860a88bb77e24d28f4
Associated tags:
- 34b6f9e5-d69c-4146-98f9-7663888361ff
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk:34b6f9e5-d69c-4146-98f9-7663888361ff].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:d9fbb623502630780f55a25570548b069f3b0be2507931860a88bb77e24d28f4].
> Task :sdks:python:test-suites:dataflow:py37:validatesContainer FAILED
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/test-suites/dataflow/common.gradle'>
line: 359
* What went wrong:
Execution failed for task
':sdks:python:test-suites:dataflow:py39:validatesContainer'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/test-suites/dataflow/common.gradle'>
line: 359
* What went wrong:
Execution failed for task
':sdks:python:test-suites:dataflow:py37:validatesContainer'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
See
https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 56m 36s
51 actionable tasks: 41 executed, 4 from cache, 6 up-to-date
Publishing build scan...
https://gradle.com/s/t6mrknwq6wmve
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
