See <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/153/display/redirect>
Changes: ------------------------------------------ [...truncated 2.01 MB...] [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-06-09T02:26:14.474Z: JOB_MESSAGE_BASIC: Worker pool stopped. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-06-09T02:26:14.493Z: JOB_MESSAGE_DEBUG: Tearing down pending resources... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:193 Job 2023-06-08_18_36_26-16900237887035727023 is in state JOB_STATE_FAILED [1m[31mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:1563 Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2023-06-08_18_36_26-16900237887035727023?project=<ProjectId> [33m=============================== warnings summary ===============================[0m ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses from imp import load_source ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:121: DeprecationWarning: pkg_resources is deprecated as an API warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: 162 warnings <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: 117 warnings <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.cloud')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2349: 36 warnings <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2349: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(parent) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.logging')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.iam')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py>:20: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.rpc')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages pkg_resources.declare_namespace(__name__) apache_beam/typehints/pandas_type_compatibility_test.py:67 apache_beam/typehints/pandas_type_compatibility_test.py:67 apache_beam/typehints/pandas_type_compatibility_test.py:67 apache_beam/typehints/pandas_type_compatibility_test.py:67 apache_beam/typehints/pandas_type_compatibility_test.py:67 apache_beam/typehints/pandas_type_compatibility_test.py:67 apache_beam/typehints/pandas_type_compatibility_test.py:67 apache_beam/typehints/pandas_type_compatibility_test.py:67 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:67: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. }).set_index(pd.Int64Index(range(123, 223), name='an_index')), apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. pd.Int64Index(range(123, 223), name='an_index'), apache_beam/typehints/pandas_type_compatibility_test.py:91 apache_beam/typehints/pandas_type_compatibility_test.py:91 apache_beam/typehints/pandas_type_compatibility_test.py:91 apache_beam/typehints/pandas_type_compatibility_test.py:91 apache_beam/typehints/pandas_type_compatibility_test.py:91 apache_beam/typehints/pandas_type_compatibility_test.py:91 apache_beam/typehints/pandas_type_compatibility_test.py:91 apache_beam/typehints/pandas_type_compatibility_test.py:91 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:91: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. pd.Int64Index(range(475, 575), name='another_index'), apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py::CloudMLTFTBenchmarkTest::test_cloudml_benchmark_cirteo_no_shuffle_10GB apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py::CloudMLTFTBenchmarkTest::test_cloudml_benchmark_criteo_small <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py>:484: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead. bq_dataset_ref = self._client.dataset(dataset_name) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/pytest_TFTransformTests-df-py39.xml> - [36m[1m=========================== short test summary info ============================[0m [31mFAILED[0m apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py::[1mCloudMLTFTBenchmarkTest::test_cloudml_benchmark_criteo_10GB[0m - apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error: Traceback (most recent call last): File "apache_beam/runners/common.py", line 1418, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 838, in apache_beam.runners.common.PerWindowInvoker.invoke_process File "apache_beam/runners/common.py", line 984, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window File "<https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/apache_beam/transforms/core.py";,> line 1961, in <lambda> wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)] File "<https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tensorflow_transform/beam/tft_beam_io/transform_fn_io.py";,> line 105, in publish_outputs tf.io.gfile.rename(metadata_source_path, metadata_path, overwrite=True) File "/usr/local/lib/python3.9/site-packages/tensorflow/python/lib/io/file_io.py", line 622, in rename_v2 _pywrap_file_io.RenameFile( tensorflow.python.framework.errors_impl.NotFoundError: Error executing an HTTP request: HTTP response code 404 with body '{ "error": { "code": 404, "message": "No such object: temp-storage-for-end-to-end-tests/tft/4ec7e24fb97e4823905a3abd6989ed84/transform_tmp/feab29ad774e46059edf78d8b85fd21b", "errors": [ { "message": "No such object: temp-storage-for-end-to-end-tests/tft/4ec7e24fb97e4823905a3abd6989ed84/transform_tmp/feab29ad774e46059edf78d8b85fd21b", "domain": "global", "reason": "notFound" } ] } } ' when renaming gs://temp-storage-for-end-to-end-tests/tft/4ec7e24fb97e4823905a3abd6989ed84/transform_tmp/feab29ad774e46059edf78d8b85fd21b to gs://temp-storage-for-end-to-end-tests/tft/4ec7e24fb97e4823905a3abd6989ed84/transformed_metadata During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 295, in _execute response = task() File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 370, in <lambda> lambda: self.create_worker().do_instruction(request), request) File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 629, in do_instruction return getattr(self, request_type)( File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 667, in process_bundle bundle_processor.process_bundle(instruction_id)) File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1061, in process_bundle input_op_by_transform_id[element.transform_id].process_encoded( File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 231, in process_encoded self.output(decoded_value) File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process File "apache_beam/runners/common.py", line 1420, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 1492, in apache_beam.runners.common.DoFnRunner._reraise_augmented File "apache_beam/runners/common.py", line 1418, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 624, in apache_beam.runners.common.SimpleInvoker.invoke_process File "apache_beam/runners/common.py", line 1582, in apache_beam.runners.common._OutputHandler.handle_process_outputs File "apache_beam/runners/common.py", line 1695, in apache_beam.runners.common._OutputHandler._write_value_to_tag File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process File "apache_beam/runners/common.py", line 1420, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 1492, in apache_beam.runners.common.DoFnRunner._reraise_augmented File "apache_beam/runners/common.py", line 1418, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 624, in apache_beam.runners.common.SimpleInvoker.invoke_process File "apache_beam/runners/common.py", line 1582, in apache_beam.runners.common._OutputHandler.handle_process_outputs File "apache_beam/runners/common.py", line 1695, in apache_beam.runners.common._OutputHandler._write_value_to_tag File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process File "apache_beam/runners/common.py", line 1420, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 1508, in apache_beam.runners.common.DoFnRunner._reraise_augmented File "apache_beam/runners/common.py", line 1418, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 838, in apache_beam.runners.common.PerWindowInvoker.invoke_process File "apache_beam/runners/common.py", line 984, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window File "<https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/apache_beam/transforms/core.py";,> line 1961, in <lambda> wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)] File "<https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tensorflow_transform/beam/tft_beam_io/transform_fn_io.py";,> line 105, in publish_outputs tf.io.gfile.rename(metadata_source_path, metadata_path, overwrite=True) File "/usr/local/lib/python3.9/site-packages/tensorflow/python/lib/io/file_io.py", line 622, in rename_v2 _pywrap_file_io.RenameFile( RuntimeError: tensorflow.python.framework.errors_impl.NotFoundError: Error executing an HTTP request: HTTP response code 404 with body '{ "error": { "code": 404, "message": "No such object: temp-storage-for-end-to-end-tests/tft/4ec7e24fb97e4823905a3abd6989ed84/transform_tmp/feab29ad774e46059edf78d8b85fd21b", "errors": [ { "message": "No such object: temp-storage-for-end-to-end-tests/tft/4ec7e24fb97e4823905a3abd6989ed84/transform_tmp/feab29ad774e46059edf78d8b85fd21b", "domain": "global", "reason": "notFound" } ] } } ' when renaming gs://temp-storage-for-end-to-end-tests/tft/4ec7e24fb97e4823905a3abd6989ed84/transform_tmp/feab29ad774e46059edf78d8b85fd21b to gs://temp-storage-for-end-to-end-tests/tft/4ec7e24fb97e4823905a3abd6989ed84/transformed_metadata [while running 'WriteTransformFn/PublishMetadataAndTransformFn-ptransform-27'] [31m====== [31m[1m1 failed[0m, [32m2 passed[0m, [33m8 skipped[0m, [33m386 warnings[0m[31m in 6978.00s (1:56:17)[0m[31m =======[0m > Task :sdks:python:test-suites:dataflow:py39:tftTests FAILED FAILURE: Build failed with an exception. * Where: Script '<https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 428 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py39:tftTests'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 1h 58m 58s 16 actionable tasks: 10 executed, 4 from cache, 2 up-to-date Publishing build scan... https://ge.apache.org/s/eflm44dcjznso Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
