See <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/3564/display/redirect?page=changes>
Changes: [noreply] Daily p0/p1/flaky reports for issues (#21725) [noreply] Remove dataframe warnings from py38-docs logs (#21861) [noreply] Update references to Jira to GH for the Java SDK (#21836) [noreply] [21709] - Fix for "beam_PostCommit_Java_ValidatesRunner_Samza Failing" [noreply] Update references to jira to GH for the Runners (#21835) [noreply] Update remaining references to Jira to GH (#21834) ------------------------------------------ [...truncated 52.22 KB...] > Task :runners:core-java:compileJava > Task :sdks:python:test-suites:dataflow:py39:installGcpTest Successfully installed apache-beam-2.41.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.24.1 azure-storage-blob-12.12.0 boto3-1.24.10 botocore-1.27.10 cachetools-4.2.4 certifi-2022.6.15 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.1.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.5.1 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.6 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.4 google-cloud-bigquery-storage-2.13.2 google-cloud-bigtable-1.7.2 google-cloud-core-1.7.2 google-cloud-datastore-1.15.5 google-cloud-dlp-3.7.1 google-cloud-language-1.3.2 google-cloud-pubsub-2.13.0 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.3 google-cloud-videointelligence-1.16.3 google-cloud-vision-1.0.2 google-crc32c-1.3.0 google-resumable-media-2.3.3 googleapis-common-protos-1.56.2 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.3 hdfs-2.7.0 httplib2-0.20.4 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.13.0 msrest-0.7.1 numpy-1.22.4 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.7.2 overrides-6.1.0 pandas-1.4.2 parameterized-0.7.5 pbr-5.9.0 pluggy-0.13.1 proto-plus-1.20.6 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.28.0 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.6.0 scikit-learn-1.1.1 scipy-1.8.1 sqlalchemy-1.4.37 tenacity-5.1.5 testcontainers-3.6.0 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1 > Task :runners:core-java:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :runners:core-java:classes > Task :runners:core-java:jar > Task :sdks:java:extensions:google-cloud-platform-core:compileJava FROM-CACHE > Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:jar > Task :sdks:java:harness:compileJava FROM-CACHE > Task :sdks:java:harness:classes UP-TO-DATE > Task :sdks:java:harness:jar > Task :runners:java-fn-execution:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :runners:java-fn-execution:classes > Task :runners:java-fn-execution:jar > Task :sdks:java:expansion-service:compileJava FROM-CACHE > Task :sdks:java:expansion-service:classes UP-TO-DATE > Task :sdks:java:expansion-service:jar > Task :sdks:java:io:kafka:compileJava FROM-CACHE > Task :sdks:java:io:kafka:classes UP-TO-DATE > Task :sdks:java:io:kafka:jar > Task :sdks:java:io:google-cloud-platform:compileJava FROM-CACHE > Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:jar > Task :runners:google-cloud-dataflow-java:compileJava FROM-CACHE > Task :runners:google-cloud-dataflow-java:classes > Task :runners:google-cloud-dataflow-java:jar > Task :runners:google-cloud-dataflow-java:worker:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :runners:google-cloud-dataflow-java:worker:classes > Task :runners:google-cloud-dataflow-java:worker:shadowJar > Task :sdks:python:test-suites:dataflow:py39:validatesRunnerBatchTests >>> RUNNING integration tests with pipeline options: >>> --runner=TestDataflowRunner --project=apache-beam-testing >>> --region=us-central1 >>> --staging_location=gs://temp-storage-for-end-to-end-tests/staging-it >>> --temp_location=gs://temp-storage-for-end-to-end-tests/temp-it >>> --output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output >>> --sdk_location=<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/ws/src/sdks/python/build/apache-beam.tar.gz> >>> --requirements_file=postcommit_requirements.txt --num_workers=1 >>> --sleep_secs=20 >>> --dataflow_worker_jar=<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/ws/src/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.41.0-SNAPSHOT.jar> >>> --experiments=use_runner_v2 --experiments=shuffle_mode=appliance >>> --experiments=beam_fn_api >>> pytest options: --capture=no --timeout=4500 --color=yes >>> --log-cli-level=INFO --numprocesses=8 >>> collect markers: -m=it_validatesrunner and not no_sickbay_batch [1m============================= test session starts ==============================[0m platform linux -- Python 3.9.10, pytest-4.6.11, py-1.11.0, pluggy-0.13.1 rootdir: <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/ws/src/sdks/python,> inifile: pytest.ini plugins: xdist-1.34.0, timeout-1.4.2, forked-1.4.0, requests-mock-1.9.3 timeout: 4500.0s timeout method: signal timeout func_only: False gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I [1m[gw2] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0] [0m[1m[gw1] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0] [0m[1m[gw4] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0] [0m[1m[gw0] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0] [0m[1m[gw6] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0] [0m[1m[gw5] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0] [0m[1m[gw3] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0] [0m[1m[gw7] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0] [0mgw0 [36] / gw1 [36] / gw2 [36] / gw3 [36] / gw4 [36] / gw5 [36] / gw6 [36] / gw7 [36] scheduling tests via LoadScheduling apache_beam/pipeline_test.py::DoFnTest::test_key_param apache_beam/runners/portability/fn_api_runner/translations_test.py::TranslationsTest::test_run_packable_combine_limit apache_beam/transforms/combinefn_lifecycle_test.py::CombineFnLifecycleTest::test_combine apache_beam/metrics/metric_test.py::MetricsTest::test_user_counter_using_pardo apache_beam/runners/portability/fn_api_runner/translations_test.py::TranslationsTest::test_run_packable_combine_globally apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiBasedStateBackedCoderTest::test_gbk_many_values apache_beam/pipeline_test.py::DoFnTest::test_element_param apache_beam/runners/portability/fn_api_runner/translations_test.py::TranslationsTest::test_run_packable_combine_per_key [gw0] [32mPASSED[0m apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiBasedStateBackedCoderTest::test_gbk_many_values apache_beam/transforms/combiners_test.py::TimestampCombinerTest::test_combiner_latest [gw0] [32mPASSED[0m apache_beam/transforms/combiners_test.py::TimestampCombinerTest::test_combiner_latest apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_pcollections [gw1] [32mPASSED[0m apache_beam/pipeline_test.py::DoFnTest::test_key_param apache_beam/transforms/combinefn_lifecycle_test.py::CombineFnLifecycleTest::test_non_liftable_combine [gw3] [32mPASSED[0m apache_beam/runners/portability/fn_api_runner/translations_test.py::TranslationsTest::test_run_packable_combine_per_key apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_multiple_pcollections_having_multiple_consumers [gw4] [32mPASSED[0m apache_beam/metrics/metric_test.py::MetricsTest::test_user_counter_using_pardo apache_beam/transforms/combiners_test.py::TimestampCombinerTest::test_combiner_earliest [gw4] [32mPASSED[0m apache_beam/transforms/combiners_test.py::TimestampCombinerTest::test_combiner_earliest apache_beam/transforms/ptransform_test.py::PTransformTest::test_par_do_with_multiple_outputs_and_using_return [gw2] [32mPASSED[0m apache_beam/pipeline_test.py::DoFnTest::test_element_param apache_beam/transforms/combinefn_lifecycle_test.py::CombineFnLifecycleTest::test_combining_value_state [gw2] [33mSKIPPED[0m apache_beam/transforms/combinefn_lifecycle_test.py::CombineFnLifecycleTest::test_combining_value_state apache_beam/transforms/ptransform_test.py::PTransformTest::test_read_metrics [gw7] [32mPASSED[0m apache_beam/transforms/combinefn_lifecycle_test.py::CombineFnLifecycleTest::test_combine apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_one_single_pcollection [gw5] [32mPASSED[0m apache_beam/runners/portability/fn_api_runner/translations_test.py::TranslationsTest::test_run_packable_combine_limit apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_a_flattened_pcollection [gw6] [32mPASSED[0m apache_beam/runners/portability/fn_api_runner/translations_test.py::TranslationsTest::test_run_packable_combine_globally apache_beam/transforms/dofn_lifecycle_test.py::DoFnLifecycleTest::test_dofn_lifecycle [gw0] [32mPASSED[0m apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_pcollections apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_same_pcollections [gw1] [32mPASSED[0m apache_beam/transforms/combinefn_lifecycle_test.py::CombineFnLifecycleTest::test_non_liftable_combine apache_beam/transforms/ptransform_test.py::PTransformTest::test_impulse [gw3] [32mPASSED[0m apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_multiple_pcollections_having_multiple_consumers apache_beam/transforms/ptransform_test.py::PTransformTest::test_multiple_empty_outputs [gw2] [32mPASSED[0m apache_beam/transforms/ptransform_test.py::PTransformTest::test_read_metrics apache_beam/transforms/ptransform_test.py::PTransformTest::test_undeclared_outputs [gw5] [32mPASSED[0m apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_a_flattened_pcollection apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_as_list_and_as_dict_side_inputs [gw4] [32mPASSED[0m apache_beam/transforms/ptransform_test.py::PTransformTest::test_par_do_with_multiple_outputs_and_using_return apache_beam/transforms/ptransform_test.py::PTransformTest::test_par_do_with_multiple_outputs_and_using_yield [gw6] [32mPASSED[0m apache_beam/transforms/dofn_lifecycle_test.py::DoFnLifecycleTest::test_dofn_lifecycle apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_as_list_twice [gw7] [32mPASSED[0m apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_one_single_pcollection apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_as_dict_twice [gw0] [32mPASSED[0m apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_same_pcollections apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_as_singleton_with_different_defaults [gw2] node down: Not properly terminated [gw2] [31mFAILED[0m apache_beam/transforms/ptransform_test.py::PTransformTest::test_undeclared_outputs replacing crashed worker gw2 [1m[gw8] linux Python 3.9.10 cwd: <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/ws/src/sdks/python> [0m[1m[gw8] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0] [0m[gw1] node down: Not properly terminated [gw1] [31mFAILED[0m apache_beam/transforms/ptransform_test.py::PTransformTest::test_impulse replacing crashed worker gw1 Terminated Remove shutdown hook failed java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) at java.lang.Runtime.removeShutdownHook(Runtime.java:239) at org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:38) at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:208) at org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:365) at org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108) at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84) at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) > Task :sdks:python:test-suites:dataflow:py39:validatesRunnerBatchTests FAILED > Task :sdks:python:test-suites:dataflow:py39:validatesRunnerStreamingTests > FAILED Remove shutdown hook failed java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) at java.lang.Runtime.removeShutdownHook(Runtime.java:239) at org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:38) at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:208) at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:369) at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:87) at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) FAILURE: Build completed with 2 failures. 1: Task failed with an exception. ----------- * Where: Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 232 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py39:validatesRunnerBatchTests'. > Process 'command 'sh'' finished with non-zero exit value 143 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py39:validatesRunnerStreamingTests'. > A problem occurred starting process 'command 'sh'' * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 27m 50s 93 actionable tasks: 64 executed, 27 from cache, 2 up-to-date Publishing build scan... The message received from the daemon indicates that the daemon has disappeared. Build request sent: Build{id=d6d6710e-7254-4ce3-b3bb-e097a6ada3a4, currentDir=<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/ws/src}> Attempting to read last messages from the daemon log... Daemon pid: 3266170 log file: /home/jenkins/.gradle/daemon/7.4/daemon-3266170.out.log ----- Last 20 lines from daemon log file - daemon-3266170.out.log ----- BUILD FAILED in 27m 50s 93 actionable tasks: 64 executed, 27 from cache, 2 up-to-date Publishing build scan... Remove shutdown hook failed java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) at java.lang.Runtime.removeShutdownHook(Runtime.java:239) at org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:38) at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:208) at org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:365) at org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108) at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84) at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ----- End of the daemon log ----- FAILURE: Build failed with an exception. * What went wrong: Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed) * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. * Get more help at https://help.gradle.org Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
