See
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/930/display/redirect>
Changes:
------------------------------------------
[...truncated 1.74 MB...]
message_thread = threading.Thread(
target=read_messages, name='wait_until_finish_read')
message_thread.daemon = True
message_thread.start()
if duration:
state_thread = threading.Thread(
target=functools.partial(self._observe_state, message_thread),
name='wait_until_finish_state_observer')
state_thread.daemon = True
state_thread.start()
start_time = time.time()
duration_secs = duration / 1000
while (time.time() - start_time < duration_secs and
state_thread.is_alive()):
time.sleep(1)
else:
self._observe_state(message_thread)
if self._runtime_exception:
> raise self._runtime_exception
E RuntimeError: Pipeline
BeamApp-jenkins-0928140720-f47091c_5816b26e-f105-4252-8a02-b97187bdaf70 failed
in state FAILED: java.lang.IllegalArgumentException: PCollectionNodes
[PCollectionNode{id=ref_PCollection_PCollection_60, PCollection=unique_name:
"64SaveToBQ/BigQueryBatchFileLoads/TriggerLoadJobsWithoutTempTables.None"
E coder_id: "ref_Coder_FastPrimitivesCoder_3"
E is_bounded: BOUNDED
E windowing_strategy_id: "ref_Windowing_Windowing_1"
E }] were consumed but never produced
apache_beam/runners/portability/portable_runner.py:607: RuntimeError
------------------------------ Captured log call -------------------------------
INFO apache_beam.runners.worker.worker_pool_main:worker_pool_main.py:93
Listening for workers at localhost:42303
INFO root:environments.py:376 Default Python SDK image for environment is
apache/beam_python3.7_sdk:2.43.0.dev
INFO
apache_beam.runners.portability.fn_api_runner.translations:translations.py:714
==================== <function pack_combiners at 0x7ffb005adf80>
====================
INFO
apache_beam.runners.portability.fn_api_runner.translations:translations.py:714
==================== <function lift_combiners at 0x7ffb005b1050>
====================
INFO
apache_beam.runners.portability.fn_api_runner.translations:translations.py:714
==================== <function sort_stages at 0x7ffb005b17a0>
====================
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 22/09/28
14:07:20 INFO
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Staging
artifacts for job_0493bfce-d21e-4b95-955a-a43ee46a687e.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 22/09/28
14:07:20 INFO
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Resolving
artifacts for
job_0493bfce-d21e-4b95-955a-a43ee46a687e.ref_Environment_default_environment_1.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 22/09/28
14:07:20 INFO
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Getting 0
artifacts for job_0493bfce-d21e-4b95-955a-a43ee46a687e.null.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 22/09/28
14:07:20 INFO
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Artifacts
fully staged for job_0493bfce-d21e-4b95-955a-a43ee46a687e.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 22/09/28
14:07:20 INFO org.apache.beam.runners.spark.SparkJobInvoker: Invoking job
BeamApp-jenkins-0928140720-f47091c_5816b26e-f105-4252-8a02-b97187bdaf70
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 22/09/28
14:07:20 INFO org.apache.beam.runners.jobsubmission.JobInvocation: Starting job
invocation
BeamApp-jenkins-0928140720-f47091c_5816b26e-f105-4252-8a02-b97187bdaf70
INFO apache_beam.runners.portability.portable_runner:portable_runner.py:458
Environment "LOOPBACK" has started a component necessary for the execution. Be
sure to run the pipeline using
with Pipeline() as p:
p.apply(..)
This ensures that the pipeline finishes before this program exits.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 22/09/28
14:07:20 ERROR org.apache.beam.runners.jobsubmission.JobInvocation: Error
during job invocation
BeamApp-jenkins-0928140720-f47091c_5816b26e-f105-4252-8a02-b97187bdaf70.
INFO apache_beam.runners.portability.portable_runner:portable_runner.py:583
Job state changed to STOPPED
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126
java.lang.IllegalArgumentException: PCollectionNodes
[PCollectionNode{id=ref_PCollection_PCollection_60, PCollection=unique_name:
"64SaveToBQ/BigQueryBatchFileLoads/TriggerLoadJobsWithoutTempTables.None"
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 coder_id:
"ref_Coder_FastPrimitivesCoder_3"
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126
is_bounded: BOUNDED
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126
windowing_strategy_id: "ref_Windowing_Windowing_1"
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 }] were
consumed but never produced
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:440)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.runners.core.construction.graph.QueryablePipeline.buildNetwork(QueryablePipeline.java:234)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.runners.core.construction.graph.QueryablePipeline.<init>(QueryablePipeline.java:127)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.runners.core.construction.graph.QueryablePipeline.forTransforms(QueryablePipeline.java:107)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.runners.core.construction.graph.FusedPipeline.toPipeline(FusedPipeline.java:81)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.runners.spark.SparkPipelineRunner.run(SparkPipelineRunner.java:109)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.runners.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:86)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
java.lang.Thread.run(Thread.java:750)
INFO apache_beam.runners.portability.portable_runner:portable_runner.py:583
Job state changed to STARTING
INFO apache_beam.runners.portability.portable_runner:portable_runner.py:583
Job state changed to RUNNING
ERROR root:portable_runner.py:577 java.lang.IllegalArgumentException:
PCollectionNodes [PCollectionNode{id=ref_PCollection_PCollection_60,
PCollection=unique_name:
"64SaveToBQ/BigQueryBatchFileLoads/TriggerLoadJobsWithoutTempTables.None"
coder_id: "ref_Coder_FastPrimitivesCoder_3"
is_bounded: BOUNDED
windowing_strategy_id: "ref_Windowing_Windowing_1"
}] were consumed but never produced
INFO apache_beam.runners.portability.portable_runner:portable_runner.py:583
Job state changed to FAILED
INFO apache_beam.io.gcp.tests.utils:utils.py:98 Clean up a BigQuery table
with project: apache-beam-testing, dataset: FiltersTestIT, table:
cold_days_1664374038847.
=============================== warnings summary ===============================
../../build/gradleenv/2022703441/lib/python3.7/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/hdfs/config.py>:15:
DeprecationWarning: the imp module is deprecated in favour of importlib; see
the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2022703441/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/google/api_core/grpc_helpers.py>:42:
DeprecationWarning: Support for grpcio-gcp is deprecated. This feature will be
removed from `google-api-core` after January 1, 2024. If you need
to
continue to use this feature, please pin to a specific version of
`google-api-core`.
DeprecationWarning,
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63:
PendingDeprecationWarning: Client.dataset is deprecated and will be removed in
a future version. Use a string like 'my_project.my_dataset' or a
cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2285:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2291:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2775:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2777:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2808:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
| _PassThroughThenCleanup(files_to_remove_pcoll))
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2281:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1126:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
self.project = self.project or p.options.view_as(GoogleCloudOptions).project
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100:
PendingDeprecationWarning: Client.dataset is deprecated and will be removed in
a future version. Use a string like 'my_project.my_dataset' or a
cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",>
line 671, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",>
line 654, in _read_inputs
for elements in elements_iterator:
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",>
line 426, in __next__
return self._next()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",>
line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that
terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "UNKNOWN:Error received from peer
ipv6:%5B::1%5D:40773 {created_time:"2022-09-28T14:10:38.746452332+00:00",
grpc_status:1, grpc_message:"Multiplexer hanging up"}"
>
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:47:
FutureWarning: Dropping of nuisance columns in DataFrame reductions (with
'numeric_only=None') is deprecated; in a future version this will raise
TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/dataframe/io.py>:664:
FutureWarning: WriteToFiles is experimental.
sink=lambda _: _WriteToPandasFileSink(
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/fileio.py>:590:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/pytest_postCommitExamples-spark-py37.xml>
-
=========================== short test summary info ============================
FAILED
apache_beam/examples/wordcount_debugging_test.py::WordCountDebuggingTest::test_basics
FAILED
apache_beam/examples/wordcount_minimal_test.py::WordCountMinimalTest::test_basics
FAILED apache_beam/examples/wordcount_test.py::WordCountTest::test_basics - F...
FAILED
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
FAILED
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
ERROR
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
ERROR
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
= 6 failed, 16 passed, 4 skipped, 6646 deselected, 41 warnings, 2 errors in
2541.51s (0:42:21) =
> Task :sdks:python:test-suites:portable:py37:sparkExamples FAILED
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/test-suites/portable/common.gradle'>
line: 271
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py39:sparkExamples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/test-suites/portable/common.gradle'>
line: 271
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py37:sparkExamples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
See
https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 44m 31s
80 actionable tasks: 51 executed, 27 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/xa2kvpurbuivu
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]