See
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/1210/display/redirect?page=changes>
Changes:
[Kenneth Knowles] Suppress keyfor warnings
[Kenneth Knowles] Suppress checker warnings that are confusing and difficult
[Kenneth Knowles] Add @Pure annotations to MongoDbIO autovalue fields
[Kenneth Knowles] Suppress checker in FnApiDoFnRunner due to crash
[Kenneth Knowles] Suppress checker framework in Dataflow
[Kenneth Knowles] Fix some nullness errors in Spark runner
[Kenneth Knowles] Upgrade checker framework to 3.27.0
[noreply] fix go lints (#24566)
[noreply] Samza runner support for non unique stateId across multiple ParDos
[noreply] Bump to Hadoop 3.3.4 for performance tests (#24550)
------------------------------------------
[...truncated 42.48 MB...]
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
create_file(input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
extra_opts = {'input': input, 'output': output}
> coders.run(test_pipeline.get_full_options_as_args(**extra_opts))
apache_beam/examples/cookbook/coders_it_test.py:72:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam/examples/cookbook/coders.py:92: in run
| 'write' >> WriteToText(known_args.output, coder=JsonCoder()))
apache_beam/pipeline.py:600: in __exit__
self.result = self.run()
apache_beam/pipeline.py:577: in run
return self.runner.run_pipeline(self, self._options)
apache_beam/runners/portability/flink_runner.py:45: in run_pipeline
return super().run_pipeline(pipeline, options)
apache_beam/runners/portability/portable_runner.py:441: in run_pipeline
job_service_handle.submit(proto_pipeline)
apache_beam/runners/portability/portable_runner.py:107: in submit
prepare_response = self.prepare(proto_pipeline)
apache_beam/runners/portability/portable_runner.py:201: in prepare
pipeline_options=self.get_pipeline_options()),
apache_beam/runners/portability/portable_runner.py:144: in get_pipeline_options
options_response = send_options_request()
apache_beam/runners/portability/portable_runner.py:141: in send_options_request
raise e
apache_beam/runners/portability/portable_runner.py:134: in send_options_request
timeout=self.timeout)
../../build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py:946:
in __call__
return _end_unary_response_blocking(state, call, False, None)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
state = <grpc._channel._RPCState object at 0x7fd54453ddd0>
call = <grpc._cython.cygrpc.SegregatedCall object at 0x7fd568470190>
with_call = False, deadline = None
def _end_unary_response_blocking(state, call, with_call, deadline):
if state.code is grpc.StatusCode.OK:
if with_call:
rendezvous = _MultiThreadedRendezvous(state, call, None,
deadline)
return state.response, rendezvous
else:
return state.response
else:
> raise _InactiveRpcError(state)
E grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that
terminated with:
E status = StatusCode.UNIMPLEMENTED
E details = "Method not found:
org.apache.beam.model.job_management.v1.JobService/DescribePipelineOptions"
E debug_error_string = "UNKNOWN:Error received from peer
ipv6:%5B::1%5D:45535 {created_time:"2022-12-07T22:19:29.339560004+00:00",
grpc_status:12, grpc_message:"Method not found:
org.apache.beam.model.job_management.v1.JobService/DescribePipelineOptions"}"
E >
../../build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py:849:
_InactiveRpcError
------------------------------ Captured log call -------------------------------
INFO apache_beam.runners.worker.worker_pool_main:worker_pool_main.py:113
Listening for workers at localhost:35091
INFO root:environments.py:376 Default Python SDK image for environment is
apache/beam_python3.7_sdk:2.45.0.dev
INFO
apache_beam.runners.portability.fn_api_runner.translations:translations.py:710
==================== <function pack_combiners at 0x7fd5ba00e950>
====================
INFO
apache_beam.runners.portability.fn_api_runner.translations:translations.py:710
==================== <function lift_combiners at 0x7fd5ba00e9e0>
====================
INFO
apache_beam.runners.portability.fn_api_runner.translations:translations.py:710
==================== <function sort_stages at 0x7fd5ba00d170>
====================
INFO apache_beam.utils.subprocess_server:subprocess_server.py:116 Starting
service with ['java' '-jar'
'<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/runners/flink/1.15/job-server/build/libs/beam-runners-flink-1.15-job-server-2.45.0-SNAPSHOT.jar'>
'--flink-master' '[auto]' '--artifacts-dir'
'/tmp/beam-temphnc7to49/artifacts90oj7411' '--job-port' '45535'
'--artifact-port' '0' '--expansion-port' '0']
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 Dec 07,
2022 10:19:25 PM software.amazon.awssdk.regions.internal.util.EC2MetadataUtils
getItems
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 WARNING:
Unable to retrieve the requested metadata.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 Dec 07,
2022 10:19:25 PM org.apache.beam.runners.jobsubmission.JobServerDriver
createArtifactStagingService
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 INFO:
ArtifactStagingService started on localhost:42835
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 Dec 07,
2022 10:19:25 PM org.apache.beam.runners.jobsubmission.JobServerDriver
createExpansionService
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 INFO:
Java ExpansionService started on localhost:39947
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 Dec 07,
2022 10:19:25 PM org.apache.beam.runners.jobsubmission.JobServerDriver run
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 WARNING:
Exception during job server creation
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126
java.io.IOException: Failed to bind to address 0.0.0.0/0.0.0.0:45535
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.vendor.grpc.v1p48p1.io.grpc.netty.NettyServer.start(NettyServer.java:328)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ServerImpl.start(ServerImpl.java:183)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ServerImpl.start(ServerImpl.java:92)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.sdk.fn.server.ServerFactory$InetSocketAddressServerFactory.createServer(ServerFactory.java:162)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.sdk.fn.server.ServerFactory$InetSocketAddressServerFactory.create(ServerFactory.java:145)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.sdk.fn.server.GrpcFnServer.create(GrpcFnServer.java:110)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.runners.jobsubmission.JobServerDriver.createJobServer(JobServerDriver.java:238)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.runners.jobsubmission.JobServerDriver.run(JobServerDriver.java:176)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 at
org.apache.beam.runners.flink.FlinkJobServerDriver.main(FlinkJobServerDriver.java:77)
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 Caused
by:
org.apache.beam.vendor.grpc.v1p48p1.io.netty.channel.unix.Errors$NativeIoException:
bind(..) failed: Address already in use
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 Dec 07,
2022 10:19:25 PM org.apache.beam.runners.jobsubmission.JobServerDriver stop
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 INFO:
ArtifactStagingServer stopped on localhost:42835
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 Dec 07,
2022 10:19:25 PM org.apache.beam.runners.jobsubmission.JobServerDriver stop
INFO apache_beam.utils.subprocess_server:subprocess_server.py:126 INFO:
Expansion stopped on localhost:39947
=============================== warnings summary ===============================
../../build/gradleenv/2022703441/lib/python3.7/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/hdfs/config.py>:15:
DeprecationWarning: the imp module is deprecated in favour of importlib; see
the module's documentation for alternative uses
from imp import load_source
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63:
PendingDeprecationWarning: Client.dataset is deprecated and will be removed in
a future version. Use a string like 'my_project.my_dataset' or a
cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1992:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1998:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1149:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1151:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2485:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2487:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2518:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
| _PassThroughThenCleanup(files_to_remove_pcoll))
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1988:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1142:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
self.project = self.project or p.options.view_as(GoogleCloudOptions).project
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100:
PendingDeprecationWarning: Client.dataset is deprecated and will be removed in
a future version. Use a string like 'my_project.my_dataset' or a
cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:47:
FutureWarning: Dropping of nuisance columns in DataFrame reductions (with
'numeric_only=None') is deprecated; in a future version this will raise
TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/dataframe/io.py>:664:
FutureWarning: WriteToFiles is experimental.
sink=lambda _: _WriteToPandasFileSink(
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/io/fileio.py>:591:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/pytest_postCommitExamples-flink-py37.xml>
-
=========================== short test summary info ============================
FAILED
apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
- grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated
with:
status = StatusCode.UNIMPLEMENTED
details = "Method not found:
org.apache.beam.model.job_management.v1.JobService/DescribePipelineOptions"
Exception in thread read_grpc_client_inputs:
debug_error_string = "UNKNOWN:Error received from peer
ipv6:%5B::1%5D:45535 {created_time:"2022-12-07T22:19:29.339560004+00:00",
grpc_status:12, grpc_message:"Method not found:
org.apache.beam.model.job_management.v1.JobService/DescribePipelineOptions"}"
>
= 1 failed, 17 passed, 6 skipped, 6764 deselected, 35 warnings in 812.61s
(0:13:32) =
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",>
line 671, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",>
line 654, in _read_inputs
for elements in elements_iterator:
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",>
line 426, in __next__
return self._next()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",>
line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that
terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "UNKNOWN:Error received from peer
ipv6:%5B::1%5D:36479 {grpc_message:"Multiplexer hanging up", grpc_status:1,
created_time:"2022-12-07T22:21:24.115364368+00:00"}"
>
Exception in thread run_worker_2-1:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py",>
line 254, in run
for work_request in self._control_stub.Control(get_responses()):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",>
line 426, in __next__
return self._next()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",>
line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that
terminated with:
status = StatusCode.CANCELLED
details = "Runner closed connection"
debug_error_string = "UNKNOWN:Error received from peer
ipv6:%5B::1%5D:34897 {created_time:"2022-12-07T22:21:39.634646614+00:00",
grpc_status:1, grpc_message:"Runner closed connection"}"
>
> Task :sdks:python:test-suites:portable:py37:flinkExamples FAILED
FAILURE: Build failed with an exception.
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/ws/src/sdks/python/test-suites/portable/common.gradle'>
line: 245
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py37:flinkExamples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
See
https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 16m 36s
133 actionable tasks: 87 executed, 44 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/3zgiuxz434xhu
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]