See
<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/1622/display/redirect?page=changes>
Changes:
[kirillkozlov] Fix MongoDb SQL Integration Tests
[kirillkozlov] Add MongoDbIT back to build file
[kirillkozlov] Update JavaDoc comment and remove pipeline options
------------------------------------------
[...truncated 1.33 MB...]
19/11/25 18:02:51 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer:
Hanged up for unknown endpoint.
19/11/25 18:02:51 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job
test_windowed_pardo_state_timers_1574704965.96_86f8fc35-b902-475f-b232-756f5ca4d93f
finished.
19/11/25 18:02:51 WARN
org.apache.beam.runners.spark.SparkPipelineResult$BatchMode: Collecting
monitoring infos is not implemented yet in Spark portable runner.
19/11/25 18:02:51 INFO
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService:
Manifest at
/tmp/sparktestn5Xgq_/job_75e9ec83-f03c-4c4a-8c2c-30f486aafb98/MANIFEST has 0
artifact locations
19/11/25 18:02:51 INFO
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService:
Removed dir /tmp/sparktestn5Xgq_/job_75e9ec83-f03c-4c4a-8c2c-30f486aafb98/
INFO:apache_beam.runners.portability.portable_runner:Job state changed to DONE
.INFO:apache_beam.runners.portability.fn_api_runner_transforms:====================
<function lift_combiners at 0x7f77bda41230> ====================
19/11/25 18:02:52 INFO org.apache.beam.runners.spark.SparkJobInvoker: Invoking
job test_windowing_1574704971.37_eb845bb5-00c1-4c5b-906b-006f0a3cc0e8
19/11/25 18:02:52 INFO
org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation: Starting job
invocation test_windowing_1574704971.37_eb845bb5-00c1-4c5b-906b-006f0a3cc0e8
INFO:apache_beam.runners.portability.portable_runner:Job state changed to
RUNNING
19/11/25 18:02:52 INFO org.apache.beam.runners.spark.SparkPipelineRunner:
PipelineOptions.filesToStage was not specified. Defaulting to files from the
classpath
19/11/25 18:02:52 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Will
stage 1 files. (Enable logging at DEBUG level to see which files will be
staged.)
19/11/25 18:02:52 INFO org.apache.beam.runners.spark.SparkPipelineRunner:
Running job test_windowing_1574704971.37_eb845bb5-00c1-4c5b-906b-006f0a3cc0e8
on Spark master local
19/11/25 18:02:52 WARN
org.apache.beam.runners.spark.translation.GroupNonMergingWindowsFunctions:
Either coder LengthPrefixCoder(ByteArrayCoder) or GlobalWindow$Coder is not
consistent with equals. That might cause issues on some runners.
19/11/25 18:02:52 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job
test_windowing_1574704971.37_eb845bb5-00c1-4c5b-906b-006f0a3cc0e8: Pipeline
translated successfully. Computing outputs
19/11/25 18:02:52 INFO
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService:
GetManifest for
/tmp/sparktestn5Xgq_/job_72db3842-f758-4c7a-a5e6-0639753c0db6/MANIFEST
19/11/25 18:02:52 INFO
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService:
Manifest at
/tmp/sparktestn5Xgq_/job_72db3842-f758-4c7a-a5e6-0639753c0db6/MANIFEST has 0
artifact locations
19/11/25 18:02:52 INFO
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService:
GetManifest for
/tmp/sparktestn5Xgq_/job_72db3842-f758-4c7a-a5e6-0639753c0db6/MANIFEST -> 0
artifacts
19/11/25 18:02:53 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging
client connected.
19/11/25 18:02:53 INFO sdk_worker_main.main: Logging handler created.
19/11/25 18:02:53 INFO sdk_worker_main.start: Status HTTP server running at
localhost:40841
19/11/25 18:02:53 INFO sdk_worker_main.main: semi_persistent_directory: /tmp
19/11/25 18:02:53 WARN sdk_worker_main._load_main_session: No session file
found: /tmp/staged/pickled_main_session. Functions defined in __main__
(interactive session) may fail.
19/11/25 18:02:53 WARN pipeline_options.get_all_options: Discarding unparseable
args:
[u'--app_name=test_windowing_1574704971.37_eb845bb5-00c1-4c5b-906b-006f0a3cc0e8',
u'--job_server_timeout=60', u'--pipeline_type_check',
u'--direct_runner_use_stacked_bundle', u'--spark_master=local',
u'--options_id=30', u'--enable_spark_metric_sinks']
19/11/25 18:02:53 INFO sdk_worker_main.main: Python sdk harness started with
pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'],
'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type':
u'PROCESS', 'sdk_location': u'container', 'job_name':
u'test_windowing_1574704971.37', 'environment_config': u'{"command":
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',>
'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint':
u'localhost:33413', 'job_port': u'0'}
19/11/25 18:02:53 INFO statecache.__init__: Creating state cache with size 0
19/11/25 18:02:53 INFO sdk_worker.__init__: Creating insecure control channel
for localhost:40113.
19/11/25 18:02:53 INFO
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam
Fn Control client connected with id 261-1
19/11/25 18:02:53 INFO sdk_worker.__init__: Control channel established.
19/11/25 18:02:53 INFO sdk_worker.__init__: Initializing SDKHarness with
unbounded number of workers.
19/11/25 18:02:53 INFO sdk_worker.create_state_handler: Creating insecure state
channel for localhost:34777.
19/11/25 18:02:53 INFO sdk_worker.create_state_handler: State channel
established.
19/11/25 18:02:53 INFO data_plane.create_data_channel: Creating client data
channel for localhost:36689
19/11/25 18:02:53 INFO
org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client
connected.
19/11/25 18:02:53 INFO
org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing
environment urn: "beam:env:process:v1"
payload:
"\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh";>
19/11/25 18:02:53 INFO sdk_worker.run: No more requests from control plane
19/11/25 18:02:53 INFO sdk_worker.run: SDK Harness waiting for in-flight
requests to complete
19/11/25 18:02:53 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer:
Hanged up for unknown endpoint.
19/11/25 18:02:53 INFO data_plane.close: Closing all cached grpc data channels.
19/11/25 18:02:53 INFO sdk_worker.close: Closing all cached gRPC state handlers.
19/11/25 18:02:53 INFO sdk_worker.run: Done consuming work.
19/11/25 18:02:53 INFO sdk_worker_main.main: Python sdk harness exiting.
19/11/25 18:02:53 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client
hanged up.
19/11/25 18:02:53 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer:
Hanged up for unknown endpoint.
19/11/25 18:02:53 INFO
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService:
GetManifest for
/tmp/sparktestn5Xgq_/job_72db3842-f758-4c7a-a5e6-0639753c0db6/MANIFEST
19/11/25 18:02:53 INFO
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService:
GetManifest for
/tmp/sparktestn5Xgq_/job_72db3842-f758-4c7a-a5e6-0639753c0db6/MANIFEST -> 0
artifacts
19/11/25 18:02:53 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging
client connected.
19/11/25 18:02:53 INFO sdk_worker_main.main: Logging handler created.
19/11/25 18:02:53 INFO sdk_worker_main.start: Status HTTP server running at
localhost:41363
19/11/25 18:02:53 INFO sdk_worker_main.main: semi_persistent_directory: /tmp
19/11/25 18:02:53 WARN sdk_worker_main._load_main_session: No session file
found: /tmp/staged/pickled_main_session. Functions defined in __main__
(interactive session) may fail.
19/11/25 18:02:53 WARN pipeline_options.get_all_options: Discarding unparseable
args:
[u'--app_name=test_windowing_1574704971.37_eb845bb5-00c1-4c5b-906b-006f0a3cc0e8',
u'--job_server_timeout=60', u'--pipeline_type_check',
u'--direct_runner_use_stacked_bundle', u'--spark_master=local',
u'--options_id=30', u'--enable_spark_metric_sinks']
19/11/25 18:02:53 INFO sdk_worker_main.main: Python sdk harness started with
pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'],
'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type':
u'PROCESS', 'sdk_location': u'container', 'job_name':
u'test_windowing_1574704971.37', 'environment_config': u'{"command":
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',>
'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint':
u'localhost:33413', 'job_port': u'0'}
19/11/25 18:02:53 INFO statecache.__init__: Creating state cache with size 0
19/11/25 18:02:53 INFO sdk_worker.__init__: Creating insecure control channel
for localhost:34201.
19/11/25 18:02:53 INFO sdk_worker.__init__: Control channel established.
19/11/25 18:02:53 INFO sdk_worker.__init__: Initializing SDKHarness with
unbounded number of workers.
19/11/25 18:02:53 INFO
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam
Fn Control client connected with id 262-1
19/11/25 18:02:53 INFO sdk_worker.create_state_handler: Creating insecure state
channel for localhost:44899.
19/11/25 18:02:53 INFO sdk_worker.create_state_handler: State channel
established.
19/11/25 18:02:53 INFO data_plane.create_data_channel: Creating client data
channel for localhost:42715
19/11/25 18:02:53 INFO
org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client
connected.
19/11/25 18:02:53 INFO
org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing
environment urn: "beam:env:process:v1"
payload:
"\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh";>
19/11/25 18:02:53 INFO sdk_worker.run: No more requests from control plane
19/11/25 18:02:53 INFO sdk_worker.run: SDK Harness waiting for in-flight
requests to complete
19/11/25 18:02:53 INFO data_plane.close: Closing all cached grpc data channels.
19/11/25 18:02:53 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer:
Hanged up for unknown endpoint.
19/11/25 18:02:53 INFO sdk_worker.close: Closing all cached gRPC state handlers.
19/11/25 18:02:53 INFO sdk_worker.run: Done consuming work.
19/11/25 18:02:53 INFO sdk_worker_main.main: Python sdk harness exiting.
19/11/25 18:02:53 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client
hanged up.
19/11/25 18:02:54 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer:
Hanged up for unknown endpoint.
19/11/25 18:02:54 INFO
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService:
GetManifest for
/tmp/sparktestn5Xgq_/job_72db3842-f758-4c7a-a5e6-0639753c0db6/MANIFEST
19/11/25 18:02:54 INFO
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService:
GetManifest for
/tmp/sparktestn5Xgq_/job_72db3842-f758-4c7a-a5e6-0639753c0db6/MANIFEST -> 0
artifacts
19/11/25 18:02:54 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging
client connected.
19/11/25 18:02:54 INFO sdk_worker_main.main: Logging handler created.
19/11/25 18:02:54 INFO sdk_worker_main.start: Status HTTP server running at
localhost:35859
19/11/25 18:02:54 INFO sdk_worker_main.main: semi_persistent_directory: /tmp
19/11/25 18:02:54 WARN sdk_worker_main._load_main_session: No session file
found: /tmp/staged/pickled_main_session. Functions defined in __main__
(interactive session) may fail.
19/11/25 18:02:54 WARN pipeline_options.get_all_options: Discarding unparseable
args:
[u'--app_name=test_windowing_1574704971.37_eb845bb5-00c1-4c5b-906b-006f0a3cc0e8',
u'--job_server_timeout=60', u'--pipeline_type_check',
u'--direct_runner_use_stacked_bundle', u'--spark_master=local',
u'--options_id=30', u'--enable_spark_metric_sinks']
19/11/25 18:02:54 INFO sdk_worker_main.main: Python sdk harness started with
pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'],
'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type':
u'PROCESS', 'sdk_location': u'container', 'job_name':
u'test_windowing_1574704971.37', 'environment_config': u'{"command":
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',>
'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint':
u'localhost:33413', 'job_port': u'0'}
19/11/25 18:02:54 INFO statecache.__init__: Creating state cache with size 0
19/11/25 18:02:54 INFO sdk_worker.__init__: Creating insecure control channel
for localhost:40877.
19/11/25 18:02:54 INFO sdk_worker.__init__: Control channel established.
19/11/25 18:02:54 INFO sdk_worker.__init__: Initializing SDKHarness with
unbounded number of workers.
19/11/25 18:02:54 INFO
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam
Fn Control client connected with id 263-1
19/11/25 18:02:54 INFO sdk_worker.create_state_handler: Creating insecure state
channel for localhost:41493.
19/11/25 18:02:54 INFO sdk_worker.create_state_handler: State channel
established.
19/11/25 18:02:54 INFO data_plane.create_data_channel: Creating client data
channel for localhost:40425
19/11/25 18:02:54 INFO
org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client
connected.
19/11/25 18:02:54 INFO
org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing
environment urn: "beam:env:process:v1"
payload:
"\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh";>
19/11/25 18:02:54 INFO sdk_worker.run: No more requests from control plane
19/11/25 18:02:54 INFO sdk_worker.run: SDK Harness waiting for in-flight
requests to complete
19/11/25 18:02:54 INFO data_plane.close: Closing all cached grpc data channels.
19/11/25 18:02:54 INFO sdk_worker.close: Closing all cached gRPC state handlers.
19/11/25 18:02:54 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer:
Hanged up for unknown endpoint.
19/11/25 18:02:54 INFO sdk_worker.run: Done consuming work.
19/11/25 18:02:54 INFO sdk_worker_main.main: Python sdk harness exiting.
19/11/25 18:02:54 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client
hanged up.
19/11/25 18:02:54 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer:
Hanged up for unknown endpoint.
19/11/25 18:02:54 INFO
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService:
GetManifest for
/tmp/sparktestn5Xgq_/job_72db3842-f758-4c7a-a5e6-0639753c0db6/MANIFEST
19/11/25 18:02:54 INFO
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService:
GetManifest for
/tmp/sparktestn5Xgq_/job_72db3842-f758-4c7a-a5e6-0639753c0db6/MANIFEST -> 0
artifacts
19/11/25 18:02:55 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging
client connected.
19/11/25 18:02:55 INFO sdk_worker_main.main: Logging handler created.
19/11/25 18:02:55 INFO sdk_worker_main.start: Status HTTP server running at
localhost:40325
19/11/25 18:02:55 INFO sdk_worker_main.main: semi_persistent_directory: /tmp
19/11/25 18:02:55 WARN sdk_worker_main._load_main_session: No session file
found: /tmp/staged/pickled_main_session. Functions defined in __main__
(interactive session) may fail.
19/11/25 18:02:55 WARN pipeline_options.get_all_options: Discarding unparseable
args:
[u'--app_name=test_windowing_1574704971.37_eb845bb5-00c1-4c5b-906b-006f0a3cc0e8',
u'--job_server_timeout=60', u'--pipeline_type_check',
u'--direct_runner_use_stacked_bundle', u'--spark_master=local',
u'--options_id=30', u'--enable_spark_metric_sinks']
19/11/25 18:02:55 INFO sdk_worker_main.main: Python sdk harness started with
pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'],
'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type':
u'PROCESS', 'sdk_location': u'container', 'job_name':
u'test_windowing_1574704971.37', 'environment_config': u'{"command":
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',>
'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint':
u'localhost:33413', 'job_port': u'0'}
19/11/25 18:02:55 INFO statecache.__init__: Creating state cache with size 0
19/11/25 18:02:55 INFO sdk_worker.__init__: Creating insecure control channel
for localhost:43215.
19/11/25 18:02:55 INFO sdk_worker.__init__: Control channel established.
19/11/25 18:02:55 INFO sdk_worker.__init__: Initializing SDKHarness with
unbounded number of workers.
19/11/25 18:02:55 INFO
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam
Fn Control client connected with id 264-1
19/11/25 18:02:55 INFO sdk_worker.create_state_handler: Creating insecure state
channel for localhost:36697.
19/11/25 18:02:55 INFO sdk_worker.create_state_handler: State channel
established.
19/11/25 18:02:55 INFO data_plane.create_data_channel: Creating client data
channel for localhost:46427
19/11/25 18:02:55 INFO
org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client
connected.
19/11/25 18:02:55 INFO
org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing
environment urn: "beam:env:process:v1"
payload:
"\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh";>
19/11/25 18:02:55 INFO sdk_worker.run: No more requests from control plane
19/11/25 18:02:55 INFO sdk_worker.run: SDK Harness waiting for in-flight
requests to complete
19/11/25 18:02:55 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer:
Hanged up for unknown endpoint.
19/11/25 18:02:55 INFO data_plane.close: Closing all cached grpc data channels.
19/11/25 18:02:55 INFO sdk_worker.close: Closing all cached gRPC state handlers.
19/11/25 18:02:55 INFO sdk_worker.run: Done consuming work.
19/11/25 18:02:55 INFO sdk_worker_main.main: Python sdk harness exiting.
19/11/25 18:02:55 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client
hanged up.
19/11/25 18:02:55 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer:
Hanged up for unknown endpoint.
19/11/25 18:02:55 INFO
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService:
GetManifest for
/tmp/sparktestn5Xgq_/job_72db3842-f758-4c7a-a5e6-0639753c0db6/MANIFEST
19/11/25 18:02:55 INFO
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService:
GetManifest for
/tmp/sparktestn5Xgq_/job_72db3842-f758-4c7a-a5e6-0639753c0db6/MANIFEST -> 0
artifacts
19/11/25 18:02:56 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging
client connected.
19/11/25 18:02:56 INFO sdk_worker_main.main: Logging handler created.
19/11/25 18:02:56 INFO sdk_worker_main.start: Status HTTP server running at
localhost:46675
19/11/25 18:02:56 INFO sdk_worker_main.main: semi_persistent_directory: /tmp
19/11/25 18:02:56 WARN sdk_worker_main._load_main_session: No session file
found: /tmp/staged/pickled_main_session. Functions defined in __main__
(interactive session) may fail.
19/11/25 18:02:56 WARN pipeline_options.get_all_options: Discarding unparseable
args:
[u'--app_name=test_windowing_1574704971.37_eb845bb5-00c1-4c5b-906b-006f0a3cc0e8',
u'--job_server_timeout=60', u'--pipeline_type_check',
u'--direct_runner_use_stacked_bundle', u'--spark_master=local',
u'--options_id=30', u'--enable_spark_metric_sinks']
19/11/25 18:02:56 INFO sdk_worker_main.main: Python sdk harness started with
pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'],
'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type':
u'PROCESS', 'sdk_location': u'container', 'job_name':
u'test_windowing_1574704971.37', 'environment_config': u'{"command":
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',>
'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint':
u'localhost:33413', 'job_port': u'0'}
19/11/25 18:02:56 INFO statecache.__init__: Creating state cache with size 0
19/11/25 18:02:56 INFO sdk_worker.__init__: Creating insecure control channel
for localhost:46777.
19/11/25 18:02:56 INFO sdk_worker.__init__: Control channel established.
19/11/25 18:02:56 INFO
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam
Fn Control client connected with id 265-1
19/11/25 18:02:56 INFO sdk_worker.__init__: Initializing SDKHarness with
unbounded number of workers.
19/11/25 18:02:56 INFO sdk_worker.create_state_handler: Creating insecure state
channel for localhost:44623.
19/11/25 18:02:56 INFO sdk_worker.create_state_handler: State channel
established.
19/11/25 18:02:56 INFO data_plane.create_data_channel: Creating client data
channel for localhost:41611
19/11/25 18:02:56 INFO
org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client
connected.
19/11/25 18:02:56 INFO
org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing
environment urn: "beam:env:process:v1"
payload:
"\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh";>
19/11/25 18:02:56 INFO sdk_worker.run: No more requests from control plane
19/11/25 18:02:56 INFO sdk_worker.run: SDK Harness waiting for in-flight
requests to complete
19/11/25 18:02:56 INFO data_plane.close: Closing all cached grpc data channels.
19/11/25 18:02:56 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer:
Hanged up for unknown endpoint.
19/11/25 18:02:56 INFO sdk_worker.close: Closing all cached gRPC state handlers.
19/11/25 18:02:56 INFO sdk_worker.run: Done consuming work.
19/11/25 18:02:56 INFO sdk_worker_main.main: Python sdk harness exiting.
19/11/25 18:02:56 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client
hanged up.
19/11/25 18:02:56 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer:
Hanged up for unknown endpoint.
19/11/25 18:02:56 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job
test_windowing_1574704971.37_eb845bb5-00c1-4c5b-906b-006f0a3cc0e8 finished.
19/11/25 18:02:56 WARN
org.apache.beam.runners.spark.SparkPipelineResult$BatchMode: Collecting
monitoring infos is not implemented yet in Spark portable runner.
19/11/25 18:02:56 INFO
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService:
Manifest at
/tmp/sparktestn5Xgq_/job_72db3842-f758-4c7a-a5e6-0639753c0db6/MANIFEST has 0
artifact locations
19/11/25 18:02:56 INFO
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService:
Removed dir /tmp/sparktestn5Xgq_/job_72db3842-f758-4c7a-a5e6-0639753c0db6/
INFO:apache_beam.runners.portability.portable_runner:Job state changed to DONE
.
======================================================================
ERROR: test_pardo_state_with_custom_key_coder (__main__.SparkRunnerTest)
Tests that state requests work correctly when the key coder is an
----------------------------------------------------------------------
Traceback (most recent call last):
File "apache_beam/runners/portability/portable_runner_test.py", line 231, in
test_pardo_state_with_custom_key_coder
equal_to(expected))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 428, in
wait_until_finish
for state_response in self._state_stream:
File
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py";,>
line 395, in next
return self._next()
File
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py";,>
line 552, in _next
_common.wait(self._state.condition.wait, _response_ready)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py";,>
line 140, in wait
_wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py";,>
line 105, in _wait_once
wait_fn(timeout=timeout)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
==================== Timed out after 60 seconds. ====================
File "apache_beam/runners/portability/portable_runner_test.py", line 75, in
handler
# Thread: <Thread(wait_until_finish_read, started daemon 140151745402624)>
# Thread: <Thread(Thread-117, started daemon 140151737009920)>
raise BaseException(msg)
BaseException: Timed out after 60 seconds.
======================================================================
ERROR: test_sdf_with_watermark_tracking (__main__.SparkRunnerTest)
----------------------------------------------------------------------
# Thread: <_MainThread(MainThread, started 140152740226816)>
Traceback (most recent call last):
File "apache_beam/runners/portability/fn_api_runner_test.py", line 499, in
test_sdf_with_watermark_tracking
assert_that(actual, equal_to(list(''.join(data))))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 438, in
wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline
test_sdf_with_watermark_tracking_1574704962.48_5d4d7c3e-4fdb-41cb-96b3-148f2977583e
failed in state FAILED: java.lang.UnsupportedOperationException: The
ActiveBundle does not have a registered bundle checkpoint handler.
----------------------------------------------------------------------
Ran 38 tests in 285.147s
FAILED (errors=2, skipped=9)
> Task :sdks:python:test-suites:portable:py2:sparkValidatesRunner FAILED
FAILURE: Build failed with an exception.
* Where:
Build file
'<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build.gradle'>
line: 196
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py2:sparkValidatesRunner'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 7m 9s
60 actionable tasks: 47 executed, 13 from cache
Publishing build scan...
https://gradle.com/s/vnndw2fs3wdhg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscr...@beam.apache.org
For additional commands, e-mail: builds-h...@beam.apache.org
