See
<https://ci-beam.apache.org/job/beam_PostCommit_Python36/3530/display/redirect?page=changes>
Changes:
[David Morávek] Simplify LateDataDropping runner.
------------------------------------------
[...truncated 28.98 MB...]
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:50.806Z:
JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into
GroupAndSum/GroupByKey/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:50.834Z:
JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into
GroupAndSum/Combine
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:50.869Z:
JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:50.919Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:50.988Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/WriteBundles/WriteBundles into
Write/Write/WriteImpl/WindowInto(WindowIntoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.042Z:
JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into
Write/Write/WriteImpl/WriteBundles/WriteBundles
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.086Z:
JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify
into Write/Write/WriteImpl/Pair
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.129Z:
JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write
into Write/Write/WriteImpl/GroupByKey/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.182Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/GroupByKey/GroupByWindow into
Write/Write/WriteImpl/GroupByKey/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.224Z:
JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into
Write/Write/WriteImpl/GroupByKey/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.263Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/PreFinalize/_DataflowIterableAsMultimapSideInput(MapToVoidKey1.out.0)/CreateIsmShardKeyAndSortKey
into Write/Write/WriteImpl/PreFinalize/MapToVoidKey1
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.337Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/PreFinalize/_DataflowIterableAsMultimapSideInput(MapToVoidKey1.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write
into
Write/Write/WriteImpl/PreFinalize/_DataflowIterableAsMultimapSideInput(MapToVoidKey1.out.0)/CreateIsmShardKeyAndSortKey
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.391Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/PreFinalize/_DataflowIterableAsMultimapSideInput(MapToVoidKey1.out.0)/ToIsmRecordForMultimap
into
Write/Write/WriteImpl/PreFinalize/_DataflowIterableAsMultimapSideInput(MapToVoidKey1.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.427Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/CreateIsmShardKeyAndSortKey
into Write/Write/WriteImpl/FinalizeWrite/MapToVoidKey0
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.468Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write
into
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/CreateIsmShardKeyAndSortKey
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.510Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/ToIsmRecordForMultimap
into
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.561Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey1.out.0)/CreateIsmShardKeyAndSortKey
into Write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.626Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey1.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write
into
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey1.out.0)/CreateIsmShardKeyAndSortKey
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.672Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey1.out.0)/ToIsmRecordForMultimap
into
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey1.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.742Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey2.out.0)/CreateIsmShardKeyAndSortKey
into Write/Write/WriteImpl/FinalizeWrite/MapToVoidKey2
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.791Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey2.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write
into
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey2.out.0)/CreateIsmShardKeyAndSortKey
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.834Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey2.out.0)/ToIsmRecordForMultimap
into
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey2.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.880Z:
JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.928Z:
JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:51.966Z:
JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:52.016Z:
JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:52.528Z:
JOB_MESSAGE_DEBUG: Executing wait step start129
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:52.661Z:
JOB_MESSAGE_BASIC: Executing operation
Write/Write/WriteImpl/WriteBundles/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:52.708Z:
JOB_MESSAGE_BASIC: Executing operation
Write/Write/WriteImpl/PreFinalize/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:52.723Z:
JOB_MESSAGE_DEBUG: Starting worker pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:52.747Z:
JOB_MESSAGE_BASIC: Executing operation
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:52.771Z:
JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-c...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:52.803Z:
JOB_MESSAGE_BASIC: Executing operation
Read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:53.443Z:
JOB_MESSAGE_BASIC: Finished operation
Write/Write/WriteImpl/WriteBundles/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:53.443Z:
JOB_MESSAGE_BASIC: Finished operation
Write/Write/WriteImpl/PreFinalize/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:53.465Z:
JOB_MESSAGE_BASIC: Finished operation
Read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:53.476Z:
JOB_MESSAGE_BASIC: Finished operation
Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:53.584Z:
JOB_MESSAGE_DEBUG: Value
"Write/Write/WriteImpl/WriteBundles/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Session"
materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:53.632Z:
JOB_MESSAGE_DEBUG: Value
"Write/Write/WriteImpl/PreFinalize/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Session"
materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:53.680Z:
JOB_MESSAGE_DEBUG: Value
"Read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:53.736Z:
JOB_MESSAGE_DEBUG: Value
"Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Session"
materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:53.808Z:
JOB_MESSAGE_BASIC: Executing operation
Read/Read/Impulse+Read/Read/Split+Read/Read/Reshuffle/AddRandomKeys+Read/Read/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:53.857Z:
JOB_MESSAGE_BASIC: Executing operation
Write/Write/WriteImpl/DoOnce/Impulse+Write/Write/WriteImpl/DoOnce/FlatMap(<lambda
at
core.py:2957>)+Write/Write/WriteImpl/DoOnce/Map(decode)+Write/Write/WriteImpl/InitializeWrite+Write/Write/WriteImpl/WriteBundles/MapToVoidKey0+Write/Write/WriteImpl/PreFinalize/MapToVoidKey0+Write/Write/WriteImpl/FinalizeWrite/MapToVoidKey0+Write/Write/WriteImpl/WriteBundles/MapToVoidKey0+Write/Write/WriteImpl/PreFinalize/MapToVoidKey0+Write/Write/WriteImpl/FinalizeWrite/MapToVoidKey0+Write/Write/WriteImpl/WriteBundles/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/CreateIsmShardKeyAndSortKey+Write/Write/WriteImpl/WriteBundles/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write+Write/Write/WriteImpl/PreFinalize/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/CreateIsmShardKeyAndSortKey+Write/Write/WriteImpl/PreFinalize/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write+Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/CreateIsmShardKeyAndSortKey+Write/Write/WriteImpl/FinalizeWrite/_DataflowIterableAsMultimapSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-02-12T12:26:53.996Z:
JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 so that
the pipeline can catch up with its backlog and keep up with its input rate.
> Task :sdks:python:test-suites:portable:py36:postCommitPy36IT
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.jobsubmission.JobServerDriver createExpansionService'
INFO:apache_beam.utils.subprocess_server:b'INFO: Java ExpansionService started
on localhost:43733'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.jobsubmission.JobServerDriver createJobServer'
INFO:apache_beam.utils.subprocess_server:b'INFO: JobService started on
localhost:38251'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.jobsubmission.JobServerDriver run'
INFO:apache_beam.utils.subprocess_server:b'INFO: Job server now running,
terminate with Ctrl+C'
DEBUG:root:Waiting for grpc channel to be ready at localhost:38251.
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'experiments' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'job_name'
was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'runner'
was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'temp_location' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'dataflow_kms_key' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'enable_streaming_engine' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'project'
was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'worker_region' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'worker_zone' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'zone' was
already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'pubsub_root_url' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'streaming'
was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'environment_cache_millis' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'job_endpoint' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'output_executable_path' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'sdk_worker_parallelism' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'files_to_stage' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'flink_master' was already added
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2 onNext'
INFO:apache_beam.utils.subprocess_server:b'INFO: Staging artifacts for
job_37aa650d-da98-496f-a544-697604a26f57.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2
resolveNextEnvironment'
INFO:apache_beam.utils.subprocess_server:b'INFO: Resolving artifacts for
job_37aa650d-da98-496f-a544-697604a26f57.external_10beam:env:docker:v1.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2 onNext'
INFO:apache_beam.utils.subprocess_server:b'INFO: Getting 7 artifacts for
job_37aa650d-da98-496f-a544-697604a26f57.ref_Environment_default_environment_1.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:32 PM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2
resolveNextEnvironment'
INFO:apache_beam.utils.subprocess_server:b'INFO: Resolving artifacts for
job_37aa650d-da98-496f-a544-697604a26f57.ref_Environment_default_environment_1.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:32 PM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2 onNext'
INFO:apache_beam.utils.subprocess_server:b'INFO: Getting 0 artifacts for
job_37aa650d-da98-496f-a544-697604a26f57.null.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:33 PM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2
finishStaging'
INFO:apache_beam.utils.subprocess_server:b'INFO: Artifacts fully staged for
job_37aa650d-da98-496f-a544-697604a26f57.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:34 PM
org.apache.beam.runners.flink.FlinkJobInvoker invokeWithExecutor'
INFO:apache_beam.utils.subprocess_server:b'INFO: Invoking job
BeamApp-jenkins-0212122633-2c564c56_7523f628-beb1-403a-8586-3377fa73d4e7 with
pipeline runner org.apache.beam.runners.flink.FlinkPipelineRunner@7413c14b'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:34 PM
org.apache.beam.runners.jobsubmission.JobInvocation start'
INFO:apache_beam.utils.subprocess_server:b'INFO: Starting job invocation
BeamApp-jenkins-0212122633-2c564c56_7523f628-beb1-403a-8586-3377fa73d4e7'
INFO:apache_beam.runners.portability.portable_runner:Environment "LOOPBACK" has
started a component necessary for the execution. Be sure to run the pipeline
using
with Pipeline() as p:
p.apply(..)
This ensures that the pipeline finishes before this program exits.
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:34 PM
org.apache.beam.runners.flink.FlinkPipelineRunner runPipelineWithTranslator'
INFO:apache_beam.utils.subprocess_server:b'INFO: Translating pipeline to Flink
program.'
INFO:apache_beam.runners.portability.portable_runner:Job state changed to
STOPPED
INFO:apache_beam.runners.portability.portable_runner:Job state changed to
STARTING
INFO:apache_beam.runners.portability.portable_runner:Job state changed to
RUNNING
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:34 PM
org.apache.beam.runners.flink.FlinkExecutionEnvironments
createBatchExecutionEnvironment'
INFO:apache_beam.utils.subprocess_server:b'INFO: Creating a Batch Execution
Environment.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:35 PM
org.apache.flink.api.java.utils.PlanGenerator logTypeRegistrationDetails'
INFO:apache_beam.utils.subprocess_server:b'INFO: The job has 0 registered types
and 0 default Kryo serializers'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:36 PM
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils
setConfigOptionToDefaultIfNotSet'
INFO:apache_beam.utils.subprocess_server:b'INFO: The configuration option
taskmanager.cpu.cores required for local execution is not set, setting it to
the maximal possible value.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:36 PM
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils
setConfigOptionToDefaultIfNotSet'
INFO:apache_beam.utils.subprocess_server:b'INFO: The configuration option
taskmanager.memory.task.heap.size required for local execution is not set,
setting it to the maximal possible value.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:36 PM
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils
setConfigOptionToDefaultIfNotSet'
INFO:apache_beam.utils.subprocess_server:b'INFO: The configuration option
taskmanager.memory.task.off-heap.size required for local execution is not set,
setting it to the maximal possible value.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:36 PM
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils
setConfigOptionToDefaultIfNotSet'
INFO:apache_beam.utils.subprocess_server:b'INFO: The configuration option
taskmanager.memory.network.min required for local execution is not set, setting
it to its default value 64 mb.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:36 PM
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils
setConfigOptionToDefaultIfNotSet'
INFO:apache_beam.utils.subprocess_server:b'INFO: The configuration option
taskmanager.memory.network.max required for local execution is not set, setting
it to its default value 64 mb.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:36 PM
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils
setConfigOptionToDefaultIfNotSet'
INFO:apache_beam.utils.subprocess_server:b'INFO: The configuration option
taskmanager.memory.managed.size required for local execution is not set,
setting it to its default value 128 mb.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:36 PM
org.apache.flink.runtime.minicluster.MiniCluster start'
INFO:apache_beam.utils.subprocess_server:b'INFO: Starting Flink Mini Cluster'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:36 PM
org.apache.flink.runtime.minicluster.MiniCluster start'
INFO:apache_beam.utils.subprocess_server:b'INFO: Starting Metrics Registry'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:36 PM
org.apache.flink.runtime.metrics.MetricRegistryImpl <init>'
INFO:apache_beam.utils.subprocess_server:b'INFO: No metrics reporter
configured, no metrics will be exposed/reported.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:36 PM
org.apache.flink.runtime.minicluster.MiniCluster start'
INFO:apache_beam.utils.subprocess_server:b'INFO: Starting RPC Service(s)'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:36 PM
org.apache.flink.runtime.clusterframework.BootstrapTools startLocalActorSystem'
INFO:apache_beam.utils.subprocess_server:b'INFO: Trying to start local actor
system'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
akka.event.slf4j.Slf4jLogger$$anonfun$receive$1 applyOrElse'
INFO:apache_beam.utils.subprocess_server:b'INFO: Slf4jLogger started'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.clusterframework.BootstrapTools startActorSystem'
INFO:apache_beam.utils.subprocess_server:b'INFO: Actor system started at
akka://flink'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.clusterframework.BootstrapTools startLocalActorSystem'
INFO:apache_beam.utils.subprocess_server:b'INFO: Trying to start local actor
system'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
akka.event.slf4j.Slf4jLogger$$anonfun$receive$1 applyOrElse'
INFO:apache_beam.utils.subprocess_server:b'INFO: Slf4jLogger started'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.clusterframework.BootstrapTools startActorSystem'
INFO:apache_beam.utils.subprocess_server:b'INFO: Actor system started at
akka://flink-metrics'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.rpc.akka.AkkaRpcService startServer'
INFO:apache_beam.utils.subprocess_server:b'INFO: Starting RPC endpoint for
org.apache.flink.runtime.metrics.dump.MetricQueryService at
akka://flink-metrics/user/rpc/MetricQueryService .'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.minicluster.MiniCluster createHighAvailabilityServices'
INFO:apache_beam.utils.subprocess_server:b'INFO: Starting high-availability
services'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.blob.BlobServer <init>'
INFO:apache_beam.utils.subprocess_server:b'INFO: Created BLOB server storage
directory /tmp/blobStore-0468a36f-2539-48b8-909a-1b36e58b95d2'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.blob.BlobServer <init>'
INFO:apache_beam.utils.subprocess_server:b'INFO: Started BLOB server at
0.0.0.0:33615 - max concurrent requests: 50 - max backlog: 1000'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.blob.AbstractBlobCache <init>'
INFO:apache_beam.utils.subprocess_server:b'INFO: Created BLOB cache storage
directory /tmp/blobStore-48a5405c-9469-4a9e-a9a6-98600b057097'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.blob.AbstractBlobCache <init>'
INFO:apache_beam.utils.subprocess_server:b'INFO: Created BLOB cache storage
directory /tmp/blobStore-0b8f3d2e-a5a8-4e98-8dc4-a894b1972776'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.minicluster.MiniCluster startTaskManagers'
INFO:apache_beam.utils.subprocess_server:b'INFO: Starting 1 TaskManger(s)'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.taskexecutor.TaskManagerRunner startTaskManager'
INFO:apache_beam.utils.subprocess_server:b'INFO: Starting TaskManager with
ResourceID: 9007f7ff-4a28-4938-a46c-c110d81c6779'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.taskexecutor.TaskManagerServices checkTempDirs'
INFO:apache_beam.utils.subprocess_server:b"INFO: Temporary file directory
'/tmp': total 484 GB, usable 302 GB (62.40% usable)"
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.io.disk.FileChannelManagerImpl createFiles'
INFO:apache_beam.utils.subprocess_server:b'INFO: FileChannelManager uses
directory /tmp/flink-io-6c118404-4329-4968-8864-a7607ab3ca33 for spill files.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.io.disk.FileChannelManagerImpl createFiles'
INFO:apache_beam.utils.subprocess_server:b'INFO: FileChannelManager uses
directory /tmp/flink-netty-shuffle-140c22e0-af59-4958-83c2-c4498d81760e for
spill files.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.io.network.buffer.NetworkBufferPool <init>'
INFO:apache_beam.utils.subprocess_server:b'INFO: Allocated 64 MB for network
buffer pool (number of memory segments: 2048, bytes per segment: 32768).'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.io.network.NettyShuffleEnvironment start'
INFO:apache_beam.utils.subprocess_server:b'INFO: Starting the network
environment and its components.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.taskexecutor.KvStateService start'
INFO:apache_beam.utils.subprocess_server:b'INFO: Starting the kvState service
and its components.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.rpc.akka.AkkaRpcService startServer'
INFO:apache_beam.utils.subprocess_server:b'INFO: Starting RPC endpoint for
org.apache.flink.runtime.taskexecutor.TaskExecutor at
akka://flink/user/rpc/taskmanager_0 .'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService start'
INFO:apache_beam.utils.subprocess_server:b'INFO: Start job leader service.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.filecache.FileCache <init>'
INFO:apache_beam.utils.subprocess_server:b'INFO: User file cache uses directory
/tmp/flink-dist-cache-4ceefd2f-c493-4516-bc92-4fd7d010237f'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:37 PM
org.apache.flink.runtime.rest.RestServerEndpoint start'
INFO:apache_beam.utils.subprocess_server:b'INFO: Starting rest endpoint.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:38 PM
org.apache.flink.runtime.webmonitor.WebMonitorUtils$LogFileLocation find'
INFO:apache_beam.utils.subprocess_server:b"WARNING: Log file environment
variable 'log.file' is not set."
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:38 PM
org.apache.flink.runtime.webmonitor.WebMonitorUtils$LogFileLocation find'
INFO:apache_beam.utils.subprocess_server:b"WARNING: JobManager log files are
unavailable in the web dashboard. Log file location not found in environment
variable 'log.file' or configuration key 'web.log.path'."
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:38 PM
org.apache.flink.runtime.rest.RestServerEndpoint start'
INFO:apache_beam.utils.subprocess_server:b'INFO: Rest endpoint listening at
localhost:41323'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:38 PM
org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService
updateLeader'
INFO:apache_beam.utils.subprocess_server:b'INFO: Proposing leadership to
contender http://localhost:41323'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:38 PM
org.apache.flink.runtime.webmonitor.WebMonitorEndpoint startInternal'
INFO:apache_beam.utils.subprocess_server:b'INFO: Web frontend listening at
http://localhost:41323.'
> Task :sdks:python:test-suites:portable:py36:postCommitPy36IT
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.jobsubmission.JobServerDriver createExpansionService'
INFO:apache_beam.utils.subprocess_server:b'INFO: Java ExpansionService started
on localhost:43733'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.jobsubmission.JobServerDriver createJobServer'
INFO:apache_beam.utils.subprocess_server:b'INFO: JobService started on
localhost:38251'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.jobsubmission.JobServerDriver run'
INFO:apache_beam.utils.subprocess_server:b'INFO: Job server now running,
terminate with Ctrl+C'
DEBUG:root:Waiting for grpc channel to be ready at localhost:38251.
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'experiments' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'job_name'
was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'runner'
was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'temp_location' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'dataflow_kms_key' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'enable_streaming_engine' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'project'
was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'worker_region' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'worker_zone' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'zone' was
already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'pubsub_root_url' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'streaming'
was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'environment_cache_millis' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'job_endpoint' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'output_executable_path' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'sdk_worker_parallelism' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'files_to_stage' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'flink_master' was already added
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2 onNext'
> Task :sdks:python:test-suites:portable:py36:postCommitPy36IT
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.jobsubmission.JobServerDriver createExpansionService'
INFO:apache_beam.utils.subprocess_server:b'INFO: Java ExpansionService started
on localhost:43733'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.jobsubmission.JobServerDriver createJobServer'
INFO:apache_beam.utils.subprocess_server:b'INFO: JobService started on
localhost:38251'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.jobsubmission.JobServerDriver run'
INFO:apache_beam.utils.subprocess_server:b'INFO: Job server now running,
terminate with Ctrl+C'
DEBUG:root:Waiting for grpc channel to be ready at localhost:38251.
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'experiments' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'job_name'
was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'runner'
was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'temp_location' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'dataflow_kms_key' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'enable_streaming_engine' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'project'
was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'worker_region' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'worker_zone' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'zone' was
already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'pubsub_root_url' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option 'streaming'
was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'environment_cache_millis' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'job_endpoint' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'output_executable_path' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'sdk_worker_parallelism' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'files_to_stage' was already added
DEBUG:apache_beam.runners.portability.portable_runner:Runner option
'flink_master' was already added
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2 onNext'
INFO:apache_beam.utils.subprocess_server:b'INFO: Staging artifacts for
job_37aa650d-da98-496f-a544-697604a26f57.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2
resolveNextEnvironment'
INFO:apache_beam.utils.subprocess_server:b'INFO: Resolving artifacts for
job_37aa650d-da98-496f-a544-697604a26f57.external_10beam:env:docker:v1.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:31 PM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2 onNext'
INFO:apache_beam.utils.subprocess_server:b'INFO: Getting 7 artifacts for
job_37aa650d-da98-496f-a544-697604a26f57.ref_Environment_default_environment_1.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:32 PM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2
resolveNextEnvironment'
INFO:apache_beam.utils.subprocess_server:b'INFO: Resolving artifacts for
job_37aa650d-da98-496f-a544-697604a26f57.ref_Environment_default_environment_1.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:32 PM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2 onNext'
INFO:apache_beam.utils.subprocess_server:b'INFO: Getting 0 artifacts for
job_37aa650d-da98-496f-a544-697604a26f57.null.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:33 PM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2
finishStaging'
INFO:apache_beam.utils.subprocess_server:b'INFO: Artifacts fully staged for
job_37aa650d-da98-496f-a544-697604a26f57.'
INFO:apache_beam.utils.subprocess_server:b'Feb 12, 2021 12:26:34 PM
org.apache.beam.runners.flink.FlinkJobInvoker invokeWithExecutor'
INFO:apache_beam.utils.subprocess_server:b'INFO: Invoking job
BeamApp-jenkins-0212122633-2c564c56_7523f628-beb1-403a-8586-3377fa73d4e7 with
pipeline runner org.apache.beam.runners.flink.FlinkPipelineRunner@7413c14b'
java.lang.OutOfMemoryError: GC overhead limit exceeded
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
