See
<https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/951/display/redirect?page=changes>
Changes:
[chamikaramj] Fixes a case where BigQuery sink swallows HttpErrors preventing
retries.
[Boyuan Zhang] [BEAM-10670] Use non-SDF based translation for Read by default
on all
[noreply] [BEAM-9547] Add support for drop_duplicates and duplicated (#14786)
[noreply] Don't compile with 8 if flag is set (#14838)
[noreply] [BEAM-11644] Remove pack_all_combiners optimizer phase (#14647)
[noreply] [BEAM-12341] Enable Portable job submission for Dataflow (#14822)
[noreply] [BEAM-9615] Enable Schemas for struct coding. (#14826)
[noreply] [BEAM-7372] cleanup py2 codepath from combiners.py (#14634)
[noreply] [BEAM-7372] cleanup py2 and py35 codepath from apache_beam/typehints
------------------------------------------
[...truncated 162.78 KB...]
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "ref_AppliedPTransform_stateful_15"
}
}, {
"urn": "beam:metric:ptransform_execution_time:total_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "fn/read/ref_PCollection_PCollection_9:0"
}
}, {
"urn": "beam:metric:user:sum_int64:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "bg==",
"labels": {
"NAMESPACE": "apache_beam.runners.portability.flink_runner_test.DoFn",
"PTRANSFORM": "ref_AppliedPTransform_stateful_15",
"NAME": "elem_counter"
}
}, {
"urn": "beam:metric:statecache:clear_total",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA=="
}, {
"urn": "beam:metric:sampled_byte_size:v1",
"type": "beam:metrics:distribution_int64:v1",
"payload": "JoYFERE=",
"labels": {
"PCOLLECTION": "ref_PCollection_PCollection_9"
}
}, {
"urn": "beam:metric:sampled_byte_size:v1",
"type": "beam:metrics:distribution_int64:v1",
"payload": "AAD//////////3+AgICAgICAgIAB",
"labels": {
"PCOLLECTION": "ref_PCollection_PCollection_10"
}
}, {
"urn": "beam:metric:pardo_execution_time:start_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM":
"ref_AppliedPTransform_create-FlatMap-lambda-at-core-py-2939-_4"
}
}, {
"urn": "beam:metric:pardo_execution_time:process_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM":
"ref_AppliedPTransform_create-FlatMap-lambda-at-core-py-2939-_4"
}
}, {
"urn": "beam:metric:pardo_execution_time:finish_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM":
"ref_AppliedPTransform_create-FlatMap-lambda-at-core-py-2939-_4"
}
}, {
"urn": "beam:metric:pardo_execution_time:process_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "fn/read/ref_PCollection_PCollection_1:0"
}
}, {
"urn": "beam:metric:ptransform_execution_time:total_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM":
"ref_AppliedPTransform_create-FlatMap-lambda-at-core-py-2939-_4"
}
}, {
"urn": "beam:metric:pardo_execution_time:start_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "fn/read/ref_PCollection_PCollection_1:0"
}
}, {
"urn": "beam:metric:pardo_execution_time:process_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "fn/write/ref_PCollection_PCollection_2:0"
}
}, {
"urn": "beam:metric:data_channel:read_index:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AQ==",
"labels": {
"PTRANSFORM": "fn/read/ref_PCollection_PCollection_1:0"
}
}, {
"urn": "beam:metric:pardo_execution_time:start_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "fn/write/ref_PCollection_PCollection_2:0"
}
}, {
"urn": "beam:metric:pardo_execution_time:finish_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "fn/write/ref_PCollection_PCollection_2:0"
}
}, {
"urn": "beam:metric:ptransform_execution_time:total_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "fn/write/ref_PCollection_PCollection_2:0"
}
}, {
"urn": "beam:metric:ptransform_execution_time:total_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "fn/read/ref_PCollection_PCollection_1:0"
}
}, {
"urn": "beam:metric:element_count:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "bg==",
"labels": {
"PCOLLECTION": "ref_PCollection_PCollection_2"
}
}, {
"urn": "beam:metric:pardo_execution_time:finish_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "fn/read/ref_PCollection_PCollection_1:0"
}
}, {
"urn": "beam:metric:element_count:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AQ==",
"labels": {
"PCOLLECTION": "ref_PCollection_PCollection_1"
}
}, {
"urn": "beam:metric:sampled_byte_size:v1",
"type": "beam:metrics:distribution_int64:v1",
"payload": "JYYEDg4=",
"labels": {
"PCOLLECTION": "ref_PCollection_PCollection_2"
}
}, {
"urn": "beam:metric:sampled_byte_size:v1",
"type": "beam:metrics:distribution_int64:v1",
"payload": "AQ0NDQ==",
"labels": {
"PCOLLECTION": "ref_PCollection_PCollection_1"
}
}]
}
}
May 20, 2021 12:23:28 AM
org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager shutdown
INFO: Shutting down TaskExecutorLocalStateStoresManager.
May 20, 2021 12:23:28 AM org.apache.flink.runtime.webmonitor.WebMonitorEndpoint
lambda$shutDownInternal$5
INFO: Removing cache directory /tmp/flink-web-ui
May 20, 2021 12:23:28 AM org.apache.flink.runtime.rest.RestServerEndpoint
lambda$closeAsync$1
INFO: Shut down complete.
May 20, 2021 12:23:28 AM
org.apache.flink.runtime.resourcemanager.ResourceManager deregisterApplication
INFO: Shut down cluster because application is in CANCELED, diagnostics
DispatcherResourceManagerComponent has been closed..
May 20, 2021 12:23:28 AM
org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent
closeAsyncInternal
INFO: Closing components.
May 20, 2021 12:23:28 AM
org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess
closeInternal
INFO: Stopping SessionDispatcherLeaderProcess.
May 20, 2021 12:23:28 AM org.apache.flink.runtime.dispatcher.Dispatcher onStop
INFO: Stopping dispatcher akka://flink/user/rpc/dispatcher_46.
May 20, 2021 12:23:28 AM org.apache.flink.runtime.dispatcher.Dispatcher
terminateRunningJobs
INFO: Stopping all currently running jobs of dispatcher
akka://flink/user/rpc/dispatcher_46.
May 20, 2021 12:23:28 AM
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl close
INFO: Closing the SlotManager.
May 20, 2021 12:23:28 AM
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl suspend
INFO: Suspending the SlotManager.
May 20, 2021 12:23:28 AM
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl suspend
INFO: Suspending SlotPool.
May 20, 2021 12:23:28 AM org.apache.flink.runtime.jobmaster.JobMaster
dissolveResourceManagerConnection
INFO: Close ResourceManager connection c36e7efa93b9a9135bf5a3c579f0275f:
Stopping JobMaster for job
test_flink_metrics_1621470203.2124438(fbe16d32f5083b9932744cffdd2b1bb0)..
May 20, 2021 12:23:28 AM
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl close
INFO: Stopping SlotPool.
May 20, 2021 12:23:28 AM
org.apache.flink.runtime.rest.handler.legacy.backpressure.BackPressureRequestCoordinator
shutDown
INFO: Shutting down back pressure request coordinator.
May 20, 2021 12:23:28 AM org.apache.flink.runtime.dispatcher.Dispatcher
lambda$onStop$0
INFO: Stopped dispatcher akka://flink/user/rpc/dispatcher_46.
------------------------------ Captured log call -------------------------------
WARNING root:environments.py:374 Make sure that locally built Python SDK
docker image has Python 3.6 interpreter.
WARNING apache_beam.options.pipeline_options:pipeline_options.py:324
Discarding invalid overrides: {'max_bundle_time_millis': 95130590130}
--------------------------- Captured stderr teardown ---------------------------
May 20, 2021 12:23:28 AM
org.apache.flink.runtime.io.disk.FileChannelManagerImpl lambda$getFileCloser$0
INFO: FileChannelManager removed spill file directory
/tmp/flink-io-b5a43705-6216-43e7-b5c3-cb6bc7a880c7
May 20, 2021 12:23:28 AM
org.apache.flink.runtime.io.network.NettyShuffleEnvironment close
INFO: Shutting down the network environment and its components.
May 20, 2021 12:23:28 AM
org.apache.flink.runtime.io.disk.FileChannelManagerImpl lambda$getFileCloser$0
INFO: FileChannelManager removed spill file directory
/tmp/flink-netty-shuffle-5d902602-60e5-4d42-936f-a1c86e034af7
May 20, 2021 12:23:28 AM org.apache.flink.runtime.taskexecutor.KvStateService
shutdown
INFO: Shutting down the kvState service and its components.
May 20, 2021 12:23:28 AM
org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService stop
INFO: Stop job leader service.
May 20, 2021 12:23:28 AM org.apache.flink.runtime.filecache.FileCache shutdown
INFO: removed file cache directory
/tmp/flink-dist-cache-5548ca18-e50c-4beb-b50c-8226f4e9eca5
May 20, 2021 12:23:28 AM org.apache.flink.runtime.taskexecutor.TaskExecutor
handleOnStopException
INFO: Stopped TaskExecutor akka://flink/user/rpc/taskmanager_44.
May 20, 2021 12:23:28 AM org.apache.flink.runtime.rpc.akka.AkkaRpcService
stopService
INFO: Stopping Akka RPC service.
May 20, 2021 12:23:28 AM org.apache.beam.runners.flink.metrics.FileReporter
close
INFO: wrote metrics to
<https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/ws/src/sdks/python/test-suites/portable/py36/build/srcs/sdks/python/target/.tox-flink-runner-test/flink-runner-test/tmp/flinktest-confknner048/test-metrics.txt>
May 20, 2021 12:23:28 AM org.apache.flink.runtime.rpc.akka.AkkaRpcService
stopService
INFO: Stopping Akka RPC service.
May 20, 2021 12:23:28 AM org.apache.flink.runtime.rpc.akka.AkkaRpcService
lambda$stopService$7
INFO: Stopped Akka RPC service.
- generated xml file:
<https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/ws/src/sdks/python/test-suites/portable/py36/build/srcs/sdks/python/pytest_flink-runner-test.xml>
-
============= 1 failed, 126 passed, 38 skipped in 1120.64 seconds ==============
[31mERROR: InvocationError for command
<https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/ws/src/sdks/python/test-suites/portable/py36/build/srcs/sdks/python/scripts/pytest_validates_runner.sh>
flink-runner-test apache_beam/runners/portability/flink_runner_test.py
'--flink_job_server_jar=<https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/ws/src/runners/flink/1.12/job-server/build/libs/beam-runners-flink-1.12-job-server-2.31.0-SNAPSHOT.jar>
--environment_type=LOOPBACK' (exited with code 1)
[0m[1mflink-runner-test run-test-post: commands[0] | bash
<https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/ws/src/sdks/python/test-suites/portable/py36/build/srcs/sdks/python/scripts/run_tox_cleanup.sh>
[0m___________________________________ summary
____________________________________
[31mERROR: flink-runner-test: commands failed
[0m
> Task :sdks:python:test-suites:portable:py36:flinkCompatibilityMatrixLOOPBACK
> FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py36:flinkCompatibilityMatrixLOOPBACK'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 27m 36s
139 actionable tasks: 98 executed, 39 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/53qmo35bgj7ms
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
