[
https://issues.apache.org/jira/browse/FLINK-24855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442555#comment-17442555
]
WangMinChao commented on FLINK-24855:
-------------------------------------
Hi，[~chesnay] [~Leonard Xu] ，The following is my debug logs：
{code:java}
//代码占位符
ff8e61e6-7282-11ea-bd57-6c92bf629d86:1-7487158, row=0, event=0}}}]
2021-11-11 17:32:41,454 INFO
com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator [] -
The enumerator receives finished split offsets
FinishedSnapshotSplitsReportEvent{finishedOffsets={saas_erp.finance_charge_component_mws:24254={ts_sec=0,
file=mysql-bin.058350, pos=267428709,
gtids=01ecaf60-7fb1-11e9-a248-6c92bf487b6a:1-4498271,
020ac0c5-d0f8-11e9-8483-b8599f2ebcf1:1-48706030,
03acd05f-72fc-11ea-972e-6c92bf628d82:1-1667917206,
0fee5e30-72fc-11ea-ba61-6c92bf62a568:1-91952,
134ab306-c50f-11e9-ae6f-98039bbb4c1f:1-15508246,
18799a0f-b233-11e9-85cc-98039bb6f642:1-771071,
1ac34c9b-c50f-11e9-b43b-b8599f3b1903:1-6800350,
23cbf1c9-ab9a-11e9-83e3-98039bad6fec:1-15197293,
24784255-97ee-11e9-848b-98039bad6fec:1-32199714,
25553581-dbce-11ea-8ed7-6c92bf5bb851:1-3547175,
255d07d7-b233-11e9-b73c-98039bb6ed62:1-49712498,
26736cad-a87b-11e9-9181-98039bad3ec4:1-13114247,
2c0ce471-ab9a-11e9-a1be-98039bb6e722:1-19390458,
2c42592f-97ee-11e9-bc72-98039bb6e722:1-14138587,
2def1322-a87b-11e9-9701-98039bbb4b03:1-3165380,
2eb629b3-a2fd-11e9-9b63-6c92bf622312:1-36626908,
330e84b9-ce0b-11e9-ad80-98039bbb4c1f:1-17546882,
35f851c5-8026-11e9-abf8-246e96805d08:1-58426,
3623ad2b-d8e0-11e9-a1b4-98039bbb4c1f:1-5240296,
37c297f7-5475-11e9-a2d9-246e96765b98:1-22119715,
38ff2745-8c9c-11eb-b538-1c34da535135:1-3994277132,
39fd3c35-2ee0-11ea-84bf-98039bb6ed62:1-805660894,
3a90b65e-ce0b-11e9-ab19-b8599f3b1903:1-27670439,
3ca00c49-a2fd-11e9-b47d-6c92bf62eb5a:1-13246862,
3d6f9eb7-d8e0-11e9-a7bb-b8599f3b1903:1-31884251,
423a0002-8026-11e9-b8dc-6c92bf487b6a:1-11429152,
4292c686-5223-11eb-90ea-043f72de208f:1-703846787,
43f51855-5475-11e9-a059-6c92bf5db002:1-3235912,
45f5291a-92cb-11e9-9944-98039bad6fec:1-11143363,
4d73774f-92cb-11e9-aeb3-98039bb6e722:1-47071013,
4da434bc-546e-11e9-a770-246e96765b98:1-5,
5947cd68-d3ea-11e9-b3c0-b8599f2e4a1c:1-1854410,
5b360b99-a020-11e9-b9d4-6c92bf5dcaa8:1-17640889,
5f810055-6bbe-11e9-ba6c-246e96805d08:1-16119953,
60a07b1b-d3ea-11e9-93aa-b8599f2e8fe4:1-6669015,
686e4f64-a020-11e9-9e6b-246e96731f3a:1-3879557,
6a325ce7-859f-11eb-9d42-0c42a1209fbb:1-561438443,
6beec825-6bbe-11e9-96d6-6c92bf487b6a:1-28641680,
71b28276-eda8-11e9-af81-b8599f2e4a71:1-84167922,
72d9b2a9-672a-11e9-9314-6c92bf6288f4:1-7102912,
7dec8e3c-b42d-11e9-b1fc-98039bad3ec4:1-37011764,
7e06b6e0-eda8-11e9-8c0b-b8599f0fa448:1-9145975,
7ea69321-8a8f-11e9-931b-246e96765b98:1-2712172,
7eed673c-672a-11e9-9122-6c92bf46f8c0:1-9655358,
7fd0c6fb-185d-11eb-8033-0c42a125a02e:1-568940452,
83dd7c85-72ca-11e9-99f7-6c92bf628978:1-6381876,
85a17d17-b42d-11e9-a318-98039bbb4b03:1-20227426,
86027463-e0fd-11e9-b463-98039bad3ec4:1-21191425,
872b4919-e925-11ea-9996-0c42a125f156:1-831445601,
8b25ab00-72ca-11e9-a0a9-6c92bf46f498:1-7169278,
8b86fea9-8a8f-11e9-b915-6c92bf5db002:1-50772519,
8e096735-cc21-11e9-b941-98039bad6fec:1-10910682,
9231e9a5-e0fd-11e9-a070-98039bbb4b03:1-104924985,
967666a6-85d8-11e9-9b24-246e967380e8:1-37053803,
97d11b20-78a7-11e9-9a02-6c92bf628978:1-46636477,
9a37b0ef-cc21-11e9-9448-98039bb6e722:1-12430829,
9aa762fc-f6ea-11e9-8528-98039bb6f642:1-880339,
a2c7771e-85d8-11e9-be8f-246e967708b0:1-29847777,
a3da1925-78a7-11e9-a868-6c92bf46f498:1-2956713,
a4fec594-ddb9-11e9-b12c-98039bad6fec:1-153880,
a7f302e7-dddb-11e9-a133-98039bbb4c1f:1-25426484,
ac5c0db5-ddb9-11e9-b210-98039bb6e722:1-2666206,
adaee5a3-be2a-11e9-b375-98039bad6fec:1-28120481,
ae144a99-aaca-11e9-9cad-98039bad3ec4:1-5101431,
af3e456e-dddb-11e9-b54e-b8599f3b1903:1-4643639,
b2c94ce0-e925-11ea-9cff-0c42a1644ee5:1-581061105,
b5b050e6-aaca-11e9-ace5-98039bbb4b03:1-51359001,
b5c1c335-be2a-11e9-8fdc-98039bb6e722:1-51512164,
bf1ad8e3-8416-11e9-aed1-246e967380e8:1-5394686,
c2d622e2-a212-11e9-8c6e-6c92bf6221e6:1-5674947,
cad7ed3a-8d8a-11e9-b3f7-246e96499060:1-356675,
cb4776a3-8416-11e9-b534-246e967708b0:1-5399318,
cf9571b0-a212-11e9-be97-6c92bf62183e:1-2791994,
d25e8e82-f707-11e9-bd5f-6c0b84d5a92e:1-726774256,
d2d0ab4a-84e5-11e9-8fa3-246e967380e8:1-4658916,
d4eba30e-da19-11e9-a3f8-98039bbb4c1f:1-5386715,
d6de64a3-8d8a-11e9-9e2c-f875884a38d6:1-41397102,
da20e90c-51ca-11eb-a8a9-b026284353c0:1-770104161,
ded0f5d0-da19-11e9-a423-b8599f3b1903:1-1564284,
df271f42-84e5-11e9-a099-246e967708b0:1-11626849,
e270f6fd-80f7-11e9-82e0-6c92bf628978:1-15062968,
ec0ff43c-0ee6-11eb-aeae-0c42a1259f96:1-61576610,
eed28746-80f7-11e9-979a-6c92bf46f498:1-6981252,
f279d2b9-a6a5-11e9-8620-6c92bf48c9f4:1-5135222,
f438104e-7fb0-11e9-a306-246e96805d08:1-96778,
fab159cf-a6a5-11e9-be9d-6c92bf48d2a2:1-11529998,
facc1a3e-d0f7-11e9-9ff3-b8599f2e4999:1-1884165,
ff8e61e6-7282-11ea-bd57-6c92bf629d86:1-7487158, row=0, event=0}}} from subtask
0.
2021-11-11 17:32:41,454 DEBUG
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Handling
event from subtask 0 of source Source: mysqlcdc-source: RequestSplitEvent
(host='10.49.2.144')
2021-11-11 17:32:41,454 INFO
com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator [] -
Assign split MySqlSnapshotSplit{tableId=saas_erp.finance_charge_component_mws,
splitId='saas_erp.finance_charge_component_mws:24258', splitKeyType=[`id`
BIGINT NOT NULL], splitStart=[618627517], splitEnd=[618653019],
highWatermark=null} to subtask 0
2021-11-11 17:32:41,552 DEBUG com.qcloud.cos.http.DefaultCosHttpClient
[] - Received error response:
com.qcloud.cos.exception.CosServiceException: Not Found (Status Code: 404;
Error Code: 404 Not Found; Request ID:
NjE4Y2UzMzlfZWIxNDZiMGJfMjAwMWFfNTQwNGU3Nw==); Trace ID:
OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODc0OWRkZjk0ZDM1NmI1M2E2MTRlY2MzZDhmNmI5MWI1OTBjYzE2MjAxN2M1MzJiOTdkZjMxMDVlYTZjN2FiMmI0YzE1ODg2YjYxODQ2YjgzNGNhZTUzYWQyN2Q3OTFhYjA=
2021-11-11 17:32:41,571 DEBUG com.qcloud.cos.http.DefaultCosHttpClient
[] - Received error response:
com.qcloud.cos.exception.CosServiceException: Not Found (Status Code: 404;
Error Code: 404 Not Found; Request ID:
NjE4Y2UzMzlfZWIxNDZiMGJfMjAwMmNfNTQxYzYwMA==); Trace ID:
OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODc0OWRkZjk0ZDM1NmI1M2E2MTRlY2MzZDhmNmI5MWI1OTBjYzE2MjAxN2M1MzJiOTdkZjMxMDVlYTZjN2FiMmI0YzE1ODg2YjYxODQ2YjgzNGNhZTUzYWQyN2Q3OTFhYjA=
2021-11-11 17:32:41,599 DEBUG com.qcloud.cos.internal.XmlResponsesSaxParser
[] - Sanitizing XML document destined for handler class
com.qcloud.cos.internal.XmlResponsesSaxParser$ListBucketHandler
2021-11-11 17:32:41,599 DEBUG com.qcloud.cos.internal.XmlResponsesSaxParser
[] - Parsing XML response document with handler: class
com.qcloud.cos.internal.XmlResponsesSaxParser$ListBucketHandler
2021-11-11 17:32:41,599 DEBUG com.qcloud.cos.internal.XmlResponsesSaxParser
[] - Examining listing for bucket: akd-flink-test-1254213275
2021-11-11 17:32:41,617 DEBUG com.qcloud.cos.http.DefaultCosHttpClient
[] - Received error response:
com.qcloud.cos.exception.CosServiceException: Not Found (Status Code: 404;
Error Code: 404 Not Found; Request ID:
NjE4Y2UzMzlfZWIxNDZiMGJfMjAwMjFfNTQ3MzlkZA==); Trace ID:
OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODc0OWRkZjk0ZDM1NmI1M2E2MTRlY2MzZDhmNmI5MWI1OTBjYzE2MjAxN2M1MzJiOTdkZjMxMDVlYTZjN2FiMmI0YzE1ODg2YjYxODQ2YjgzNGNhZTUzYWQyN2Q3OTFhYjA=
2021-11-11 17:32:41,654 DEBUG com.qcloud.cos.http.DefaultCosHttpClient
[] - Received error response:
com.qcloud.cos.exception.CosServiceException: Not Found (Status Code: 404;
Error Code: 404 Not Found; Request ID:
NjE4Y2UzMzlfZWIxNDZiMGJfMjAwMzRfNTQxZDllMw==); Trace ID:
OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODc0OWRkZjk0ZDM1NmI1M2E2MTRlY2MzZDhmNmI5MWI1OTBjYzE2MjAxN2M1MzJiOTdkZjMxMDVlYTZjN2FiMmI0YzE1ODg2YjYxODQ2YjgzNGNhZTUzYWQyN2Q3OTFhYjA=
2021-11-11 17:32:41,673 DEBUG com.qcloud.cos.http.DefaultCosHttpClient
[] - Received error response:
com.qcloud.cos.exception.CosServiceException: Not Found (Status Code: 404;
Error Code: 404 Not Found; Request ID:
NjE4Y2UzMzlfZWIxNDZiMGJfMWZmZThfNTNjOWE5YQ==); Trace ID:
OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODc0OWRkZjk0ZDM1NmI1M2E2MTRlY2MzZDhmNmI5MWI1OTBjYzE2MjAxN2M1MzJiOTdkZjMxMDVlYTZjN2FiMmI0YzE1ODg2YjYxODQ2YjgzNGNhZTUzYWQyN2Q3OTFhYjA=
2021-11-11 17:32:41,700 DEBUG com.qcloud.cos.internal.XmlResponsesSaxParser
[] - Sanitizing XML document destined for handler class
com.qcloud.cos.internal.XmlResponsesSaxParser$ListBucketHandler
2021-11-11 17:32:41,700 DEBUG com.qcloud.cos.internal.XmlResponsesSaxParser
[] - Parsing XML response document with handler: class
com.qcloud.cos.internal.XmlResponsesSaxParser$ListBucketHandler
2021-11-11 17:32:41,700 DEBUG com.qcloud.cos.internal.XmlResponsesSaxParser
[] - Examining listing for bucket: akd-flink-test-1254213275
2021-11-11 17:32:41,749 DEBUG com.qcloud.cos.http.DefaultCosHttpClient
[] - Received error response:
com.qcloud.cos.exception.CosServiceException: Not Found (Status Code: 404;
Error Code: 404 Not Found; Request ID:
NjE4Y2UzMzlfZWIxNDZiMGJfMjAwM2FfNTM2ZGFjMA==); Trace ID:
OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODc0OWRkZjk0ZDM1NmI1M2E2MTRlY2MzZDhmNmI5MWI1OTBjYzE2MjAxN2M1MzJiOTdkZjMxMDVlYTZjN2FiMmI0YzE1ODg2YjYxODQ2YjgzNGNhZTUzYWQyN2Q3OTFhYjA=
2021-11-11 17:32:41,770 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 331 (type=CHECKPOINT) @ 1636623161522 for job
fdc482410b221a858bd46ec14dcb3780.
2021-11-11 17:32:41,770 DEBUG
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Taking a
state snapshot on operator Source: mysqlcdc-source for checkpoint 331
2021-11-11 17:32:43,899 DEBUG
com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.PoolBase [] -
connection-pool-172.16.99.11:3306 - Closing connection
com.mysql.cj.jdbc.ConnectionImpl@6eee313f: (connection has passed maxLifetime)
2021-11-11 17:32:43,900 ERROR
org.apache.flink.runtime.util.FatalExitExceptionHandler [] - FATAL: Thread
'SourceCoordinator-Source: mysqlcdc-source' produced an uncaught exception.
Stopping the process...
java.lang.Error: Source Coordinator Thread already exists. There should never
be more than one thread driving the actions of a Source Coordinator. Existing
Thread: Thread[SourceCoordinator-Source: mysqlcdc-source,5,main]
at
org.apache.flink.runtime.source.coordinator.SourceCoordinatorProvider$CoordinatorExecutorThreadFactory.newThread(SourceCoordinatorProvider.java:119)
[flink-dist_2.12-1.13.3.jar:1.13.3]
at
java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619)
~[?:1.8.0_191]
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
~[?:1.8.0_191]
at
java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1025)
~[?:1.8.0_191]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
~[?:1.8.0_191]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
2021-11-11 17:32:43,901 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Shutting
YarnJobClusterEntrypoint down with application status UNKNOWN. Diagnostics
Cluster entrypoint has been closed externally..
2021-11-11 17:32:43,902 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shutting
down rest endpoint.
2021-11-11 17:32:43,902 INFO org.apache.flink.runtime.blob.BlobServer
[] - Stopped BLOB server at 0.0.0.0:36980
2021-11-11 17:32:43,908 DEBUG
com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool [] -
connection-pool-172.16.99.11:3306 - Added connection
com.mysql.cj.jdbc.ConnectionImpl@54a7a770
2021-11-11 17:32:43,909 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 8
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-20
2021-11-11 17:32:43,909 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 7
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-25
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 8
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-11
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 9
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-3
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 10
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-13
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 8
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-8
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 9
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-23
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 9
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-15
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 9
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-9
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 9
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-5
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 10
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-12
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 8
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-18
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 8
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-14
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 8
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-30
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 8
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-16
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 8
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-24
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 8
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-27
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 8
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-17
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 9
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-6
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 8
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-4
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 8
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-7
2021-11-11 17:32:43,913 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 9
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-32
2021-11-11 17:32:43,914 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 9
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-29
2021-11-11 17:32:43,914 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 9
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-31
2021-11-11 17:32:43,914 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 9
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-2
2021-11-11 17:32:43,914 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 8
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-19
2021-11-11 17:32:43,914 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 9
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-21
2021-11-11 17:32:43,914 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 8
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-22
2021-11-11 17:32:43,914 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 7
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-26
2021-11-11 17:32:43,914 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 9
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-1
2021-11-11 17:32:43,915 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 9
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-10
2021-11-11 17:32:43,915 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 9
thread-local buffer(s) from thread: flink-rest-server-netty-worker-thread-28
2021-11-11 17:32:43,917 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Removing
cache directory /tmp/flink-web-0988b053-3945-4968-afac-d6b23803a11b/flink-web-ui
2021-11-11 17:32:43,919 INFO
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Stopping DefaultLeaderElectionService.
2021-11-11 17:32:43,919 INFO
org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionDriver [] -
Closing ZooKeeperLeaderElectionDriver{leaderPath='/leader/rest_server_lock'}
2021-11-11 17:32:43,921 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shut down
complete.
2021-11-11 17:32:43,921 INFO
org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent
[] - Closing components.
2021-11-11 17:32:43,921 INFO
org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] -
Stopping DefaultLeaderRetrievalService.
2021-11-11 17:32:43,921 INFO
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalDriver [] -
Closing ZookeeperLeaderRetrievalDriver{retrievalPath='/leader/dispatcher_lock'}.
2021-11-11 17:32:43,921 INFO
org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] -
Stopping DefaultLeaderRetrievalService.
2021-11-11 17:32:43,921 INFO
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalDriver [] -
Closing
ZookeeperLeaderRetrievalDriver{retrievalPath='/leader/resource_manager_lock'}.
2021-11-11 17:32:43,921 INFO
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Stopping DefaultLeaderElectionService.
2021-11-11 17:32:43,921 INFO
org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionDriver [] -
Closing ZooKeeperLeaderElectionDriver{leaderPath='/leader/dispatcher_lock'}
2021-11-11 17:32:43,921 INFO
org.apache.flink.runtime.dispatcher.runner.JobDispatcherLeaderProcess [] -
Stopping JobDispatcherLeaderProcess.
2021-11-11 17:32:43,921 INFO
org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Stopping
dispatcher akka.tcp://[email protected]:36456/user/rpc/dispatcher_1.
2021-11-11 17:32:43,921 INFO
org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Stopping all
currently running jobs of dispatcher
akka.tcp://[email protected]:36456/user/rpc/dispatcher_1.
2021-11-11 17:32:43,921 DEBUG
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner [] -
Terminating the leadership runner for job fdc482410b221a858bd46ec14dcb3780.
2021-11-11 17:32:43,922 DEBUG
org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess [] -
Terminating the JobMasterService process for job
fdc482410b221a858bd46ec14dcb3780 under leader id
62876805-1ef2-4d1d-aa94-a0325107ade6.
2021-11-11 17:32:43,924 INFO
org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Job
fdc482410b221a858bd46ec14dcb3780 reached terminal state SUSPENDED.
2021-11-11 17:32:43,924 INFO
org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Shutting down
cluster with state SUSPENDED, jobCancelled: false, executionMode: DETACHED
2021-11-11 17:32:43,947 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager []
- Closing the slot manager.
2021-11-11 17:32:43,947 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager []
- Suspending the slot manager.
2021-11-11 17:32:43,947 DEBUG
org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager []
- Unregistering task executor 8f2a20d3e80e6b4cc952016415f4c86f from the slot
manager.
2021-11-11 17:32:43,947 DEBUG
org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager []
- Unregistering task executor 9c8fe5844af415c25c0ee31e5b627f37 from the slot
manager.
2021-11-11 17:32:43,948 DEBUG
org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager []
- Unregistering task executor 56ef6ea6501ae49609139e06d607a959 from the slot
manager.
2021-11-11 17:32:43,948 DEBUG
org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager []
- Unregistering task executor 8a7abcd42b323a4a71540dbfd5734781 from the slot
manager.
2021-11-11 17:32:43,948 INFO
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Stopping DefaultLeaderElectionService.
2021-11-11 17:32:43,948 INFO
org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionDriver [] -
Closing
ZooKeeperLeaderElectionDriver{leaderPath='/leader/resource_manager_lock'}
2021-11-11 17:32:43,948 INFO
org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] -
Stopping DefaultLeaderRetrievalService.
2021-11-11 17:32:43,948 INFO
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalDriver [] -
Closing
ZookeeperLeaderRetrievalDriver{retrievalPath='/leader/fdc482410b221a858bd46ec14dcb3780/job_manager_lock'}.
2021-11-11 17:32:43,953 DEBUG
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor [] - The
RpcEndpoint resourcemanager_0 terminated successfully.
2021-11-11 17:32:43,957 DEBUG org.apache.flink.runtime.rpc.akka.SupervisorActor
[] - AkkaRpcActor akka://flink/user/rpc/resourcemanager_0 has
terminated.
2021-11-11 17:32:43,958 INFO org.apache.flink.runtime.history.FsJobArchivist
[] - Job fdc482410b221a858bd46ec14dcb3780 has been archived at
hdfs:/flink/completed-jobs/fdc482410b221a858bd46ec14dcb3780.
2021-11-11 17:32:44,222 WARN akka.remote.ReliableDeliverySupervisor
[] - Association with remote system
[akka.tcp://[email protected]:34103] has failed, address is now gated for [50]
ms. Reason: [Disassociated]
2021-11-11 17:32:44,222 WARN akka.remote.ReliableDeliverySupervisor
[] - Association with remote system
[akka.tcp://[email protected]:46867] has failed, address is now gated
for [50] ms. Reason: [Disassociated]
2021-11-11 17:32:44,228 WARN akka.remote.ReliableDeliverySupervisor
[] - Association with remote system
[akka.tcp://[email protected]:33326] has failed, address is now gated
for [50] ms. Reason: [Disassociated]
2021-11-11 17:32:44,228 WARN akka.remote.ReliableDeliverySupervisor
[] - Association with remote system
[akka.tcp://[email protected]:39261] has failed, address is now gated for [50]
ms. Reason: [Disassociated]
2021-11-11 17:32:44,244 WARN akka.remote.ReliableDeliverySupervisor
[] - Association with remote system
[akka.tcp://[email protected]:41545] has failed, address is now gated
for [50] ms. Reason: [Disassociated]
2021-11-11 17:32:44,245 WARN akka.remote.ReliableDeliverySupervisor
[] - Association with remote system
[akka.tcp://[email protected]:42439] has failed, address is now gated for [50]
ms. Reason: [Disassociated]
2021-11-11 17:32:44,264 WARN akka.remote.ReliableDeliverySupervisor
[] - Association with remote system
[akka.tcp://[email protected]:38888] has failed, address is now gated
for [50] ms. Reason: [Disassociated]
2021-11-11 17:32:44,264 WARN akka.remote.ReliableDeliverySupervisor
[] - Association with remote system
[akka.tcp://[email protected]:43905] has failed, address is now gated for [50]
ms. Reason: [Disassociated]
2021-11-11 17:32:44,747 DEBUG
com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.PoolBase [] -
connection-pool-172.16.99.11:3306 - Closing connection
com.mysql.cj.jdbc.ConnectionImpl@408a2079: (connection has passed maxLifetime)
2021-11-11 17:32:44,756 DEBUG
com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool [] -
connection-pool-172.16.99.11:3306 - Added connection
com.mysql.cj.jdbc.ConnectionImpl@2570929d
2021-11-11 17:32:48,209 DEBUG
org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter [] - Invalid
type for Gauge
org.apache.flink.runtime.checkpoint.CheckpointStatsTracker$LatestCompletedCheckpointExternalPathGauge@4ff366e8:
java.lang.String, only number types and booleans are supported by this
reporter.End of LogType:jobmanager.log
{code}
> Source Coordinator Thread already exists. There should never be more than one
> thread driving the actions of a Source Coordinator.
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-24855
> URL: https://issues.apache.org/jira/browse/FLINK-24855
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Runtime / Coordination
> Affects Versions: 1.13.3
> Environment: flink 1.13.3
> flink-cdc 2.1
> Reporter: WangMinChao
> Priority: Critical
>
>
> When I am synchronizing large tables, have the following problems :
> 2021-11-09 20:33:04,222 INFO
> com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator []
> - Assign split MySqlSnapshotSplit\{tableId=db.table, splitId='db.table:383',
> splitKeyType=[`id` BIGINT NOT NULL], splitStart=[9798290],
> splitEnd=[9823873], highWatermark=null} to subtask 1
> 2021-11-09 20:33:04,248 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 101 (type=CHECKPOINT) @ 1636461183945 for job
> 3cee105643cfee78b80cd0a41143b5c1.
> 2021-11-09 20:33:10,734 ERROR
> org.apache.flink.runtime.util.FatalExitExceptionHandler [] - FATAL: Thread
> 'SourceCoordinator-Source: mysqlcdc-source -> Sink: kafka-sink' produced an
> uncaught exception. Stopping the process...
> java.lang.Error: Source Coordinator Thread already exists. There should never
> be more than one thread driving the actions of a Source Coordinator. Existing
> Thread: Thread[SourceCoordinator-Source: mysqlcdc-source -> Sink:
> kafka-sink,5,main]
> at
> org.apache.flink.runtime.source.coordinator.SourceCoordinatorProvider$CoordinatorExecutorThreadFactory.newThread(SourceCoordinatorProvider.java:119)
> [flink-dist_2.12-1.13.3.jar:1.13.3]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619)
> ~[?:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
> ~[?:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1025)
> ~[?:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
> ~[?:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_191]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
