[
https://issues.apache.org/jira/browse/KAFKA-8104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16911182#comment-16911182
]
Gregory Koshelev commented on KAFKA-8104:
-----------------------------------------
[~guozhang], I found server.log (there are 3 brokers, but only one of them has
relevant log entries in log):
{code:text}
[2019-03-10 02:46:43,627] INFO [GroupCoordinator 2]: Stabilized group
hercules.sink.elastic.legacy_logs_elk_c2 generation 10693
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:43,630] INFO [GroupCoordinator 2]: Assignment received from
leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10693
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:45,397] INFO [GroupCoordinator 2]: Preparing to rebalance
group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10693
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member
consumer-7-cc600ccb-72b6-4582-b288-3ea966f33c7c in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member
consumer-1-f284c5c8-22fc-40d5-aa30-46c2875463bd in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member
consumer-5-e608adfc-e9ad-427e-99b5-99ff1538611d in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member
consumer-6-daa568cc-a5ee-49a9-aa7c-bd34fbb1da52 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member
consumer-2-da70d65b-53e6-4467-b06d-63f02ff89eec in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member
consumer-12-a0cb16ff-46eb-4945-a87a-244a20261af9 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member
consumer-10-c10cb15d-edc3-4c6a-b21a-68e2b8298284 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member
consumer-16-d80c8a16-be83-48f3-ac9d-08431b87c198 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member
consumer-5-75899722-91a9-439a-a0c5-f23eb7c6c93f in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member
consumer-11-14fe64e9-1553-49e1-abf9-5ea623847c84 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member
consumer-3-9b62bde0-ff71-46f7-b6a4-7be14e6055b5 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-14-42336dad-be7c-49da-a76b-72d13a856943 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-6-1d57e91e-a6cd-4917-9560-19fa6579d02e in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-13-a310873e-5c66-402c-a031-0b2fce8eb4cc in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-1-7c591ed8-01d8-4f2c-b756-a5c3d3f63eea in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-7-70c381bc-5d32-4e2c-923b-2c15c0f15aa2 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-15-5de2a785-0cd8-43f8-b3e2-6b1da2983910 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-10-92f6d7a8-cc6c-4091-9eb4-1a8be9a08541 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-5-bd971902-bea5-49ef-83f6-3788e7363924 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-11-18b7fff3-f477-44c8-bece-4bde079bb578 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-2-bb9ff074-6738-4092-a2c0-f3d49e5e7e2a in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-9-ef6b36b3-376f-4a04-a0cb-709c8a28712b in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-13-57c73f24-4e8d-407e-874b-ab95409f7b1c in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-10-265e41ea-c8c3-47f2-96d2-33309f975f29 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-15-f3ea2d41-09df-454a-a13b-c2007e0bff32 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-11-dcbc5baf-b7e4-433f-9580-b5e846418748 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-14-3cfd686d-e590-4055-9c8c-70f673a0a624 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-8-43d4c7af-a8f5-4616-8346-33bed246a253 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-11-bf244902-001f-460c-8cf5-e9ac5e7cb4a6 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-10-74f336ca-6a0f-4603-b93a-fcee81953f9f in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-1-65fe1595-6bff-4506-8a9b-6f35f3663f48 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-16-470b7eb9-942d-48fa-b8c1-d128ae21900d in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member
consumer-2-d587e29f-4d7e-4234-a774-21813af82415 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-7-a1fbd1a5-a4a8-42a8-bf7e-d6742edb8003 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-16-47eaa6a5-1fe9-4216-8afa-aaa5a7684755 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-10-1cba9737-26c2-4879-82c7-d3ef6399a2b3 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-13-b4031cae-43c3-45a4-b258-b9cfe2645eb6 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-3-e23f1744-7781-4e09-b394-6e3a09a5ed6d in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-4-59459730-4118-4379-a208-83bfba9a751e in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-9-4291927f-119f-4610-bd00-f9503847ff05 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-6-b8d722e7-58d3-4e2f-af03-92ab01e23f8d in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-14-b2504ede-535c-4884-aea5-03734661bb4d in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-6-afb90678-4548-4a23-bbbb-3ce2ffb6b59c in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-15-ecd3673c-c585-4fab-98ae-b14f5583e59d in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-7-d534cf40-ecf6-4cac-a823-c5bb7d515ffe in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-15-6c51bd3b-88c0-407a-a606-f21f8a16b22e in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-16-47e4622c-aea3-4a7f-b3c4-c4e88c553c63 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-13-6aef6aef-2fa7-4499-a138-8132f10662fd in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-1-d5563ddf-63dc-4308-b456-881b0f3bd64f in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-3-323295d3-d5b5-49f4-a364-18c681fcc534 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-3-494d66e9-5225-4673-8400-ebf1f96d5e5e in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-1-5035e670-86ff-4365-9be1-daf28bb85894 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-7-2bccc8ef-31f0-4974-8042-4721d60d0be4 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-13-e5e10c2d-28a2-4def-a5ec-2e33e3162c6c in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-4-b18fc59c-d2e0-474b-9eb7-76150001406a in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-1-c99e56c9-bd3b-4230-a3fa-c01ae0208333 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-13-c7331006-f001-4cee-b7f7-feb4be3db7e5 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-5-fcf52494-4825-43da-8b8f-622b69f19707 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-4-7f88af3b-bcc8-475c-84e0-e254871998b3 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-16-b5952574-6146-4f4f-9fb9-e5a1a0165296 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-3-0b751ea6-b228-489b-9679-381173ac7bb8 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-6-5f5ed961-0ac6-4463-8e2d-87a0d40de556 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-9-180c4643-92f2-4989-b807-b6a4f8ab4ac9 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-15-24048d2d-7442-4392-af17-b4a959e58bf7 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-5-6fc6f5b5-cd67-4cb6-a8fb-b4b4dd8fc2ea in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-8-9fca323f-e32d-4e4a-99eb-356557892607 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-3-3b403a81-75d5-44df-b9d5-c0accc173e67 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-15-dbd7b016-fb76-4ce0-be8e-753995fa701e in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-12-39204442-c071-46d5-8f5d-45a89dbe9426 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-11-dd4283b7-9226-496c-834e-4abfba06396c in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-2-f5f81416-0d70-4d44-86b4-cf834add0c5c in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member
consumer-12-a889fd7d-edc8-4323-9aea-4e7c98f24e64 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member
consumer-6-cbe5c48d-24ac-4aab-87a8-227c1292bffa in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member
consumer-2-5e1893ac-76ee-495c-9624-ee06b8004565 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member
consumer-5-3f0832ba-0b9e-4ff1-8e10-1ba9c61a7b67 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member
consumer-8-ab32a5e0-4781-487c-8e8f-2a163d1437dd in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member
consumer-7-4457a612-a996-43ed-95ee-9a0e1c9e5371 in group
hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0
expired offsets in 0 milliseconds.
(kafka.coordinator.group.GroupMetadataManager)
[2019-03-10 02:47:09,911] WARN Attempting to send response via channel for
which there is no open connection, connection id
10.0.3.151:9092-10.0.3.60:41792-1811027 (kafka.network.Processor)
[2019-03-10 02:47:15,431] INFO [GroupCoordinator 2]: Stabilized group
hercules.sink.elastic.legacy_logs_elk_c2 generation 10694
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:15,433] INFO [GroupCoordinator 2]: Preparing to rebalance
group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10694
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:15,436] INFO [GroupCoordinator 2]: Stabilized group
hercules.sink.elastic.legacy_logs_elk_c2 generation 10695
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:15,437] INFO [GroupCoordinator 2]: Assignment received from
leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10695
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:22,774] INFO [GroupCoordinator 2]: Preparing to rebalance
group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10695
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:42,111] INFO [GroupCoordinator 2]: Stabilized group
hercules.sink.elastic.legacy_logs_elk_c2 generation 10696
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:42,113] INFO [GroupCoordinator 2]: Assignment received from
leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10696
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:48:04,754] WARN Attempting to send response via channel for
which there is no open connection, connection id
10.0.3.151:9092-10.0.3.60:43668-1811310 (kafka.network.Processor)
[2019-03-10 02:48:10,975] WARN Attempting to send response via channel for
which there is no open connection, connection id
10.0.3.151:9092-10.0.3.153:56162-1811343 (kafka.network.Processor)
[2019-03-10 02:48:32,082] WARN Attempting to send response via channel for
which there is no open connection, connection id
10.0.3.151:9092-10.0.3.60:44598-1811455 (kafka.network.Processor)
[2019-03-10 02:50:09,387] WARN Attempting to send response via channel for
which there is no open connection, connection id
10.0.3.151:9092-10.0.3.60:48904-1811956 (kafka.network.Processor)
[2019-03-10 02:50:16,787] INFO [GroupCoordinator 2]: Preparing to rebalance
group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10696
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:50:22,630] INFO [GroupCoordinator 2]: Stabilized group
hercules.sink.elastic.legacy_logs_elk_c2 generation 10697
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:50:22,633] INFO [GroupCoordinator 2]: Assignment received from
leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10697
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:51:13,251] INFO [GroupCoordinator 2]: Preparing to rebalance
group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10697
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:51:13,967] INFO [GroupCoordinator 2]: Stabilized group
hercules.sink.elastic.legacy_logs_elk_c2 generation 10698
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:51:13,969] INFO [GroupCoordinator 2]: Assignment received from
leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10698
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:57:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0
expired offsets in 0 milliseconds.
(kafka.coordinator.group.GroupMetadataManager)
[2019-03-10 03:13:44,854] WARN Attempting to send response via channel for
which there is no open connection, connection id
10.0.3.151:9092-10.0.3.153:42480-1819319 (kafka.network.Processor)
[2019-03-10 03:16:11,256] INFO [GroupCoordinator 2]: Preparing to rebalance
group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10698
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:16:39,196] INFO [GroupCoordinator 2]: Stabilized group
hercules.sink.elastic.legacy_logs_elk_c2 generation 10699
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:16:39,204] INFO [GroupCoordinator 2]: Assignment received from
leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10699
(kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:16:39,221] INFO [GroupCoordinator 2]: Preparing to rebalance
group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10699
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:17:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0
expired offsets in 0 milliseconds.
(kafka.coordinator.group.GroupMetadataManager)
[2019-03-10 03:17:31,693] INFO [GroupCoordinator 2]: Stabilized group
hercules.sink.elastic.legacy_logs_elk_c2 generation 10700
(__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:17:31,695] INFO [GroupCoordinator 2]: Assignment received from
leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10700
(kafka.coordinator.group.GroupCoordinator)
{code}
[2019-03-10 03:20:05,312] WARN Attempting to send response via channel for
which there is no open connection, connection id
10.0.3.151:9092-10.0.3.60:60490-1821319 (kafka.network.Processor)
[2019-03-10 03:22:56,928] WARN Attempting to send response via channel for
which there is no open connection, connection id
10.0.3.151:9092-10.0.3.60:38542-1822186 (kafka.network.Processor)
[2019-03-10 03:24:43,989] WARN Attempting to send response via channel for
which there is no open connection, connection id
10.0.3.151:9092-10.0.3.60:42936-1822733 (kafka.network.Processor)
[2019-03-10 03:27:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0
expired offsets in 0 milliseconds.
(kafka.coordinator.group.GroupMetadataManager)
[2019-03-10 03:37:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0
expired offsets in 0 milliseconds.
(kafka.coordinator.group.GroupMetadataManager)
{code}
I've skipped logs about segment deletion and so on.
> Consumer cannot rejoin to the group after rebalancing
> -----------------------------------------------------
>
> Key: KAFKA-8104
> URL: https://issues.apache.org/jira/browse/KAFKA-8104
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 2.0.0
> Reporter: Gregory Koshelev
> Priority: Critical
> Attachments: consumer-rejoin-fail.log
>
>
> TL;DR; {{KafkaConsumer}} cannot rejoin to the group due to inconsistent
> {{AbstractCoordinator.generation}} (which is {{NO_GENERATION}} and
> {{AbstractCoordinator.joinFuture}} (which is succeeded {{RequestFuture}}).
> See explanation below.
> There are 16 consumers in single process (threads from pool-4-thread-1 to
> pool-4-thread-16). All of them belong to single consumer group
> {{hercules.sink.elastic.legacy_logs_elk_c2}}. Rebalancing has been acquired
> and consumers have got {{CommitFailedException}} as expected:
> {noformat}
> 2019-03-10T03:16:37.023Z [pool-4-thread-10] WARN
> r.k.vostok.hercules.sink.SimpleSink - Commit failed due to rebalancing
> org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be
> completed since the group has already rebalanced and assigned the partitions
> to another member. This means that the time between subsequent calls to
> poll() was longer than the configured max.poll.interval.ms, which typically
> implies that the poll loop is spending too much time message processing. You
> can address this either by increasing the session timeout or by reducing the
> maximum size of batches returned in poll() with max.poll.records.
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.sendOffsetCommitRequest(ConsumerCoordinator.java:798)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:681)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1334)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1298)
> at ru.kontur.vostok.hercules.sink.Sink.commit(Sink.java:156)
> at ru.kontur.vostok.hercules.sink.SimpleSink.run(SimpleSink.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> After that, most of them successfully rejoined to the group with generation
> 10699:
> {noformat}
> 2019-03-10T03:16:39.208Z [pool-4-thread-13] INFO
> o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-13,
> groupId=hercules.sink.elastic.legacy_logs_elk_c2] Successfully joined group
> with generation 10699
> 2019-03-10T03:16:39.209Z [pool-4-thread-13] INFO
> o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-13,
> groupId=hercules.sink.elastic.legacy_logs_elk_c2] Setting newly assigned
> partitions [legacy_logs_elk_c2-18]
> ...
> 2019-03-10T03:16:39.216Z [pool-4-thread-11] INFO
> o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-11,
> groupId=hercules.sink.elastic.legacy_logs_elk_c2] Successfully joined group
> with generation 10699
> 2019-03-10T03:16:39.217Z [pool-4-thread-11] INFO
> o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-11,
> groupId=hercules.sink.elastic.legacy_logs_elk_c2] Setting newly assigned
> partitions [legacy_logs_elk_c2-10, legacy_logs_elk_c2-11]
> ...
> 2019-03-10T03:16:39.218Z [pool-4-thread-15] INFO
> o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-15,
> groupId=hercules.sink.elastic.legacy_logs_elk_c2] Setting newly assigned
> partitions [legacy_logs_elk_c2-24]
> 2019-03-10T03:16:42.320Z [kafka-coordinator-heartbeat-thread |
> hercules.sink.elastic.legacy_logs_elk_c2] INFO
> o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-6,
> groupId=hercules.sink.elastic.legacy_logs_elk_c2] Attempt to heartbeat failed
> since group is rebalancing
> 2019-03-10T03:16:42.320Z [kafka-coordinator-heartbeat-thread |
> hercules.sink.elastic.legacy_logs_elk_c2] INFO
> o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-5,
> groupId=hercules.sink.elastic.legacy_logs_elk_c2] Attempt to heartbeat failed
> since group is rebalancing
> 2019-03-10T03:16:42.323Z [kafka-coordinator-heartbeat-thread |
> hercules.sink.elastic.legacy_logs_elk_c2] INFO
> o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-7,
> groupId=hercules.sink.elastic.legacy_logs_elk_c2] Attempt to heartbeat failed
> since group is rebalancing
> 2019-03-10T03:17:13.235Z [pool-4-thread-4] INFO
> o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-4,
> groupId=hercules.sink.elastic.legacy_logs_elk_c2] Successfully joined group
> with generation -1
> {noformat}
> But one consumer (pool-4-thread-4) got strange generation -1 (see last log
> record from above).
> Further log records in attached log file.
> Finally, 15 consumers successfully rejoined. But consumer with thread
> {{pool-4-thread-4}} didn't rejoin:
> {noformat}
> 2019-03-10T03:17:13.355Z [pool-4-thread-4] ERROR
> r.k.vostok.hercules.sink.SimpleSink - Unspecified exception has been acquired
> java.lang.IllegalStateException: Coordinator selected invalid assignment
> protocol: null
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:241)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:422)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:352)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:337)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:333)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1175)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1154)
> at ru.kontur.vostok.hercules.sink.Sink.poll(Sink.java:152)
> at ru.kontur.vostok.hercules.sink.SimpleSink.run(SimpleSink.java:70)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-03-10T03:17:13.360Z [pool-4-thread-4] ERROR
> r.k.vostok.hercules.sink.SimpleSink - Unspecified exception has been acquired
> java.lang.IllegalStateException: Coordinator selected invalid assignment
> protocol: null
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:241)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:422)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:352)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:337)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:333)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1175)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1154)
> at ru.kontur.vostok.hercules.sink.Sink.poll(Sink.java:152)
> at ru.kontur.vostok.hercules.sink.SimpleSink.run(SimpleSink.java:70)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)}}
> {noformat}
> It is important to note, that {{KafkaConsumer.coordinator.joinFuture}} is not
> null and succeeded, but {{ConsumerCoordinator}} cannot perform
> {{resetJoinGroupFuture()}} due to exception was thrown from
> {{onJoinComplete()}}:
> {code:java}
> if (future.succeeded()) {
> // Duplicate the buffer in case `onJoinComplete` does not
> complete and needs to be retried.
> ByteBuffer memberAssignment = future.value().duplicate();
> onJoinComplete(generation.generationId, generation.memberId,
> generation.protocol, memberAssignment);
> // We reset the join group future only after the completion
> callback returns. This ensures
> // that if the callback is woken up, we will retry it on the
> next joinGroupIfNeeded.
> resetJoinGroupFuture();
> needsJoinPrepare = true;
> }
> {code}
> If I understood correctly, the generation was changed to {{NO_GENERATION}} in
> another thread by one of CoordinatorResponseHandlers.
>
> [^consumer-rejoin-fail.log]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
