chandrakanthkannam opened a new issue #12192: URL: https://github.com/apache/druid/issues/12192
### Affected Version Docker image: apache/druid:0.21.0 ### Description Please include as much detailed information about the problem as possible. - Running druid containers on EKS cluster, and the cluster has coordinator and overlord running as individual pods, the issue is after the cluster setup is completed and for some reason, overlord pod crashes(example: pod gets evicted due to resource limitation) and a new pod comes up, by the time new pod comes online coordinator is taking over ``` of role[overlord] went offline. 2022-01-22T23:19:00,746 INFO [LeaderSelector[/druid/overlord/_OVERLORD]] org.apache.druid.indexing.overlord.TaskMaster - By the power of Grayskull, I have the power! 2022-01-22T23:19:00,806 INFO [LeaderSelector[/druid/overlord/_OVERLORD]] org.apache.druid.indexing.overlord.TaskLockbox - Synced 0 locks for 0 activeTasks from storage (0 locks ignored). ``` and I see coordinator pod as a leader here: `http://localhost:8081/druid/coordinator/v1/leader` when the actual overlord pod comes up it is announcing itself as coordinator instead ``` INFO [main] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Announced self [{"druidNode":{"service":"druid/coordinator","host":"XXXXXXX","bindOnHost":false,"plaintextPort":8081,"port":-1,"tlsPort":-1,"enablePlaintextPort":true,"enableTlsPort":false},"nodeType":"overlord","services":{}}]. ``` - workaround: when this happens we are forced to kill both coordinator and overlord pods and start from scratch again to make themselves in the right state Please help me in fixing this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
