jreyeshdez commented on issue #15942: URL: https://github.com/apache/druid/issues/15942#issuecomment-2514443944
> The issue is caused because of a change in date format in the leader election config maps that are stored in kubernetes. The steps to fix are as follows: > > `kubectl get cm -n <druid namespace>` > > you should see 2 config maps of the form: > > ``` > <cluster-identifier>-leaderelection-coordinator > <cluster-identifier>-leaderelection-overlord > ``` > > verify the date from your error message appears in the annotations: > > ``` > kubectl describe cm <cluster-identifier>-leaderelection-coordinator -n <druid namespace> > kubectl describe cm <cluster-identifier>-leaderelection-overlord -n <druid namespace> > ``` > > remove these by: > > ``` > kubectl delete cm <cluster-identifier>-leaderelection-coordinator -n <druid namespace> > kubectl delete cm <cluster-identifier>-leaderelection-overlord -n <druid namespace> > ``` > > These will be recreated by the services whcih will then successfully start. Thanks @m17kea for the explanation. I actually attempted your solution awhile ago and unfortunately did not work either. Worth saying I did it through k9s but somehow the config maps were not recreated with updated time. Link of the conversation https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1712231718374279?thread_ts=1708510329.253319&cid=C0309C9L90D In case you cant see above slack link, this is what the message says: ``` just checked and see druid-deployment-dev-leaderelection-coordinator and druid-deployment-dev-leaderelection-overlord both have "acquireTime":"20240228T111729.078Z" and "acquireTime":"20240228T111857.334Z" ``` and ``` well unfortunately it does not work, I upgraded and applied the changes, it failed with same error but I deleted the configmaps but they got re-created back with same acquireTime format even in 29.0.1 ``` I deploy druid via Helm so not sure if there is a different procedure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
