JMeybohm added a comment.
> To be discussed with service ops: > > - Investigate and address the reasons why after a node failure k8s did not fulfill its promise of making sure that the rdf-streaming-updater deployment have 6 working replicas The problem was more that the node did not really fail (to it's complete extend). It was heavily overloaded (for an unknown reason) and that's potentially why containers/processed running there seemed dead. But from K8s perspective the Pods where still running and a new pod was scheduled as soon as I power cycled the node (e.g. K8s was able to detect a mismatch in desired end existing replicas). > - If the above is not possible could we mitigate this problem by over-allocating resources (increase the number of replicas) to the deployment to increase the chances of proper recovery if this situation happens again? If that makes sense from your POV you could do that ofc. I can't speak on how problematic this situation was compared to the potential waste of resources another pod means. But if the current workload is already maxing out the capacity of the 6 replicas you have, maybe bumping that to 7 might be smart anyways to account for peaks? In T301147#7821422 <https://phabricator.wikimedia.org/T301147#7821422>, @dcausse wrote: > @JMeybohm do you see any additional action items that would improve the resilience of k8s in such scenario? Unfortunately we don't have any data on what went wrong on the node. I think T277876 <https://phabricator.wikimedia.org/T277876> would be a step in the right direction but I also doubt it would have fully prevented this issue (ultimately I can't say). TASK DETAIL https://phabricator.wikimedia.org/T301147 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JMeybohm Cc: elukey, akosiaris, Gehel, RKemper, bking, toan, Addshore, JMeybohm, Michael, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list -- [email protected] To unsubscribe send an email to [email protected]
