JMeybohm added a comment.
In T301147#7821813 <https://phabricator.wikimedia.org/T301147#7821813>, @dcausse wrote: > The additional PODs won't be used as a flink job does not automatically scale so it would be a pure waste of resources (2.5G of reserved mem per additional POD). It would help I guess to improve redundancy in this scenario only if k8s assigns every POD to a distinct machine, in which case even with a single machine misbehaving flink would have enough redundancy to allocate the job to the spare POD. If k8s does do allocation randomly or that there are not enough k8s worker nodes (1 spare POD in our case would mean spreading the PODs over 8 different machines) then it's probably not worth the waste of resources. K8s will try to schedule replicas of one Deployment onto different Nodes by default and we can also force it to do so. But tbh I would not so that in this case as in most of the cases it should be just fine. I expect this situation to be a rare exception (and I probably jinxed that now) as we have not seen it before or happen again. So as long as it's not super critical, I would refrain from trying to optimize the workload for this type of failure. Ultimately this should be taken care of by k8s so we should invest there - especially if should happen again. TASK DETAIL https://phabricator.wikimedia.org/T301147 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JMeybohm Cc: elukey, akosiaris, Gehel, RKemper, bking, toan, Addshore, JMeybohm, Michael, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list -- [email protected] To unsubscribe send an email to [email protected]
