| Joe added a comment. |
In T173710#3584505, @Krinkle wrote:In T173710#3583445, @Joe wrote:As a side comment: this is one of the cases where I would've loved to have an elastic environment to run MediaWiki-related applications: I could've spun up 10 instances of jobrunner dedicated to refreshlinks (or, ideally, the system could have done it automagically), for example.
Yep! Very true. When I first read about Borg and whenever I read about similar principles since, the job queue always comes to mind as a great use case. Of course it would benefit app server maintenance too, but the job queue pressure tends to vary more than app server pressure. A Borg-like system would allow us to make the most of the idle time on all (app) servers and gracefully fill it up with job runners.
Of course, that doesn't apply to cases that are limited by a common resource (e.g. database). But the idea is still very attractive. Permanently setting up more job runners remains a difficult calculation for us, because in the end we must prioritise app servers for site availability. On the other hand, given how idle most app servers are most of the time, it seems like a royal waste to not put it to use.
The waste of resources we have in general on our cluster, and the pains we have with jobrunners and esp. videoscaling capacity were my prime motivators for looking into kubernetes, in fact. Even if it lacks the scheduler that borg has, it would be an enormous advantage for us.
Cc: Nikerabbit, Mholloway, Legoktm, ema, Joe, GWicke, Nemo_bis, Andreasmperu, BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Vali.matei, Avner, Lewizho99, Zppix, Maathavan, debt, Gehel, FloNight, Izno, Eevans, mobrovac, Hardikj, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
