[Wikidata-bugs] [Maniphest] [Commented On] T48643: [Story] Dispatching via delayed jobs (instead of cron script)

[Addshore]

Addshore added a comment.


  The estimation comes down to, do we:
  
  1. Just more the dispatching mechanism over to jobs, using the same or very 
similar logic that we currently have.
- Probably not the most efficient dispatching logic
- Least effort to get there
- Solves most of the issues outlined in T48643#5336132 

  2. Change the way dispatching works while we move over to jobs
- More work & time
- Probably makes dispatching faster, more efficient, more reliable and 
easier to understand
- Best use of job queue for dispatching
- Also solves the issues outlined in T48643#5336132 

  
  If number 2 then we also have to decide how exactly we will be changing the 
logic.
  
  In T48643#1522606 , @hoo 
wrote:
  
  > Do we want one job per edit or how exactly is this supposed to look? 
Wrapping the current dispatching mechanism in jobs doesn't really sound like a 
good idea to me.
  
  The "Idea" in the description of this ticket currently talks about number 1 
and keeping the same logic but making the job queue trigger it.
  I'm on @hoo side here and don't think this would be the best thing to do.

TASK DETAIL
  https://phabricator.wikimedia.org/T48643

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: alaa_wmde, Ladsgroup, Joe, Addshore, Aklapper, Tobi_WMDE_SW, JanZerebecki, 
Wikidata-bugs, Abraham, Nemo_bis, Denny, aude, Ricordisamoa, Lydia_Pintscher, 
daniel, hoo, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T48643: [Story] Dispatching via delayed jobs (instead of cron script)

[alaa_wmde]

alaa_wmde added a comment.


  @hoo would be able to provide more information from your own experience with 
this, so that we can have a better estimation for it.

TASK DETAIL
  https://phabricator.wikimedia.org/T48643

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde
Cc: alaa_wmde, Ladsgroup, Joe, Addshore, Aklapper, Tobi_WMDE_SW, JanZerebecki, 
Wikidata-bugs, Abraham, Nemo_bis, Denny, aude, Ricordisamoa, Lydia_Pintscher, 
daniel, hoo, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T48643: [Story] Dispatching via delayed jobs (instead of cron script)

[alaa_wmde]

alaa_wmde added a comment.


  We would like to understand the amount of effort needed to do this.

TASK DETAIL
  https://phabricator.wikimedia.org/T48643

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde
Cc: alaa_wmde, Ladsgroup, Joe, Addshore, Aklapper, Tobi_WMDE_SW, JanZerebecki, 
Wikidata-bugs, Abraham, Nemo_bis, Denny, aude, Ricordisamoa, Lydia_Pintscher, 
daniel, hoo, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T48643: [Story] Dispatching via delayed jobs (instead of cron script)

[alaa_wmde]

alaa_wmde added a comment.


  In T48643#5364763 , 
@Ladsgroup wrote:
  
  > In T48643#5363240 , 
@alaa_wmde wrote:
  >
  >> This sounds like a nice idea..
  >> In order to be able to order this in proper priority, do we have any 
measurements/hypothesis on what we would solve/gain with this change? or is it 
just anticipating future risks and trying to proactively do something?
  >
  > On gains, This is more about resilience IMO. We have two documented 
incident regarding dispatching 

 but I think there are more. There have been scheduled down times due to reboot 
of mwmaint nodes, etc. All of them would be avoided. I say this might gives us 
another 9 in dipsatching uptime. Would that work for you?
  
  Yeap that's more info to include here, thanks!

TASK DETAIL
  https://phabricator.wikimedia.org/T48643

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde
Cc: alaa_wmde, Ladsgroup, Joe, Addshore, Aklapper, Tobi_WMDE_SW, JanZerebecki, 
Wikidata-bugs, Abraham, Nemo_bis, Denny, aude, Ricordisamoa, Lydia_Pintscher, 
daniel, hoo, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T48643: [Story] Dispatching via delayed jobs (instead of cron script)

[Ladsgroup]

Ladsgroup added a comment.


  In T48643#5363240 , 
@alaa_wmde wrote:
  
  > This sounds like a nice idea..
  > In order to be able to order this in proper priority, do we have any 
measurements/hypothesis on what we would solve/gain with this change? or is it 
just anticipating future risks and trying to proactively do something?
  
  On gains, This is more about resilience IMO. We have two documented incident 
regarding dispatching 

 but I think there are more. There have been scheduled down times due to reboot 
of mwmaint nodes, etc. All of them would be avoided. I say this might gives us 
another 9 in dipsatching uptime. Would that work for you?

TASK DETAIL
  https://phabricator.wikimedia.org/T48643

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: alaa_wmde, Ladsgroup, Joe, Addshore, Aklapper, Tobi_WMDE_SW, JanZerebecki, 
Wikidata-bugs, Abraham, Nemo_bis, Denny, aude, Ricordisamoa, Lydia_Pintscher, 
daniel, hoo, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T48643: [Story] Dispatching via delayed jobs (instead of cron script)

[alaa_wmde]

alaa_wmde added a comment.


  > - It's a SPOF, if mwmaint1002 node goes down for HW issues, we can't 
dispatch at all. If there's a need to restart the node, dispatching has to stop 
until it's done.
  > - "Noisy neighbor" effect, people run maintenance scripts in the mwmaint 
node, it can be choked to death by other scripts and it can make running 
maintenance scripts impossible by having bugs that eats all of the resources.
  > - The distributed system we designed for this (pulling the wikis using 
three cronjobs, dispatching and picking up basically random + most stalled 
ones). This can use the great infrastructure for jobqueues we have.
  > - Cronjobs are hard to debug, moving them to jobqueue makes it easier to 
debug in logstash.
  
  This sounds like a nice idea..
  
  In order to be able to order this in proper priority, do we have any 
measurements/hypothesis on what we would solve/gain with this change? or is it 
just anticipating future risks and trying to proactively do something?

TASK DETAIL
  https://phabricator.wikimedia.org/T48643

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde
Cc: alaa_wmde, Ladsgroup, Joe, Addshore, Aklapper, Tobi_WMDE_SW, JanZerebecki, 
Wikidata-bugs, Abraham, Nemo_bis, Denny, aude, Ricordisamoa, Lydia_Pintscher, 
daniel, hoo, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs