Re: [Wikitech-l] JobQueue on Redis initial test deploy today
On 16/04/13 12:02, MZMcBride wrote: Greg Grossmeier wrote: quote name=Greg Grossmeier date=2013-04-10 time=12:49:25 -0700 If all goes well with the testing, we should be able to switch over fully by this coming Monday. All went well, we're now fully on Redis. You can rejoice now. Hi Greg. I looked at your original e-mail and this e-mail, but I'm still not sure what Redis is or how it relates to the MediaWiki job queue. Redis http://redis.io/ is an in-memory key/value store, a bit like memcached except with more features. We've been using it as a memcached replacement for session storage since the eqiad switchover in January, because it has a replication feature which can be used to synchronise data between the two data centres. It allowed us to switch from Tampa to Ashburn without logging everyone out. It's designed more as a persistent store than a cache. Memcached still wins for simple unreliable object caching, so we're still using memcached for that. We previously stored the MW job queue in MySQL. This gave us lots of useful features, like replication and indexing for duplicate removal, but it has often been hard to manage the performance implications of the high insert rate. Among its many features, Redis embeds a Lua interpreter on the server side. The new Redis job queue class provides a rich feature set superior to the MySQL job queue, primarily by the use of several server-side Lua scripts which provide high-level job queue functions. -- Tim Starling ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] JobQueue on Redis initial test deploy today
On Tue, Apr 16, 2013 at 12:14 AM, Tim Starling tstarl...@wikimedia.orgwrote: We've been using it as a memcached replacement for session storage since the eqiad switchover in January, because it has a replication feature which can be used to synchronise data between the two data centres. It allowed us to switch from Tampa to Ashburn without logging everyone out. It's designed more as a persistent store than a cache. Memcached still wins for simple unreliable object caching, so we're still using memcached for that. We previously stored the MW job queue in MySQL. This gave us lots of useful features, like replication and indexing for duplicate removal, but it has often been hard to manage the performance implications of the high insert rate. Among its many features, Redis embeds a Lua interpreter on the server side. The new Redis job queue class provides a rich feature set superior to the MySQL job queue, primarily by the use of several server-side Lua scripts which provide high-level job queue functions. I've taken the liberty of adapting this explanation and my own additions for the Redis page on MediaWiki.org Steven ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] JobQueue on Redis initial test deploy today
Steven Walling wrote: On Tue, Apr 16, 2013 at 12:14 AM, Tim Starling tstarl...@wikimedia.orgwrote: We've been using it as a memcached replacement for session storage since the eqiad switchover in January, because it has a replication feature which can be used to synchronise data between the two data centres. It allowed us to switch from Tampa to Ashburn without logging everyone out. It's designed more as a persistent store than a cache. Memcached still wins for simple unreliable object caching, so we're still using memcached for that. We previously stored the MW job queue in MySQL. This gave us lots of useful features, like replication and indexing for duplicate removal, but it has often been hard to manage the performance implications of the high insert rate. Among its many features, Redis embeds a Lua interpreter on the server side. The new Redis job queue class provides a rich feature set superior to the MySQL job queue, primarily by the use of several server-side Lua scripts which provide high-level job queue functions. I've taken the liberty of adapting this explanation and my own additions for the Redis page on MediaWiki.org Thank you both. :-) I'll try to help out with that MediaWiki.org page as well (and perhaps add some pointers from Wikitech... two wikis blergh). I had no idea about Redis being used for user sessions. That's neat. MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] JobQueue on Redis initial test deploy today
quote name=Greg Grossmeier date=2013-04-10 time=12:49:25 -0700 If all goes well with the testing, we should be able to switch over fully by this coming Monday. All went well, we're now fully on Redis. You can rejoice now. Greg -- | Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @gregA18D 1138 8E47 FAC8 1C7D | ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] JobQueue on Redis initial test deploy today
Greg Grossmeier wrote: quote name=Greg Grossmeier date=2013-04-10 time=12:49:25 -0700 If all goes well with the testing, we should be able to switch over fully by this coming Monday. All went well, we're now fully on Redis. You can rejoice now. Hi Greg. I looked at your original e-mail and this e-mail, but I'm still not sure what Redis is or how it relates to the MediaWiki job queue. I looked for more information at https://wikitech.wikimedia.org/wiki/Redis, https://wikitech.wikimedia.org/wiki/Redis_JobQueue, and https://wikitech.wikimedia.org/wiki/JobQueue, but didn't find anything. I did find https://wikitech.wikimedia.org/wiki/Job_queue, but it makes no mention of Redis (I think that page is a bit out-of-date at this point). Are there docs somewhere for those interested? (Apologies if docs have been posted to this mailing list and I've simply missed them.) (H. Just before hitting send, I thought to check mediawiki.org. There are some notes at https://www.mediawiki.org/wiki/Redis, but I'm not sure if these relate to Redis/MediaWiki generally or Wikimedia wikis specifically.) MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] JobQueue on Redis initial test deploy today
On Monday, April 15, 2013 at 7:02 PM, MZMcBride wrote: (H. Just before hitting send, I thought to check mediawiki.org (http://mediawiki.org). There are some notes at https://www.mediawiki.org/wiki/Redis, but I'm not sure if these relate to Redis/MediaWiki generally or Wikimedia wikis specifically.) Oh, oops. While the links may be of interest, the lead is incorrect. Our analytics stuff doesn't use redis. We tried it for a while and decided it wasn't a good fit. It is used for a number of other things, as you mentioned. -- Ori Livneh ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] JobQueue on Redis initial test deploy today
Hiya MZ, quote name=MZMcBride date=2013-04-15 time=22:02:03 -0400 I looked at your original e-mail and this e-mail, but I'm still not sure what Redis is or how it relates to the MediaWiki job queue. I looked for more information at [snip lots of places] Are there docs somewhere for those interested? (Apologies if docs have been posted to this mailing list and I've simply missed them.) Not really, no (at least, not that I know of). This is a part of the general site performance improvement work, tracked under this project in our monthly reports: https://www.mediawiki.org/wiki/Site_performance_and_architecture You can see the update for March was: JobQueueRedis merged, JobQueueAggregatorRedis merged and deployed. These improvements to the jobqueue should help site performance (see Tech Operations section of the monthly report). The one for April will be something like Migrated all jobs to the Redis-based JobQueue for the win. The reason I sent out these two messages is because while general performance improvements aren't always big events, this one was a blocker for some further work on a couple projects; notably WikiData and Echo/Notifications (from the Editor Engagement team). Those two make use of the JobQueue extensively and there were a couple times that the JobQueue feel over in the past couple weeks pre-Redis. Now that we're on Redis things should be much much much more stable (and they seem to be so far). I don't know the plan for documenting the switch over on mediawiki.org/wikitech; Aaron will have a better idea on that. Hope that helps, Greg ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] JobQueue on Redis initial test deploy today
Sending this out because I know many of your are wondering about the status of the Redis JobQueue work. This happened today: 18:24 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Enabled use of redis for null test jobs' What that means is that null test jobs will be sent to the new Redis-based JobQueue. Aaron will also begin sending some real jobs to the Redis-based queue later today and monitor them. If all goes well with the testing, we should be able to switch over fully by this coming Monday. Greg -- | Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @gregA18D 1138 8E47 FAC8 1C7D | ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] JobQueue
On Mar 23, 2013 12:10 AM, John phoenixoverr...@gmail.com wrote: I know Aaron has spent a lot of time on the job queue. But I have several observations and would like some feedback. The current workers apparently select jobs from the queue at random. A FIFO method would make far more sense. We have some jobs that can sit there in the queue for extended periods of time, while others added after that point may get completed in mere few minutes. Well random is not the only method. There is also timestamp based and FIFO. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] JobQueue
Don't forget about the potentials of a priority based queue! (That being said I actually have no idea what goes into our job queues; so can't say if there's good candidates for priority based queuing.) ~Matt Walker Wikimedia Foundation Fundraising Technology Team On Sat, Mar 23, 2013 at 6:30 AM, Tyler Romeo tylerro...@gmail.com wrote: On Mar 23, 2013 12:10 AM, John phoenixoverr...@gmail.com wrote: I know Aaron has spent a lot of time on the job queue. But I have several observations and would like some feedback. The current workers apparently select jobs from the queue at random. A FIFO method would make far more sense. We have some jobs that can sit there in the queue for extended periods of time, while others added after that point may get completed in mere few minutes. Well random is not the only method. There is also timestamp based and FIFO. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] JobQueue
AFAIK we already do that for some entries, Reedy will know more, I will let you investigate his mind for more knowledge on that. On Sun, Mar 24, 2013 at 7:27 AM, Matthew Walker mwal...@wikimedia.org wrote: Don't forget about the potentials of a priority based queue! (That being said I actually have no idea what goes into our job queues; so can't say if there's good candidates for priority based queuing.) ~Matt Walker Wikimedia Foundation Fundraising Technology Team On Sat, Mar 23, 2013 at 6:30 AM, Tyler Romeo tylerro...@gmail.com wrote: On Mar 23, 2013 12:10 AM, John phoenixoverr...@gmail.com wrote: I know Aaron has spent a lot of time on the job queue. But I have several observations and would like some feedback. The current workers apparently select jobs from the queue at random. A FIFO method would make far more sense. We have some jobs that can sit there in the queue for extended periods of time, while others added after that point may get completed in mere few minutes. Well random is not the only method. There is also timestamp based and FIFO. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] JobQueue
I know Aaron has spent a lot of time on the job queue. But I have several observations and would like some feedback. The current workers apparently select jobs from the queue at random. A FIFO method would make far more sense. We have some jobs that can sit there in the queue for extended periods of time, while others added after that point may get completed in mere few minutes. Second exposing job_timestamp via the API should also assist in identifying issues. whether or not some job is being ignored or the particular wiki is just extremely lagged. I am monitoring a template modification to file links that occurred about hours ago, with a job queue between 350,000 and 500,000 items this delay seems excessive. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] JobQueue changes (Re: Some Sort of Notice for Breaking Changes)
On Fri, Mar 8, 2013 at 8:35 AM, Tyler Romeo tylerro...@gmail.com wrote: Is there any way that extension developers can get some sort of notice for breaking changes, e.g., https://gerrit.wikimedia.org/r/50138? Luckily my extension's JobQueue implementation hasn't been merged yet, but if it had I would have no idea that it had been broken by the core. Hi Tyler, Sorry to hear that there might be a problem here. It's been a pet peeve of mine that we seem to be a little too eager to break backwards compatibility in places where it may not be necessary. That said, let's try to avoid a meta-process discussion before we collectively understand the example you are bringing up, and focus on the JobQueue. As near as I can tell from a quick skim of the changeset you're referencing, Aaron's changes here are purely additive. Am I reading this wrong? Is there some other changeset that changes/removes existing interfaces that you meant to reference instead? Rob ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] JobQueue changes (Re: Some Sort of Notice for Breaking Changes)
On Fri, Mar 8, 2013 at 12:18 PM, Rob Lanphier ro...@wikimedia.org wrote: As near as I can tell from a quick skim of the changeset you're referencing, Aaron's changes here are purely additive. Am I reading this wrong? Is there some other changeset that changes/removes existing interfaces that you meant to reference instead? At first glance it seems additive, but the change adds a new abstract method to the JobQueue class, meaning any child class of JobQueue that doesn't have the new method implemented will trigger a fatal error. To make it not breaking, the function would have to have a default implementation in the main JobQueue class. *--* *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerro...@gmail.com ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] JobQueue changes (Re: Some Sort of Notice for Breaking Changes)
Also, after doing a git-blame, I found https://gerrit.wikimedia.org/r/51886, which was also merged today. I could search through the core for other changes like this but it'd require an immense amount of time. *--* *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerro...@gmail.com ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
