On 11-03-14 05:14 AM, Tim Starling wrote: > On 14/03/11 19:52, Domas Mituzas wrote: >>> For now I have added a sleep() to the code to limit invalidations to >>> 100 pages per second per job runner process. >> Mass invalidations will create MJ issue in the long tail though... Need >> poolcounter ;-) > That's why Platonides wrote it for us. Now we just have to package it > and deploy it. > > There was a CPU spike after my change, but that was probably because > 100 invalidations per second was still too fast, rather than a Michael > Jackson type of problem. > > On IRC, Daniel Friesen suggested pushing new HTML into the parser > cache from the job queue instead of invalidating. Maybe after we move > the parser cache to disk, that will be a good option. > > -- Tim Starling To avoid pushing frequently used entries out, I also thought of the idea of only replacing caches that exist. ie: If it was parser cached, replace the parser cache with a new version, if not, leave it alone so we don't flush frequently used entries with infrequently used ones. Unless cache replacements still flush a lot of frequently used entries that option might have most of the advantages of updating caches from the job queue while avoiding the biggest pitfalls.
There is also the option of trying to distribute these timewise and break them up into batches so there is time for normal operations to re-cache without doing it all at once. Though that project might be a little more involved since it'll depart from the normal operation of the job queue and have jobs that shouldn't be run off the queue right away. I did always love the idea of a job daemon though. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name] _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
