The larger time range includes the removal of the mysql based parsercache which is the cause of the primary decline and not related to 1.19. The time range of the originally mentioned graph just shows a bit of context before the enwiki deployment up until the time I posted it to irc last night.
A key change as hashar suggests is a good theory but I'm not sure if the hit rate is actually recovering when looking at -24 or -8 hours, more time will tell. This may not be valid since the mysql pcache wasn't re-enabled (from an empty state) long before 1.19 but the rate of selects against it seems to be up day over day. Maybe a new key is consistently fetched before it would ever be set. 2012-02-29T11:24:00+00:00,10288.770238 2012-02-29T12:06:00+00:00,10992.754365 2012-02-29T12:48:00+00:00,11140.606746 2012-03-01T11:54:00+00:00,13912.613492 2012-03-01T12:36:00+00:00,13790.359524 2012-03-01T13:18:00+00:00,14176.010317 (from http://ganglia.wikimedia.org/latest/graph.php?c=MySQL%20pmtpa&h=db40.pmtpa.wmnet&v=15608&m=mysql_com_select&r=custom&z=default&jr=&js=&st=1330622240&cs=2%2F28%2F2012%2010%3A23&ce=3%2F1%2F2012%200%3A56&vl=stmts&ti=mysql_com_select&csv=1 ) On Thursday, March 1, 2012, Antoine Musso wrote: > Le 01/03/12 09:50, Jeroen De Dauw a écrit : > >> Hey, >> >> There's been a slight regression in our parser cache hit rate: >>> http://bit.ly/w6Gy9t >>> >> >> This one is probably more informative for people not aware of the usual >> hit >> rate http://goo.gl/YY80C >> >> Looks to me that the miss rate went up over 500% - is that really just a >> "slight" regression? :) >> > > We really want to use absolute time range: http://bit.ly/A7kcys > > > Anyway, they are absent misses. Probably a key changed somewhere in our > parser that magically invalidated roughly 15% of the parser cache. It seems > to slowly recover afterward. > > > By zooming and making the Y scale start at 50%, the event seems to have > occurred on February 17th just before 6am UTC. > > I have uploaded a screenshot on mediawiki.org : > > > https://www.mediawiki.org/**wiki/File:Parser_cache_hit_**20120217.png<https://www.mediawiki.org/wiki/File:Parser_cache_hit_20120217.png> > > From the wikitech admin logs we have: > > > 07:42 binasher:upgraded mysql on db40 to 5.1.53-facebook-r3753, enabled > innodb_use_purge_thread > ------------------------------**------------------------------**---------- > 05:39 tstarling synchronized wmf-config/InitialiseSettings.**php > 05:38 tstarling synchronized wmf-config/CommonSettings.php > ------------------------------**------------------------------**---------- > 05:35 Tim: on db40: reduced to 10M, should be causing massive delays, but > the site's not down and the purge rate is lower if anything. Going to > disable the mysql parser cache entirely. > ------------------------------**------------------------------**---------- > 05:25 Tim: on db40: purge lag is still increasing at 108 per second, so > reducing innodb_max_purge_lag to 50M > ------------------------------**------------------------------**---------- > 05:21 Tim: on db40: giving the innodb manual the benefit of the doubt and > following its advice, setting innodb_max_purge_lag to 100M, which should > give a delay of 4.5ms > ------------------------------**------------------------------**---------- > 05:13 Tim: killing purgeParserCache.php since it is probably doing more > harm than good > ------------------------------**------------------------------**---------- > 02:43 maplebed: deployed updated thumb_handler.php to ms5 to include > Content-Length in generated images > ------------------------------**------------------------------**---------- > 02:34 logmsgbot: LocalisationUpdate completed (1.19) at Fri Feb 17 > 02:34:32 UTC 2012 > ------------------------------**------------------------------**---------- > https://wikitech.wikimedia.**org/view/Server_admin_log<https://wikitech.wikimedia.org/view/Server_admin_log> > > > " Going to disable the mysql parser cache entirely." > > Seems to have been reenabled on Feb 29th at 00:20: > > 00:20 Tim: reimported schema files on db40 and re-enabled mysql parser > cache > > > -- > Antoine "hashar" Musso > > > > > ______________________________**_________________ > Wikitech-l mailing list > [email protected] > https://lists.wikimedia.org/**mailman/listinfo/wikitech-l<https://lists.wikimedia.org/mailman/listinfo/wikitech-l> > _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
