https://bugzilla.wikimedia.org/show_bug.cgi?id=63248
Bug ID: 63248
Summary: search pageimages feature causes significant extra
infrastructure load
Product: MobileFrontend
Version: unspecified
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: Unprioritized
Component: General/Unknown
Assignee: wikibugs-l@lists.wikimedia.org
Reporter: fai...@wikimedia.org
CC: aricha...@wikimedia.org, jgon...@wikimedia.org,
jrob...@wikimedia.org, kw...@wikimedia.org,
maxsem.w...@gmail.com, rkald...@wikimedia.org
Web browser: ---
Mobile Platform: ---
This is the top 50 misses from a random Varnish mobile frontend:
110.52 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=&pithumbsize=80&pilimit=50
33.80 TxURL
/wiki/List_of_NCAA_Men's_Division_I_Basketball_champions
10.76 TxURL /wiki/NCAA_Men's_Division_I_Basketball_Championship
6.56 TxURL /wiki/Special:Search?search=
5.65 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Race+and+ethnicity+in+the+United+States+Census%7CRussia%7CRomanization%7CRock+music%7CRomania%
5.54 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Rick+Ross%7CRicky+Martin%7CRick+Rubin%7CRick+Perry%7CRick+Leach%7CRicky+Gervais%7CRick+Wakeman
4.80 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Mollusca%7CMoth%7CMexico%7CMidfielder%7CMarriage%7CMember+of+Parliament%7CMajor+League+Basebal
4.68 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Keyboard+instrument%7CKenya%7CKentucky%7CKerala%7CKorea%7CKansas%7CKent%7CKolkata%7CKorean+War
4.58 TxURL
/wiki/Special:RecordImpression?result=hide&reason=empty&country=XX&uselang=en&project=wikipedia&db=enwiki&bucket=1&anonymous=false&device=android
4.09 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=John+Cage%7CJohn+Calvin%7CJohn+Cale%7CJohn+Carpenter%7CJohn+Cassavetes%7CJohn+Carradine%7CJohn
3.48 TxURL
/w/index.php?title=Special:UserLogin&returnto=Special%3AUploads
3.18 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Brazil%7CBakhsh%7CBBC%7CBillboard+(magazine)%7CBass+guitar%7CBelgium%7CBoston%7CBerlin%7CBaske
3.18 TxURL
/w/index.php?title=Wikispecies_talk:Village_Pump/Archive_17&action=edit&redlink=1
3.17 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=La+La%7CLA+Law%7CLa-La+Land+Records%7CLA+Lakers%7CLa+La+Anthony%7CLa+Laguna%7CLa+La+La+(Naught
3.12 TxURL /wiki/Special:CentralAutoLogin/start?type=1x1
2.90 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Anatolia%7CAnarchism%7CAnatomical+terms+of+location%7CAnaheim%2C+California%7CAnatomy%7CAnahei
2.80 TxURL /wiki/2012_NCAA_Men's_Division_I_Basketball_Tournament
2.74 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Thumba%7CThumba+Equatorial+Rocket+Launching+Station%7CThumbay+Moideen%7CThumbavi%7CThumbay+Gro
2.66 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=The+New+York+Times%7CTurkey%7CTexas%7CThe+Guardian%7CToronto%7CTaiwan%7CThailand%7CTokyo%7CThe
2.62 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Estados+Unidos%7CEspa%C3%B1a%7CEspecie%7CEstado+de+los+Estados+Unidos%7CEuropa%7CEndopterygota
2.54 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=X-Men%7CX-Men+(TV+series)%7CX-Men%3A+The+Last+Stand%7CX-Men+(film)%7CX-Men%3A+First+Class%7CX-
2.53 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=S%7CSMAP%7CSuica%7CSD%E3%83%A1%E3%83%A2%E3%83%AA%E3%83%BC%E3%82%AB%E3%83%BC%E3%83%89%7CSOD%E3%
2.49 TxURL /
2.43 TxURL /wiki/Special:Random
2.39 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=District+1%7CDistrict+13%7CDistrict+13%3A+Ultimatum%7CDistrict+1%2C+D%C3%BCsseldorf%7CDistrict
2.38 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Boston%7CBosnia+and+Herzegovina%7CBolivia%7CBoxing%7CBollywood%7CBoston+Red+Sox%7CBob+Dylan%7C
2.32 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Nicholas%7CNicholas+II+of+Russia%7CNicholas+I+of+Russia%7CNicholas+Briggs%7CNicholas+Monroe%7C
2.29 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=John+Calipari&pithumbsize=80&pilimit=50
2.14 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Andrew+Jackson%7CAndrew+Lloyd+Webber%7CAndrew+Johnson%7CAndrew+Carnegie%7CAndrew+the+Apostle%7
2.11 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Louisiana%7CLouisville%2C+Kentucky%7CLouis+XIV+of+France%7CLouisiana+State+University%7CLouis+
2.04 TxURL /wiki/Pandora's_box
2.01 TxURL /w/index.php?title=Special:RecentChanges&feed=atom
2.00 TxURL
/w/api.php?action=featuredfeed&feed=onthisday&feedformat=atom
1.99 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Stan+(South+Park)&pithumbsize=80&pilimit=50
1.97 TxURL
/w/api.php?hidebots=1&days=7&limit=50&hidewikidata=1&action=feedrecentchanges&feedformat=atom
1.95 TxURL
/w/index.php?title=%D9%88%DB%8C%DA%98%D9%87:%D8%AA%D8%BA%DB%8C%DB%8C%D8%B1%D8%A7%D8%AA_%D8%A7%D8%AE%DB%8C%D8%B1&feed=atom
1.95 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Jeopardy!%7CJeopardy!+broadcast+information%7CJeopardy!+Tournament+of+Champions%7CJeopardy!+Ul
1.95 TxURL
/wiki/Special:BannerRandom?uselang=es&sitename=Wikipedia&project=wikipedia&anonymous=false&bucket=1&country=XX&device=android&slot=8
1.95 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Kyrgyzstan%7CKyoto%7CKylie+Minogue%7CKyushu%7CKyiv%7CKyushu+Railway+Company%7CKyoto+Protocol%7
1.94 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Lieutenant%7CLie%7CLieutenant+colonel%7CLieutenant+general%7CLi%C3%A8ge%7CLiechtenstein%7CLieu
1.92 TxURL /wiki/Asia's_Next_Top_Model_(cycle_2)
1.92 TxURL /wiki/1996_NCAA_Men's_Division_I_Basketball_Tournament
1.92 TxURL /wiki/Bah%C3%A1'%C3%AD_Faith
1.84 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=CuSO4&pithumbsize=80&pilimit=50
1.82 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Christian%7CChristianity%7CChristmas%7CChristchurch%7CChrist+Church%2C+Oxford%7CChristina+Agui
1.80 TxURL /wiki/Harry_Potter_and_the_Philosopher's_Stone_(film)
1.80 TxURL /wiki/Ballon_d'Or_(1956%E2%80%932009)
1.78 TxURL
/w/api.php?format=json&action=query&prop=pageimages&titles=Gumbo%7CGumby%7CGumball+3000%7CGumboot%7CGumboro%7CGumbo+fil%C3%A9%7CGumbranch%7CGumbasia%7CGu
This is just one Varnish server *backend* — just compare the hits that / is
getting (at place 23) to get a sense of the scale. The Varnish req/s graphs are
broken on the yearly scale right now, but I'd bet pageimages is a significant
portion of our requests per second based on the above
Even further down the stack, the API appservers network usage (and respective
load) has tripled over the last two months:
http://ganglia.wikimedia.org/latest/graph.php?r=year&z=xlarge&c=API+application+servers+eqiad&m=ap_busy_workers&s=by+name&mc=2&g=network_report
and RPS have risen by 40-50% since January:
http://ganglia.wikimedia.org/latest/stacked.php?m=ap_rps&c=API%20application%20servers%20eqiad&r=year&st=1396059695&host_regex=
Correct me if I'm wrong but from the URLs it looks like the pageimages search
feature that got moved from beta to stable with commit
I1c657fd46a2b5be4f27aa508a5cc0d946d6b98a8 (Story 1462: Move new search overlay
to stable). The commit was merged in 1.23wmf12 which was deployed between Jan
30th (group 0) & Feb 6th (group 2), so it looks like it correlates well with
the graphs above.
Besides infrastructure problems, I also imagine that this is killing UA
performance/total page load time for search on mobile. I'll Cc Ori.
Moreover, I suppose that this feature creates yet another thumbnail size that
we didn't have before which could potentially be problematic (not so in bytes,
as they're tiny, but in object count). I haven't noticed any significant
difference in object creation count nor bytes stored in Swift, however, so
that's probably OK for now. It'd be nice to have had some estimates & advance
warning about this beforehand, though.
I'd revert this commit while you investigate, but the commit is
20 files changed, 103 insertions(+), 319 deletions(-)
and I see no obvious config tweak that would allow me to turn it off
gracefully. It may be a good idea to do such beta->stable rollouts in the
future in two steps, with a small turning-the-knob commit at the end so that
they are more easily revertable in emergencies or severe problems.
--
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.
_______________________________________________
Wikibugs-l mailing list
Wikibugs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
