On 14/07/10 12:12, Michael Hudson wrote:
On 14/07/10 11:42, Gary Poster wrote:
On Jul 10, 2010, at 6:57 PM, Gary Poster wrote:
The average request time for of the public xmlrpc server (that is,
requests that match
^https?://xmlrpc\.) just before the last rollout was .21 seconds (see
https://devpad.canonical.com/~stub/ppr/lpnet/daily_2010-07-06_2010-07-07/categories.html).
The past two days, the average request time for these requests was
over 8 seconds
(https://devpad.canonical.com/~stub/ppr/lpnet/daily_2010-07-08_2010-07-09/categories.html
and
https://devpad.canonical.com/~stub/ppr/lpnet/daily_2010-07-09_2010-07-10/categories.html
; click on "Mean time" to sort it to the top).
(1) The regex identifying these as public xmlrpc requests was
incorrect. We'll fix it.
(2) From
https://devpad.canonical.com/~stub/ppr/lpnet/daily_2010-07-13_2010-07-14/top200.html
(these are the top 200 hits, fwiw), it looks like these two URLs are
the culprits:
https://xmlrpc.lp.internal:8097/codeimportscheduler/getJobForMachine
(avg time 8.29 requests * 1066 requests)
https://xmlrpc.lp.internal/codeimportscheduler/finishJobID (avg time
7.27 requests * 428 requests)
Oh. So the thread title is wrong :-)
Neither of these methods really does very much, although finishJob is a
little involved. Can we get a soft-oops type break-down of the queries
they execute?
Ah, some of them are triggering soft timeouts, so that we can see stuff
like https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1655XMLP1
UPDATE CodeImportMachine SET heartbeat=CURRENT_TIMESTAMP AT TIME ZONE
'UTC' WHERE CodeImportMachine.id = %s took 8 seconds in this case. I
think this must be a result of some kind of locking, but I can't really
imagine what as this is the only method I can think of that writes to
this table routine. Maybe many concurrent calls to the same method are
contending with each other? Seems a tad unlikely though.
Cheers,
mwh
_______________________________________________
Mailing list: https://launchpad.net/~launchpad-dev
Post to : [email protected]
Unsubscribe : https://launchpad.net/~launchpad-dev
More help : https://help.launchpad.net/ListHelp
