Hi, > On Tue, Mar 21, 2017 at 11:11 AM, Adi Kriegisch <[email protected]> wrote: >> >> Hi! >> >> > Sometimes the performance of our Kallithea instance takes a hit, such that >> > page loading can take many many seconds, and it is unclear what causes >> > this. >> > >> > Could you suggest ways to investigate this? >> Step 1: get data. :-) >> >> Actually one of the reasons I use uwsgi for deployment is that I get >> execution >> times 'for free': 'generated 309 bytes in 268 msecs' or 'generated 436 bytes >> in 56 msecs' including the uri and all kinds of useful information. >> > That kind of monitoring is indeed what we do now, with uwsgi: > {address space usage: 1769336832 bytes/1687MB} {rss usage: 1298124800 > bytes/1237MB} [pid: 52669|app: 0|req: 349/9564] 1 > 35.252.28.229 () {40 vars in 1686 bytes} [Wed Mar 22 07:39:38 2017] GET > /review/ext/cvpsw-review/pull-request/15006/_/c > vp80 => generated 76111 bytes in 924 msecs (HTTP/1.1 200) 5 headers in 149 > bytes (1 switches on core 0) > {address space usage: 1687105536 bytes/1608MB} {rss usage: 1218387968 > bytes/1161MB} [pid: 49555|app: 0|req: 209/9565] > 35.252.28.229 () {42 vars in 1591 bytes} [Wed Mar 22 07:39:40 2017] GET > /js/graph.js?ver=0.2.1 > => generated 0 bytes in 156 msecs (HTTP/1.1 304) 3 headers in 98 bytes (1 > switches on core 0) > 2017-03-22 07:40:09.168 INFO [kallithea.RequestWrapper] IP: 135.224.206.40 > Request to /review/ms/sw-review/branches-tags time: 66.745s > > But it are these kind of peeks we want to understand on what Kallithea was > exactly waiting upon, is it pure disk/io, is it a qeury to the database that > took long (postgresql with monitoring with pg_activity) > Is there anything Kallithea debug info or pythong profiler can provide us > (without overloading kallithea itself with all the debug) that we today are > not aware of? > > Our database is 'only' 270MB big, so besides inefficient queries, I cannot > understand if it would be database access that is causing hickups of more > then a minute. Either we configured our database wrong, or I would think > postgresql can keep 270MB in memory (we might need to force it to keep it in > memory, even if other processes consume alot of memory), the server has 125GB > of memory, with a 'free' count (even with caches) of 20GB (let me be clear, > the sever is used for other applications as well). > > What I do see as queries running long on the database are these, (But I > haven't correlated these directly with the above peeks yet (but it could well > be).) > 47362 kallithea kallithea 127.0.0.1 0.0 0.0 0.00B > 0.00B 00:47.10 N N SELECT cache_invalidation.cache_id AS > cache_invalidation_cache_id, cache_invalidation.cache_key AS > cache_invalidation_cache_key, cache_invalidation.cache_args AS > cache_invalidation_cache_args, cache_invalidation.cache_active AS > cache_invalidation_cache_active FROM cache_invalidation WHERE > cache_invalidation.cache_key = 'devws048-32241review/ms/sw-review' > > As you see, that query runs for 47 seconds at the moment I catch it. > > Anybody who can explain what this 'triggers' in the database, as that table > itself is quite small. > public | cache_invalidation | > table | kallithea | 992 kB | > public | cache_invalidation_cache_id_seq | > sequence | kallithea | 8192 bytes | > > I haven' looked googled yet what this could mean, so forgive me if I'm asking > basic questions ;-)
As far as I understand it now, these queries are the result of updated repositories from 'outside' kallithea and the accompanied repo-'cache invalidation' trigger. If this is really hitting us that would mean: * requests to kallithea webgui have to wait until these queries are over? * we need to find a way to have these repos updated without the need of the cache invalidation, so instead of doing an 'outside pull', would a 'push to kallithea' prevent the need of invalidating the cache? Br, Jan > > >> My setup is similar to this: >> https://lists.sfconservancy.org/pipermail/kallithea-general/2015q1/000130.html >> >> You may even monitor execution times or graph them or do some statistics work >> on them or -- simply put -- correlate higher latency and execution time with >> system or I/O load (even simple tools like sysstat/sadc/sar are sufficient to >> gain insights in this regard). > > > I will already have a look at these simple ones, I must admit I didn't use > these until now (I did use iowait/top/...) > http://www.thegeekstuff.com/2011/03/sar-examples/?utm_source=feedburner > > >> >> >> Hope, this helps... >> >> All the best, >> Adi _______________________________________________ kallithea-general mailing list [email protected] https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
