SHAUN WEBB wrote: > > Further to my email below. > > I have a data library that contains many ~117Gb of NGS data, > uploaded via file system path. This library was always slow to open > (about 10s) but now takes several minutes or not at all. > > Thanks for any help on this. I am still experiencing a memory leak > that I can't pinpoint and it is only dissipated by restarting the > server. At the moment my debugging level is set to INFO. Is there > anything I can change in the universe file to try to trace this.
Two things that would help here: Are there other users doing things in Galaxy at this time? It'd help to be able to determine exactly what is triggering this. Also, if you set 'use_heartbeat = True' in universe_wsgi.ini, this will dump the call stack every 30 seconds to the file 'heartbeat.log' (and 'heartbeat.log.nonsleeping'). This should reveal where the thread(s) are hung. --nate > > Thanks! > Shaun > > > ----- Forwarded message from swe...@staffmail.ed.ac.uk ----- > Date: Wed, 09 Mar 2011 10:15:13 +0000 > From: SHAUN WEBB <swe...@staffmail.ed.ac.uk> > Subject: Galaxy process > To: galaxy dev <galaxy-...@bx.psu.edu> > > > > Hi, > > since making the last update I have found some new warnings in my > paster.log, it also seems as though the galaxy process starts to > gather memory and eventually hang (35% of 64G memory). > > I've posted the entries below. > > If anyone could help me understand what is going on that would be great. > > Thanks. > Shaun Webb > > > > paste.httpserver.ThreadPool INFO 2011-03-09 09:49:49,962 No idle > tasks, and only 0 busy tasks; adding 5 more workers > paste.httpserver.ThreadPool INFO 2011-03-09 09:49:58,754 No idle > tasks, and only 4 busy tasks; adding 1 more workers > paste.httpserver.ThreadPool INFO 2011-03-09 09:51:47,301 Culling 6 > extra workers (5 idle workers present) > paste.httpserver.ThreadPool INFO 2011-03-09 09:55:17,163 No idle > tasks, and only 0 busy tasks; adding 5 more workers > 129.215.14.72 - - [09/Mar/2011:09:48:40 +0100] "GET /history > HTTP/1.1" 500 - "http://bifx-core.bio.ed.ac.uk:8080/"; "Mozilla/5.0 > (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.13) Gecko/20101203 > Firefox/3.6.13 (.NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR > 3.0.4506.2152; .NET CLR 3.5.30729)" > paste.httpserver.ThreadPool INFO 2011-03-09 10:13:16,956 Culling 5 > extra workers (7 idle workers present) > 212.183.140.59 - - [09/Mar/2011:10:13:17 +0100] "GET / HTTP/1.1" 200 > - "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) > AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.102 > Safari/534.13" > paste.httpserver.ThreadPool INFO 2011-03-09 10:14:35,715 No idle > tasks, and only 2 busy tasks; adding 3 more workers > paste.httpserver.ThreadPool WARNING 2011-03-09 10:15:15,104 Thread > 140283224094464 hung (working on task for 3096 seconds) > ---------------------------------------- > Exception happened during processing of request from ('212.183.140.59', 10871) > Traceback (most recent call last): > File "/usr/lib/python2.6/SocketServer.py", line 281, in > _handle_request_noblock > self.process_request(request, client_address) > File > "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", > line 1037, in process_request > lambda: self.process_request_in_thread(request, client_address)) > File > "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", > line 617, in add_task > self.kill_hung_threads() > File > "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", > line 778, in kill_hung_threads > self.kill_worker(worker.thread_id) > File > "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", > line 705, in kill_worker > killthread.async_raise(thread_id, SystemExit) > File > "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/util/killthread.py", > line 22, in async_raise > raise ValueError("invalid thread id") > ValueError: invalid thread id > ---------------------------------------- > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > > ----- End forwarded message ----- > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
