Pmap shows its the heap [EMAIL PROTECTED]:in 8:08pm 64 # pmap 24167 24167: /bin/python /opt/mailman-2.1.9/bin/qrunner --runner=IncomingRunner:5:8 08038000 64K rwx-- [ stack ] 08050000 940K r-x-- /usr/local/stow/Python-2.5.2/bin/python 0814A000 172K rwx-- /usr/local/stow/Python-2.5.2/bin/python 08175000 312388K rwx-- [ heap ] CF210000 64K rwx-- [ anon ] <--many small libs --> total 318300K
Whether its a leak or not - we need to understand why the heap is growing and put a limit on its growth to avoid exausting memory and swapping into oblivion... None of the lists seem too big: [EMAIL PROTECTED]:lists 8:24pm 73 # du -sk */*pck | sort -nr | head | awk '{print $1}' 1392 1240 1152 1096 912 720 464 168 136 112 Researching python heap alloaction.... thanks On 7/1/08 6:14 PM, "Mark Sapiro" <[EMAIL PROTECTED]> wrote: > Fletcher Cocquyt wrote: >> >> Here is the current leaked state since the the cron 13:27 restart only 3 >> hours ago: >> last pid: 20867; load averages: 0.53, 0.47, 0.24 >> 16:04:15 >> 91 processes: 90 sleeping, 1 on cpu >> CPU states: 99.1% idle, 0.3% user, 0.6% kernel, 0.0% iowait, 0.0% swap >> Memory: 1640M real, 77M free, 1509M swap in use, 1699M swap free >> >> PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND >> 24167 mailman 1 59 0 311M 309M sleep 0:28 0.02% python >> 24158 mailman 1 59 0 308M 305M sleep 0:30 0.01% python >> 24169 mailman 1 59 0 303M 301M sleep 0:28 0.01% python >> 24165 mailman 1 59 0 29M 27M sleep 0:09 0.03% python >> 24161 mailman 1 59 0 29M 27M sleep 0:12 0.07% python >> 24164 mailman 1 59 0 28M 26M sleep 0:07 0.01% python >> 24172 mailman 1 59 0 26M 24M sleep 0:04 0.01% python >> 24160 mailman 1 59 0 26M 24M sleep 0:08 0.01% python >> 24162 mailman 1 59 0 26M 23M sleep 0:10 0.01% python >> 24166 mailman 1 59 0 26M 23M sleep 0:04 0.01% python >> 24171 mailman 1 59 0 25M 23M sleep 0:04 0.02% python >> 24163 mailman 1 59 0 24M 22M sleep 0:04 0.01% python >> 24168 mailman 1 59 0 19M 17M sleep 0:03 0.02% python >> 24170 mailman 1 59 0 9516K 6884K sleep 0:01 0.01% python >> 24159 mailman 1 59 0 9500K 6852K sleep 0:00 0.00% python >> >> And the mapping to the runners: >> [EMAIL PROTECTED]:mailman-2.1.11 4:16pm 66 # /usr/ucb/ps auxw | egrep mailman >> | awk '{print $2 " " $11}' >> 24167 --runner=IncomingRunner:5:8 >> 24165 --runner=BounceRunner:0:1 >> 24158 --runner=IncomingRunner:7:8 >> 24162 --runner=VirginRunner:0:1 >> 24163 --runner=IncomingRunner:1:8 >> 24166 --runner=IncomingRunner:0:8 >> 24168 --runner=IncomingRunner:4:8 >> 24169 --runner=IncomingRunner:2:8 >> 24171 --runner=IncomingRunner:6:8 >> 24172 --runner=IncomingRunner:3:8 >> 24160 --runner=CommandRunner:0:1 >> 24161 --runner=OutgoingRunner:0:1 >> 24164 --runner=ArchRunner:0:1 >> 24170 /bin/python >> 24159 /bin/python > > > What are these last 2? Presumably they are the missing NewsRunner and > RetryRunner, but what is the extra stuff in the ps output causing $11 > to be the python command and not the runner option? And again, why are > these two, which presumably have done nothing, seemingly the biggest. > > Here's some additional thought. > > Are you sure there is an actual leak? Do you know that if you just let > them run, they don't reach some stable size and remain there as > opposed to growing so large that they eventually throw a MemoryError > exception and get restarted by mailmanctl. > > If you allowed them to do that once, the MemoryError traceback might > provide a clue. > > Caveat! I know very little about Python's memory management. Some of > what follows may be wrong. > > Here's what I think - Python allocates more memory (from the OS) as > needed to import additional modules and create new objects. Imports > don't go away, but objects that are destroyed or become unreachable > (eg a file object that is closed or a message object whose only > reference gets assigned to something else) become candidates for > garbage collection and ultimately the memory allocated to them is > collected and reused (assuming no leaks). I *think* however, that no > memory is ever actually freed back to the OS. Thus, Python processes > that run for a long time can grow, but don't shrink. > > Now, IncomingRunner in particular can get very large if large messages > are arriving, even if those messages are ultimately not processed very > far. Incoming runner reads the entire message into memory and then > parses it into a message object which is even bigger than the message > string. So, if someone happens to send a 100MB attachment to a list, > IncomingRunner is going to need over 200MB before it ever looks at the > message itself. This memory will later become available for other use > within that IncomingRunner instance, but I don't think it is ever > freed back to the OS. > > Also, I see very little memory change between the 3 hour old snapshot > above and the 8 hour old one from your prior post. If this is really a > memory leak, I'd expect the 8 hour old ones to be perhaps twice as big > as the 3 hour old ones. > > Also, do you have any really big lists with big config.pck files. If > so, Runners will grow as they instantiate that (those) big list(s). -- Fletcher Cocquyt Senior Systems Administrator Information Resources and Technology (IRT) Stanford University School of Medicine Email: [EMAIL PROTECTED] Phone: (650) 724-7485 ------------------------------------------------------ Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9