#1999: 1.3.3 WebUI uses 100% CPU loading TorrentInfo for thousands of files
-------------------+--------------------------------------------------------
Reporter: dalek | Owner: damoxc
Type: bug | Status: new
Priority: minor | Milestone: Future
Component: webui | Version: 1.3.3
Keywords: cpu |
-------------------+--------------------------------------------------------
Hi,
Just switched to Deluge from utorrent/linux and would like to say thanks
for making it open source.
The problem I have is that when I initially connect with the WebUI (or the
console-ui) the process sits at 100% CPU for tens of minutes.
I'm only seeding a handful of torrents via a private tracker but that
equates to around 300,000 files on disk contained in those few torrents.
I ran the WebUI with --profile and it looks like the cause is the
TorrentInfo() routine in deluge/ui/common.py.
The problem I think is that when the WebUI gets a connection, it loads in
to RAM every filename contained in each torrent. This causes RAM to be
used excessively (my deluge-webui process gets to ~200MB). On top of
that, for each filename, it runs through a UTF decoding routine in
rencode.py which seems to be burning the CPU.
After tens of minutes, when the filename info is loaded, the WebUI comes
back to life. That is, unless you click on the 'files' tab for one of the
torrents with tens of thousands of files, then it hangs for ages. (I guess
as expected).
I'm not a python programmer, but to fix this issue I think you could:
- change class deluge.ui.common.TorrentInfo to not load
filename/filedata info when the class is created.
- instead, load filename/filedata info on demand via function in that
class.
- instead of loading _all_ filename/filedata info, just load say 10-20,
with a flag indicating more filename/filedata info is available if there
are more files than that in the torrent.
- in the WebUI, only load 10-20 files at a time, and present a 'next'
button to load the next 10-20 filenames in the 'file' tab.
Although I have only a few torrents but with hundreds of thousands of
files, I think this problem is similar to others in the tracker where
people have several thousand torrents, maybe with less files in each.
I'm running on Debian testing, 32bit:
libpython2.7 2.7.2-8
python2.7 2.7.2-8
python-twisted-{bin,core} 11.0.0-2
python-twisted-web 11.0.0-1
deluge-{console,common,web}, deluged 1.3.3-2
Attached is the profile output. I can provide the profile file if needed.
It was run during the first half minute or so of the webui spinning at
100% CPU. It continues like this for tens of minutes (I can time it if
needed):
{{{
>>> import hotshot, hotshot.stats
>>> stats = hotshot.stats.load("/opt/deluge/.config/deluge/deluge-
web.profile")
>>> stats.strip_dirs()
<pstats.Stats instance at 0x8ebd6ac>
>>> stats.sort_stats('time', 'calls')
<pstats.Stats instance at 0x8ebd6ac>
>>> stats.print_stats(20
... )
54843249 function calls (51486080 primitive calls) in 126.991
seconds
Ordered by: internal time, call count
List reduced from 579 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
13996456 23.969 0.000 37.435 0.000 rencode.py:220(f)
3356933/101 22.898 0.000 100.112 0.991 rencode.py:263(f)
162 22.732 0.140 126.051 0.778
selectreactor.py:93(doSelect)
16792062 16.913 0.000 16.913 0.000 utf_8.py:15(decode)
402 14.208 0.035 100.024 0.249 rencode.py:171(decode_list)
2795602 8.630 0.000 12.077 0.000
rencode.py:153(decode_string)
6312624 3.894 0.000 3.894 0.000 rencode.py:251(f)
4576888 3.849 0.000 3.849 0.000 rencode.py:135(decode_intl)
4311445 3.488 0.000 3.488 0.000 rencode.py:131(decode_inth)
105 2.659 0.025 102.783 0.979 client.py:128(dataReceived)
1508477 1.160 0.000 1.160 0.000
rencode.py:143(decode_float32)
1161239 1.039 0.000 1.039 0.000 rencode.py:139(decode_intq)
1 0.920 0.920 0.922 0.922 config.py:393(save)
127 0.449 0.004 103.301 0.813 tcp.py:443(doRead)
168 0.053 0.000 100.109 0.596 rencode.py:178(decode_dict)
41 0.037 0.001 0.037 0.001 __init__.py:827(flush)
14030 0.012 0.000 0.012 0.000 rencode.py:127(decode_intb)
5 0.007 0.001 0.007 0.001 gettext.py:343(gettext)
2 0.004 0.002 0.005 0.002
sre_compile.py:301(_optimize_unicode)
162 0.003 0.000 0.936 0.006 base.py:755(runUntilCurrent)
<pstats.Stats instance at 0x8ebd6ac>
}}}
Thanks again for this client! Any help on the above would be appreciated.
Regards,
Dale
--
Ticket URL: <http://dev.deluge-torrent.org/ticket/1999>
Deluge <http://deluge-torrent.org/>
Deluge project
--
You received this message because you are subscribed to the Google Groups
"Deluge Dev" group.
To post to this group, send email to deluge-dev@googlegroups.com.
To unsubscribe from this group, send email to
deluge-dev+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/deluge-dev?hl=en.
