I am using scrapyjs middleware to render js on a page, and as a result,
getting a memory leak after a while. From the telnet console, checking at
regular intervals there are no more than 30 Requests/Responses/Selectors at
any point in time. Here is the guppy output some time before the crash if
it helps
>>> x = hpy.heap()
>>> x.bytype
Partition of a set of 320665 objects. Total size = 54787848 bytes.
Index Count % Size % Cumulative % Type
0 42052 13 22098784 40 22098784 40 dict
1 107682 34 10957840 20 33056624 60 str
2 1615 1 5016552 9 38073176 69 unicode
3 56312 18 4683856 9 42757032 78 tuple
4 18040 6 2009856 4 44766888 82 list
5 1964 1 1768448 3 46535336 85 type
6 13076 4 1673728 3 48209064 88 types.CodeType
7 13294 4 1595280 3 49804344 91 function
8 9803 3 627392 1 50431736 92 pycparser.plyparser.Coord
9 4654 1 409552 1 50841288 93 __builtin__.weakref
<603 more rows. Type e.g. '_.more' to view.>
>>> x.bytype[0].byvia
Partition of a set of 42052 objects. Total size = 22114144 bytes.
Index Count % Size % Cumulative % Referred Via:
0 36086 86 16011920 72 16011920 72 '.__dict__'
1 501 1 1209336 5 17221256 78 '.__dict__', '.func_globals'
2 896 2 745472 3 17966728 81
3 563 1 263240 1 18229968 82 "['data']"
4 19 0 208456 1 18438424 83 '[0]'
5 517 1 198520 1 18636944 84 "['_implied']"
6 1 0 196888 1 18833832 85 "['_declarations']"
7 1 0 196888 1 19030720 86 "['moduleDict']",
"['modules']", '.i0_modules', '[1]'
8 625 1 175000 1 19205720 87 "['_Element__tagged_values']"
9 14 0 155216 1 19360936 88 '[1]'
<829 more rows. Type e.g. '_.more' to view.>
>>> x.bytype[1].byvia
Partition of a set of 107682 objects. Total size = 10957840 bytes.
Index Count % Size % Cumulative % Referred Via:
0 13071 12 1664696 15 1664696 15 '.co_code'
1 213 0 1553384 14 3218080 29
2 4921 5 1449952 13 4668032 43 '.func_doc', '[0]'
3 2464 2 839056 8 5507088 50 "['__doc__']"
4 12516 12 659432 6 6166520 56 '.co_lnotab'
5 6808 6 423400 4 6589920 60 '[0]'
6 4543 4 355288 3 6945208 63 '.co_filename'
7 3449 3 205800 2 7151008 65 '[1]'
8 3131 3 189584 2 7340592 67 "['c_name_with_marker']"
9 1845 2 108296 1 7448888 68 '[2]'
<25444 more rows. Type e.g. '_.more' to view.>
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
