It would strike me as very odd if nose was the culprit (inherently vs.
just poorly written tests). Before nova switched to testr we had over
5000 tests, which is 4x what's in keystone right now.
Also, tempest has > 1000 tests running in nose, and very clearly can
function on a devstack gate node. And we allocate lots of servers to
boot. :)
-Sean
On 06/19/2013 07:22 PM, Brant Knudson wrote:
I was looking into this with Anita Kuno today.
Keystone unit tests would fail when she was running on her vm set up for
testing. We found that the system was running out of memory. When I run
it myself, watching memory usage with top, the nosetester process uses
over 800MB. Anita's vm has 1GB total, and she sent me a screenshot
showing that the oom killer got it. So it's not that the Keystone unit
tests don't work in a venv generally -- you need enough of RAM.
Memory use growing without bound indicates a memory leak, so I started
looking into where references were piling up. I was hoping it would be
an easy fix where the Keystone test code could be changed to clear a
list or something. I haven't looked at python memory leaks before so I
might be wrong on all this, but here's my analysis.
The summary is the references are piling up in the nosetests runner as
it's storing the results of all the tests, so I don't think there's
anything we can do in Keystone code to fix it.
I made use of heapy https://pypi.python.org/pypi/guppy/0.1.10 to find
the heap usage.
After running 872 tests,
> /usr/lib/python2.7/dist-packages/nose/suite.py(223)run()
-> test(orig)
(Pdb) p result
<nose.result.TextTestResult run=869 errors=3 failures=0>
hpy.heap() shows the biggest user of the heap is unnamed dicts.
(Pdb) h1.heap()
Partition of a set of 487786 objects. Total size = 88365600 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 76542 16 27581904 31 27581904 31 dict (no owner)
1 128620 26 11649056 13 39230960 44 str
2 72387 15 6771416 8 46002376 52 tuple
3 3043 1 3189064 4 49191440 56 dict of 0x2b96700
4 21977 5 2615816 3 51807256 59 list
5 20374 4 2444880 3 54252136 61 function
6 739 0 2435656 3 56687792 64 dict of
sqlalchemy.dialects.sqlite.base.SQLiteCompiler
7 2579 1 2324312 3 59012104 67 type
8 17378 4 2224384 3 61236488 69 types.CodeType
9 854 0 2123792 2 63360280 72 dict of module
<899 more rows. Type e.g. '_.more' to view.>
By running for a while and watching the list change, one sees the "dict
(no owner)" line start out at the bottom of the list and climb up to the
top as more tests run.
Digging into the dict data a bit more, one can see what's holding the
refs to dicts,
(Pdb) h1.heap([0].get_rp(depth=1)
Reference Pattern by <[dict of] class>.
0: _ --- [-] 76503 dict (no owner): 0x1a4f150*1673, 0x1a52380*15...
1: a [+] 739 dict of
sqlalchemy.dialects.sqlite.base.SQLiteCompiler: 0x...
2: b ---- [+] 514 dict of routes.route.Route: 0x5db1b50..., 0x68f02d0
3: c [+] 905 dict of nose.case.Test: 0x2a78490, 0x2ad2c90,
0x2ad2e50...
4: d ---- [+] 873 dict (no owner): 0x1aaff60*10, 0x1bf9c90*2,
0x1f7ef10*34...
5: e [S] 157 dict of module: _warnings, errno..., os, sys, zipimport
6: f ---- [+] 762 list: 0x2a6e0e0*4, 0x2a6e488*4, 0x2a6e4d0*1,
0x2a6e518*1...
7: g [+] 146 dict of test_backend_ldap.LDAPIdentity: 0x44b36d0...
8: h ---- [+] 146 dict of
test_backend_ldap.LDAPIdentityEnabledEmulation: 0x...
9: i [+] 129 dict of test_backend_sql.SqlIdentity: 0x4522b90,
0x50b1290...
If you watch this after running a few more tests, it's the #4 entry
where the number keeps going up, so looking into this more:
(Pdb) h1.heap()[0].get_rp(depth=1)[4].get_rp()
Reference Pattern by <[dict of] class>.
0: _ --- [-] 873 dict (no owner): 0x1aaff60*10, 0x1bf9c90*2,
0x1f7ef10*34...
1: a [-] 274 dict (no owner): 0x259c0f0*1, 0x259c420*8,
0x25b5a10*1...
2: aa ---- [-] 233 dict of
keystone.catalog.backends.templated.TemplatedCata...
3: a3 [-] 233
keystone.catalog.backends.templated.TemplatedCatalog: 0x...
4: a4 ------ [-] 60 dict of test_keystoneclient.KcEssex3TestCase:
0x7a20750...
5: a5 [-] 60 test_keystoneclient.KcEssex3TestCase: 0x76703d0...
6: a6 -------- [-] 60 dict of nose.case.Test: 0x7a17650..., 0x7a17690
7: a7 [+] 60 nose.case.Test: 0x6de91d0, 0x6de9210, 0x6de9350...
8: a5b ------- [-] 60 dict of unittest2.case._TypeEqualityDict:
0x7a20b50...
9: a5ba [+] 60 unittest2.case._TypeEqualityDict: 0x6b5bad0...
<Type e.g. '_.more' for more.>
(Pdb) h1.heap()[0].get_rp(depth=1)[4].get_rp().more
10: a3b ----- [-] 57 dict of test_keystoneclient.Kc11TestCase: 0x7883150...
11: a3ba [-] 57 test_keystoneclient.Kc11TestCase: 0x7883150,
0x7b43150...
12: a3baa ----- [-] 57 dict of nose.case.Test: 0x7a389d0, 0x8394d50...
13: a3ba3 [+] 57 nose.case.Test: 0x7a38490, 0x7a385d0, 0x7a389d0...
14: a3bab ----- [-] 57 dict of unittest2.case._TypeEqualityDict:
0x7b63dd0...
15: a3baba [+] 57 unittest2.case._TypeEqualityDict: 0x7883510...
16: a3c ----- [-] 31 dict of test_keystoneclient.KcMasterTestCase:
0x6dc1490...
17: a3ca [-] 31 test_keystoneclient.KcMasterTestCase: 0x6aadc90...
18: a3caa ----- [-] 31 dict of nose.case.Test: 0x83225d0, 0x8322610...
19: a3ca3 [+] 31 nose.case.Test: 0x83225d0, 0x8322610, 0x8322890...
The smoking gun here is that the source of the references is nose.case.Test.
So I think that the nose runner is keeping references to the the dicts
that are compared. Also, it looks like nose keeps a reference to the
test instance (along with any objects created in setUp with self.xxx =
Xxx()), but this isn't using so much memory as the test dicts.
- Brant
On Mon, Jun 17, 2013 at 6:06 PM, Monty Taylor <mord...@inaugust.com
<mailto:mord...@inaugust.com>> wrote:
...
- unittests should always work on a devstack host inside of a venv
I mean, it's a venv - the fact that you're on a devstack machine is
irrelvant. Btw- apparently keystone and python-keystone client do not
work in this context - we should probably sort that out.
...
Monty
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Sean Dague
http://dague.net
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
