On 02/29/2012 09:08 PM, Johannes Erdfelt wrote:
> On Wed, Feb 29, 2012, Vishvananda Ishaya <vishvana...@gmail.com> wrote:
>> We have had a memory leak due to an interaction with eventlet for a
>> while that Johannes has just made a fix for.
>>
>> bug:
>> https://bugs.launchpad.net/nova/+bug/903199
>>
>> fix:
>> https://bitbucket.org/which_linden/eventlet/pull-request/10/monkey-patch-threadingcurrent_thread-as
>>
>> Unfortuantely, I don' t think we have a decent workaround for nova
>> while that patch is upstreamed. I wanted to make sure that all of
>> the distros are aware of it in case they want to carry an eventlet
>> patch to prevent the slow memory leak.
>
> There is one other possible workaround, but I didn't feel like it was
> safe since we would likely need to audit all of the third party libraries
> to ensure they don't cause problems.
>
> The memory leak only happens when monkey patching the
> thread/threading/Queue modules. I looked at the nova sources and did
> some tests and it doesn't appear nova needs those modules patches.
>
> However, third party modules might need it. Also, I only tested on
> xenapi. libvirt and/or vmwareapi might have problems. Or possibly other
> drivers (firewall, volume, etc) in nova that I didn't use in my tests.
>
> If you're having problems with the memory leak in eventlet and applying
> the patch isn't an option, then monkey patching everything but thread
> might something worth trying.
>
> eventlet.monkey_patch(os=True, socket=True, time=True)
Note I tried the patch but got errors
# rpm -qa python-greenlet python-eventlet
python-eventlet-0.9.16-4.fc17.noarch
python-greenlet-0.3.1-9.fc17.x86_64
# /usr/bin/nova-cert --flagfile /dev/null --config-file /etc/nova/nova.conf
--logfile /var/log/nova/cert.log
2012-03-05 15:09:09 AUDIT nova.service [-] Starting cert node (version
2012.1-LOCALBRANCH:LOCALREVISION)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 336, in
fire_timers
timer()
File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 56, in
__call__
cb(*args, **kw)
File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 192, in
main
result = function(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/service.py", line 101, in
run_server
server.start()
File "/usr/lib/python2.7/site-packages/nova/service.py", line 176, in start
self.conn = rpc.create_connection(new=True)
File "/usr/lib/python2.7/site-packages/nova/rpc/__init__.py", line 47, in
create_connection
return _get_impl().create_connection(new=new)
File "/usr/lib/python2.7/site-packages/nova/rpc/impl_qpid.py", line 507, in
create_connection
return rpc_amqp.create_connection(new, Connection.pool)
File "/usr/lib/python2.7/site-packages/nova/rpc/amqp.py", line 310, in
create_connection
return ConnectionContext(connection_pool, pooled=not new)
File "/usr/lib/python2.7/site-packages/nova/rpc/amqp.py", line 84, in __init__
server_params=server_params)
File "/usr/lib/python2.7/site-packages/nova/rpc/impl_qpid.py", line 294, in
__init__
self.connection = qpid.messaging.Connection(self.broker)
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line
178, in __init__
self._driver = Driver(self)
File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 347,
in __init__
self._selector = Selector.default()
File "/usr/lib/python2.7/site-packages/qpid/selector.py", line 54, in default
sel.start()
File "/usr/lib/python2.7/site-packages/qpid/selector.py", line 99, in start
self.thread = Thread(target=self.run)
File "/usr/lib64/python2.7/threading.py", line 446, in __init__
self.__daemonic = self._set_daemon()
File "/usr/lib64/python2.7/threading.py", line 470, in _set_daemon
return current_thread().daemon
AttributeError: '_GreenThread' object has no attribute 'daemon'
2012-03-05 15:09:10 CRITICAL nova [-] '_GreenThread' object has no attribute
'daemon'
Exception KeyError: KeyError(139994820976048,) in <module 'threading' from
'/usr/lib64/python2.7/threading.pyc'> ignored
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.7/site-packages/qpid/selector.py", line 138, in stop
self.thread.join(timeout)
AttributeError: 'NoneType' object has no attribute 'join'
Error in sys.exitfunc:
2012-03-05 15:09:10 CRITICAL nova [-] 'NoneType' object has no attribute 'join'
cheers,
Pádraig.
_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
