There's a bug on this:
https://bugs.launchpad.net/ironic/+bug/1321787?comments=all
It seems like it's been well-known for a long time that paramiko
parallelism doesn't work well with eventlet. Ironic's aggressive use of
the ssh power driver seems to hit this hard.
The sign that you're hitting a problem with this is the "multiple
simultaneous readers" warning, which is spurious (but a sign of trouble).
I've started to follow up this problem with the eventletdev mailing list,
since having had a trawl back through tickets it looks like we've seen
other issues arising from this in various places, going back at least 18
months. They're not all paramiko-related; not are they necessarily
"caused" by the thing mentioned in the TRACE lines - that's just the point
eventlet can detect the problem. I've seen the glanceclient, at the least,
also trigger this - as well as Ironic's use of utils.execute to launch
(again, parallel) qemu-img calls.
I'm just trying a tripleo run with a much reduced workers_pool_size to see
if I can at least forcibly get a run to complete successfully.
As to where the problem lies: it seems eventlet has a registered listener
backed by one FD. That FD gets recycled by another thread, which attempts
to read or write on it. That's why eventlet is carping. Paramiko seems to
trigger this quite reliably because it uses a worker thread to manage its
ssh communication.
Fixes to eventlet might be quite tricky - the bug above has a link to some
quick sketches in github - although it's just struck me that there may be
a simpler approach to investigate, which I'll pursue after sending this.
It'd be good to get some eyeballs on this eventlet problem - it's been
hitting us for quite a time - only, previously to ironic, not at a
sufficiently high rate to cause huge amounts of pain.
Cheers,
jan
PS. A longer, rambling braindump went to the eventletdev mailing list,
which can be found here:
https://lists.secondlife.com/pipermail/eventletdev/2014-June/thread.html
...I think I've a better handle on the problem now, but I still don't have
a satisfactory from-first-principles explanation of exactly the state of
events that causes paramiko to trigger this.
--
j...@ioctl.org http://ioctl.org/jan/
stty intr ^m
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
