As I'm looking at the data some more, I see in the sos reports provided that there may be heavy pressure for the async I/O and that may actually be what's causing the problems here.
In the syslog, I see the following events getting reported from multipathd soon after the the device is discovered (its part of the udev event hooks I believe): Jan 12 22:02:49 ub01 multipathd[179286]: sdch: add path (uevent) Jan 12 22:02:49 ub01 multipathd[179286]: io_setup failed Jan 12 22:02:49 ub01 multipathd[179286]: uevent trigger error The io_setup is a system call into the kernel which can fail for a few different reasons, but one of the reasons it will fail is if the number of async io requests are greater than the current amount. It will error out with an EAGAIN, but there's explicit documentation in the multipath that one of the reasons that the io_setup may fail in this case is due to the settings of the fs.aio-nr-max, which is essentially the maximum number of events for all currently active aio contexts. Checking the value collected I see these values: # Quick check on the box today fs.aio-max-nr = 65536 fs.aio-nr = 124928 # From the sosreport provided on 12/01 fs.aio-max-nr = 65536 fs.aio-nr = 131072 As you can see the aio-nr is well above the aio-max-nr. I'm not entirely sure what is causing this, but there are a large number of async io events queued up. I think the plan of action should be to: a) configure multipath to ignore the iscsi devices presented between cinder and nova b) cleanup any errors in the current paths used by multipath, so that we know that disk i/o to the actul backend is not an issue c) Possibly consider increasing the fs.aio-max-nr -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1644785 Title: Ubuntu Openstack Mitaka can only deploy up to 52 instances To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1644785/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
