Josh, Certainly in our case the the monitor hosts (in addition to IPs) would have made a difference.
On Thu, May 7, 2015 at 3:21 PM, Josh Durgin <[email protected]> wrote: > Hey folks, thanks for filing a bug for this: > > https://bugs.launchpad.net/cinder/+bug/1452641 > > Nova stores the volume connection info in its db, so updating that > would be a workaround to allow restart/migration of vms to work. > Otherwise running vms shouldn't be affected, since they'll notice any > new or deleted monitors through their existing connection to the > monitor cluster. > > Perhaps the most general way to fix this would be for cinder to return > any monitor hosts listed in ceph.conf (as they are listed, so they may > be hostnames or ips) in addition to the ips from the current monmap > (the current behavior). > > That way an out of date ceph.conf is less likely to cause problems, > and multiple clusters could still be used with the same nova node. > > Josh > > On 05/06/2015 12:46 PM, David Medberry wrote: > >> Hi Arne, >> >> We've had this EXACT same issue. >> >> I don't know of a way to force an update as you are basically pulling >> the rug out from under a running instance. I don't know if it is >> possible/feasible to update the virsh xml in place and then migrate to >> get it to actually use that data. (I think we tried that to no avail.) >> dumpxml=>massage cephmons=>import xml >> >> If you find a way, let me know, and that's part of the reason I'm >> replying so that I stay on this thread. NOTE: We did this on icehouse. >> Haven't tried since upgrading to Juno but I don't note any change >> therein that would mitigate this. So I'm guessing Liberty/post-Liberty >> for a real fix. >> >> >> >> On Wed, May 6, 2015 at 12:57 PM, Arne Wiebalck <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi, >> >> As we swapped a fraction of our Ceph mon servers between the >> pre-production and production cluster >> -- something we considered to be transparent as the Ceph config >> points to the mon alias--, we ended >> up in a situation where VMs with volumes attached were not able to >> boot (with a probability that matched >> the fraction of the servers moved between the Ceph instances). >> >> We found that the reason for this is the connection_info in >> block_device_mapping which contains the >> IP adresses of the mon servers as extracted by the rbd driver in >> initialize_connection() at the moment >> when the connection is established. From what we see, however, this >> information is not updated as long >> as the connection exists, and will hence be re-applied without >> checking even when the XML is recreated. >> >> The idea to extract the mon servers by IP from the mon map was >> probably to get all mon servers (rather >> than only one from a load-balancer or an alias), but while our >> current scenario may be special, we will face >> a similar problem the day the Ceph mons need to be replaced. And >> that makes it a more general issue. >> >> For our current problem: >> Is there a user-transparent way to force an update of that >> connection information? (Apart from fiddling >> with the database entries, of course.) >> >> For the general issue: >> Would it be possible to simply use the information from the >> ceph.conf file directly (an alias in our case) >> throughout the whole stack to avoid hard-coding IPs that will be >> obsolete one day? >> >> Thanks! >> Arne >> >> -- >> Arne Wiebalck >> CERN IT >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> [email protected]?subject:unsubscribe >> <http://[email protected]?subject:unsubscribe >> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> [email protected]?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
