Hi Josh, In our case adding the monitor hostnames (alias) would have made only a slight difference: as we moved the servers to another cluster, the client received an authorisation failure rather than a connection failure and did not try to fail over to the next IP in the list. So, adding the alias to list would have improved the chances to hit a good monitor, but it would not have eliminated the problem.
I’m not sure storing IPs in the nova database is a good idea in gerenal. Replacing (not adding) these by the hostnames is probably better. Another approach may be to generate this part of connection_info (and hence the XML) dynamically from the local ceph.conf when the connection is created. I think a mechanism like this is for instance used to select a free port for the vnc console when the instance is started. Cheers, Arne — Arne Wiebalck CERN IT On 08 May 2015, at 05:37, David Medberry <[email protected]<mailto:[email protected]>> wrote: Josh, Certainly in our case the the monitor hosts (in addition to IPs) would have made a difference. On Thu, May 7, 2015 at 3:21 PM, Josh Durgin <[email protected]<mailto:[email protected]>> wrote: Hey folks, thanks for filing a bug for this: https://bugs.launchpad.net/cinder/+bug/1452641 Nova stores the volume connection info in its db, so updating that would be a workaround to allow restart/migration of vms to work. Otherwise running vms shouldn't be affected, since they'll notice any new or deleted monitors through their existing connection to the monitor cluster. Perhaps the most general way to fix this would be for cinder to return any monitor hosts listed in ceph.conf (as they are listed, so they may be hostnames or ips) in addition to the ips from the current monmap (the current behavior). That way an out of date ceph.conf is less likely to cause problems, and multiple clusters could still be used with the same nova node. Josh On 05/06/2015 12:46 PM, David Medberry wrote: Hi Arne, We've had this EXACT same issue. I don't know of a way to force an update as you are basically pulling the rug out from under a running instance. I don't know if it is possible/feasible to update the virsh xml in place and then migrate to get it to actually use that data. (I think we tried that to no avail.) dumpxml=>massage cephmons=>import xml If you find a way, let me know, and that's part of the reason I'm replying so that I stay on this thread. NOTE: We did this on icehouse. Haven't tried since upgrading to Juno but I don't note any change therein that would mitigate this. So I'm guessing Liberty/post-Liberty for a real fix. On Wed, May 6, 2015 at 12:57 PM, Arne Wiebalck <[email protected]<mailto:[email protected]> <mailto:[email protected]<mailto:[email protected]>>> wrote: Hi, As we swapped a fraction of our Ceph mon servers between the pre-production and production cluster — something we considered to be transparent as the Ceph config points to the mon alias—, we ended up in a situation where VMs with volumes attached were not able to boot (with a probability that matched the fraction of the servers moved between the Ceph instances). We found that the reason for this is the connection_info in block_device_mapping which contains the IP adresses of the mon servers as extracted by the rbd driver in initialize_connection() at the moment when the connection is established. From what we see, however, this information is not updated as long as the connection exists, and will hence be re-applied without checking even when the XML is recreated. The idea to extract the mon servers by IP from the mon map was probably to get all mon servers (rather than only one from a load-balancer or an alias), but while our current scenario may be special, we will face a similar problem the day the Ceph mons need to be replaced. And that makes it a more general issue. For our current problem: Is there a user-transparent way to force an update of that connection information? (Apart from fiddling with the database entries, of course.) For the general issue: Would it be possible to simply use the information from the ceph.conf file directly (an alias in our case) throughout the whole stack to avoid hard-coding IPs that will be obsolete one day? Thanks! Arne — Arne Wiebalck CERN IT __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<http://[email protected]/?subject:unsubscribe> <http://[email protected]?subject:unsubscribe<http://[email protected]/?subject:unsubscribe>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<http://[email protected]/?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<http://[email protected]/?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]<mailto:[email protected]>?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
