[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
While we had to migrate/change ceph mon ip's in our openstack deployment, which is stein at the moment, we got hit by the same problem. We fixed it also by manually changing all db entries that required an update. Comment https://bugs.launchpad.net/nova/+bug/1452641/comments/23 helped us a ton with creating a sensible database query. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
Please don't set OpenStack bugs to Public Security without some explanation as to why you believe this to be an exploitable risk which needs attention from the OpenStack vulnerability managers for coordinating a possible security advisory. I'm switching this back to a normal Public bug for now, but if you suspect this report represents an actual security risk then please explain and set it to Public Security again. Thanks! ** Information type changed from Public Security to Public -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
** Information type changed from Public to Public Security -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
We are also experiencing this issue with Stein. We are deployed using rook-ceph as our ceph deployment and the monitors IP address will rotate if any of the monitors are down for more than 10 minutes. Which means anytime we do maintenance on a node running one of rook-ceph's monitors the ip address will change and we have to go into the database and update all the mon_hosts for the VMs just so they are able to start up again. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
We have the same issue with Rocky.
One of my SQL wizard colleague helped me with some query, which can change the
block_device_mapping table, and the RBD host/username/ports (if you change the
number of ceph monitors, you'll need it)
Since we have multiple Zones, and our change will only affect Zone1, and
since we have iSCSI storage too, we needed a bit more precise query.
Also my colleague pointed out, that the connection_info is JSON, and
since the MariaDB 10.2.3 have support for json, he used them, just to be
sure not to mess up the syntax.
So the three query (use with caution, and - of course - your own risk!):
update block_device_mapping as b set connection_info =
json_replace(connection_info, '$.data.auth_username', 'dev-r1z1-c4e')
where instance_uuid in (select i.uuid from instances as i where
i.deleted_at is null and i.availability_zone = 'Zone1') AND
JSON_EXISTS(b.connection_info, '$.data.hosts') = 1 and b.deleted_at is
NULL;
update block_device_mapping as b set connection_info =
json_replace(connection_info, '$.data.hosts', JSON_ARRAY("10.1.58.156",
"10.1.58.157", "10.1.58.158")) where instance_uuid in (select i.uuid from
instances as i where i.deleted_at is null and i.availability_zone = 'Zone1')
AND JSON_EXISTS(b.connection_info, '$.data.hosts') = 1 and b.deleted_at is NULL;
update block_device_mapping as b set connection_info =
json_replace(connection_info, '$.data.ports', JSON_ARRAY("6789", "6789",
"6789")) where instance_uuid in (select i.uuid from instances as i where
i.deleted_at is null and i.availability_zone = 'Zone1') AND
JSON_EXISTS(b.connection_info, '$.data.hosts') = 1 and b.deleted_at is NULL;
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1452641
Title:
Static Ceph mon IP addresses in connection_info can prevent VM startup
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
We have also been bitten by this. Apologies if this does not help solve the bug, but this issue has been floating for quite awhile and the following may help future cloud operators... In our case, we trying to re-IP ALL of our Ceph Mons. As Corey mentioned, this bug report is for *Cinder volumes*... but note that all of our instances were observed to make use of RBD-backed configuration drives which suffered the same problem as the images... so you may suffer from both problems even if you exclusively boot all instances from volume! * RBD config drives AND Glance/image-based RBD volumes DID NOT have their Ceph Mon addresses updated as part of a live-migration, even with the patch in #9. The Ceph Mon addresses for these types in volumes IN PARTICULAR are NOT stored anywhere in a database and rather seem to be derived as needed when certain actions occur and otherwise carted around from hyp to hyp by way of the libvirt domain XML. Again, see the other LP bug for this. * Trying to 'fix up' the Ceph Mon addresses via 'virsh edit' or comparable and then trying to live-migrate an instance to have those changes reflected is futile, because the Ceph Mon address changes are not reflected until a hard bounce of the VMM for that instance AND nova- compute uses the running copy of libvirt domain XML when shipping a copy to a destination hypervisor, NOT the copy on disk. What we may end up doing (that worked in a lab environment) is to respin a patch off #9 that is applied to all worknode. It searches for all instances of './devices/disk/source' in the XML document which have an 'rbd' protocol. For each entry, we replace the current host subelements with our new Ceph Mon addresses. Then live-migrate every VM exactly once. This works for all kinds of RBD volumes and, unlike 'virsh edit', works because the in-memory libvirt domain XML is rewritten prior to the VMM starting up on the destination host. Note that while you are doing the LMs and updating the domain XMLs, you must keep at least one of the old and new Ceph Mons accessible at all times. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
We're changed our ceph-monitor ip's and we're running into this issue unfortunately. It is fixed on the cinder side, but still broken on the nova side. We could fix it with patch in of Walt (comment #9) but there must be more users running into this issue unknowingly. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
** Changed in: nova (Ubuntu) Status: In Progress => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
** Changed in: nova Assignee: Seyeong Kim (xtrusia) => (unassigned) ** Changed in: nova (Ubuntu) Assignee: Seyeong Kim (xtrusia) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
** Changed in: nova (Ubuntu) Assignee: (unassigned) => Seyeong Kim (xtrusia) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
** Changed in: nova (Ubuntu) Assignee: Corey Bryant (corey.bryant) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
** Changed in: nova Assignee: Lee Yarwood (lyarwood) => Seyeong Kim (xtrusia) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
** Changed in: nova Assignee: Seyeong Kim (xtrusia) => Lee Yarwood (lyarwood) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
** Changed in: nova Assignee: Lee Yarwood (lyarwood) => Seyeong Kim (xtrusia) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
** Changed in: nova Assignee: Seyeong Kim (xtrusia) => Lee Yarwood (lyarwood) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
** Changed in: nova Assignee: Corey Bryant (corey.bryant) => Seyeong Kim (xtrusia) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
Just a clarification on the process to 'move' ceph-mon units. I added ceph mons to the cluster, and removed the old ones - in this case it was a 'juju add-unit' and 'juju remove-unit' but any process to achieve the same thing would have the same result - the mons are now all on different addresses. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
** Changed in: nova (Ubuntu) Status: New => In Progress ** Changed in: nova (Ubuntu) Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
Fix proposed to branch: master Review: https://review.openstack.org/579004 ** Changed in: nova Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
I did some initial testing with the default parameter value for refresh_conn_info set to True in _get_instance_block_device_info() and unfortunately an instance with rbd volume attached does not successfully stop/start after ceph-mon's are moved to new IP addresses. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
** Changed in: nova Assignee: (unassigned) => Corey Bryant (corey.bryant) ** Changed in: nova (Ubuntu) Assignee: (unassigned) => Corey Bryant (corey.bryant) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
FWIW, in the cloud we saw this, migrating the (stopped) instance also updated the connection info - it was just that migrating hundreds of instances wasn't practical. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
Just to summarize my understanding, and perhaps clarify for others, this bug is focused on stale connection_info for rbd volumes (not rbd images). rbd images have a related issue during live migration that is being handled in a separate bug (see comment 12 above). Focusing on connection_info for rbd volumes now (and thanks to Matt Riedemann's comments for the tips here). connection_info appears to be properly refreshed for live migration in pre_live_migration() where _get_instance_block_device_info() is called with refresh_conn_info=True (see comment 9 above and https://github.com/openstack/nova/blob/stable/queens/nova/compute/manager.py#L5977). Is the fix as simple as flipping refresh_conn_info=False to True for some of the other calls to _get_instance_block_device_info()? Below is an audit of the _get_instance_block_device_info() calls. Calls to _get_instance_block_device_info() with refresh_conn_info=False: _destroy_evacuated_instances() _init_instance() _resume_guests_state() _shutdown_instance() _power_on() _do_rebuild_instance() reboot_instance() revert_resize() _resize_instance() resume_instance() shelve_offload_instance() check_can_live_migrate_source() _do_live_migration() _post_live_migration() post_live_migration_at_destination() rollback_live_migration_at_destination() Calls to _get_instance_block_device_info() with refresh_conn_info=True: finish_revert_resize() _finish_resize() pre_live_migration() Based on xavpaice's comments in (see comment 13 above -- "... existing, running, instances were fine, fresh new instances were fine, but when we stopped instances via nova, then started them again, they failed to start ..."), it would seem that the following should also have refresh_conn_info=True: _power_on() # solves xavpaice's scenario? _do_rebuild_instance() reboot_instance() -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
The attachment "virt/libvirt/migration.py patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team. [This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.] ** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
** Also affects: nova (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
