[Bug 1919261] Re: Upgrading Ceph from 14.2.11-0ubuntu0.19.10.1~cloud4 to 15.2.8-0ubuntu0.20.04.1~cloud0 fails when ceph-mds is installed
Hello, I tried to verify this SRU, but halfway through the process... realized I was pulling from -updates (current) and not -proposed... and even so, I could not replicate the bug: So I guess it's fixed? Tyler --- # Boot a clean cloud-init'd 18.04 image: ubuntu@dev-10-34-138-116:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.6 LTS Release: 18.04 Codename: bionic # Configure apt to internal mirror of Train UCA on this LP's file date (2021-03-15) ... # Install ceph-mds, verify the intended version was installed: ubuntu@dev-10-34-138-116:~$ sudo DEBIAN_PRIORITY=critical DEBIAN_FRONTEND=noninteractive apt-get install -y ceph-mds ... ubuntu@dev-10-34-138-116:~$ dpkg-query -W ceph-mds ceph-mds 14.2.11-0ubuntu0.19.10.1~cloud4 # Remove train UCA and do-release-upgrade ... ubuntu@dev-10-34-138-116:~$ sudo apt-get update ubuntu@dev-10-34-138-116:~$ sudo apt -yu full-upgrade ubuntu@dev-10-34-138-116:~$ sudo apt -y install ubuntu-release-upgrader-core ubuntu@dev-10-34-138-116:~$ sudo do-release-upgrade -f DistUpgradeViewNonInteractive ... # Ceph upgraded without failure: ubuntu@dev-10-34-138-116:~$ dpkg-query -W ceph-mds ceph-mds 15.2.14-0ubuntu0.20.04.2 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919261 Title: Upgrading Ceph from 14.2.11-0ubuntu0.19.10.1~cloud4 to 15.2.8-0ubuntu0.20.04.1~cloud0 fails when ceph-mds is installed To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1919261/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964802] Re: [SRU] ceph 15.2.16
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964802 Title: [SRU] ceph 15.2.16 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1964802/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
We have also been bitten by this. Apologies if this does not help solve the bug, but this issue has been floating for quite awhile and the following may help future cloud operators... In our case, we trying to re-IP ALL of our Ceph Mons. As Corey mentioned, this bug report is for *Cinder volumes*... but note that all of our instances were observed to make use of RBD-backed configuration drives which suffered the same problem as the images... so you may suffer from both problems even if you exclusively boot all instances from volume! * RBD config drives AND Glance/image-based RBD volumes DID NOT have their Ceph Mon addresses updated as part of a live-migration, even with the patch in #9. The Ceph Mon addresses for these types in volumes IN PARTICULAR are NOT stored anywhere in a database and rather seem to be derived as needed when certain actions occur and otherwise carted around from hyp to hyp by way of the libvirt domain XML. Again, see the other LP bug for this. * Trying to 'fix up' the Ceph Mon addresses via 'virsh edit' or comparable and then trying to live-migrate an instance to have those changes reflected is futile, because the Ceph Mon address changes are not reflected until a hard bounce of the VMM for that instance AND nova- compute uses the running copy of libvirt domain XML when shipping a copy to a destination hypervisor, NOT the copy on disk. What we may end up doing (that worked in a lab environment) is to respin a patch off #9 that is applied to all worknode. It searches for all instances of './devices/disk/source' in the XML document which have an 'rbd' protocol. For each entry, we replace the current host subelements with our new Ceph Mon addresses. Then live-migrate every VM exactly once. This works for all kinds of RBD volumes and, unlike 'virsh edit', works because the in-memory libvirt domain XML is rewritten prior to the VMM starting up on the destination host. Note that while you are doing the LMs and updating the domain XMLs, you must keep at least one of the old and new Ceph Mons accessible at all times. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo
Ponnuvel, don't the fio tests show significantly better write IOPS and latency in the optimized case? == rocksdb%20optimized.txt ... write: IOPS=1269, BW=5076KiB/s (5198kB/s)(2048MiB/413114msec); 0 zone ... | 99.00th=[ 180], 99.50th=[ 384], 99.90th=[ 1003], 99.95th=[ 1636], | 99.99th=[ 3641] == rocksdb%20UNoptimized.txt ... write: IOPS=383, BW=1533KiB/s (1570kB/s)(2048MiB/1367654msec); 0 zone ... | 99.00th=[ 1099], 99.50th=[ 1536], 99.90th=[ 3239], 99.95th=[ 4178], | 99.99th=[ 6544] -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894453 Title: Building Ceph packages with RelWithDebInfo To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1894453/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo
Good to know it's just rocksdb! Has Canonical also considered compiling the source code with -DNDEBUG to align with what appears to be the upstream default set of RelWithDebInfo flags? The Ceph source code is quasi-judicious about its use of `ceph_assert`, which is effectively compiled out when -DNDEBUG is specified. I am unsure of the performance benefit of this in particular. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894453 Title: Building Ceph packages with RelWithDebInfo To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1894453/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo
Oof - so I hacked up debian/rules a little so that "RelWithDebInfo" is used instead of "None" for CMAKE_BUILD_TYPE. I then ^C'd the dpkg- buildpackage after it started compiling the first few source files and observed that: * When doing make V=1 VERBOSE=1 in obj-x86_64-linux-gnu/src/rocksdb *without* any changes, the source is compiled with *neither* -g nor -O2 -DNDEBUG. * When doing make V=1 VERBOSE=1 in obj-x86_64-linux-gnu/src/rocksdb *with* my changes, rocksdb is built with *both* -g and -O2 -DNDEBUG. -O2 -DNDEBUG should result in significantly better performance/lower utilization for metadata operations... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894453 Title: Building Ceph packages with RelWithDebInfo To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1894453/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo
Checked out ceph-15.2.3 source on a up-to-date focal VM. When running dpkg-buildpackage, CMakeCache.txt gets generated as expected. However, the generated CMAKE_BUILD_TYPE is "None". A bit odd, but going a step further, one can grep for CMAKE_C_FLAGS in that same file and see something to the effect of: CMAKE_C_FLAGS_DEBUG:STRING= CMAKE_C_FLAGS_NONE:STRING= ... ... CMAKE_C_FLAGS_RELWITHDEBINFO:STRING= It seems, though, that since "None" is used, all of those are actually ignored and CMAKE_C_FLAGS:STRING= is used That particular set of flags seems to be missing -DNDEBUG. It's worth noting that -DNDEBUG is defined within CMAKE_C_FLAGS_RELWITHDEBINFO:STRING and including it may result in a more optimized build. This is not quite the Gentoo issue as Trent alluded. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894453 Title: Building Ceph packages with RelWithDebInfo To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1894453/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
