[Bug 1960065] Re: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages
Going from focal distro to uca-wallaby does not require a dist-upgrade to complete: ubuntu@distro-wallaby:~$ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=20.04 DISTRIB_CODENAME=focal DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS" ubuntu@distro-wallaby:~$ dpkg -l | grep ceph ii ceph-common 15.2.14-0ubuntu0.20.04.2 amd64common utilities to mount and interact with a ceph storage cluster ii libcephfs2 15.2.14-0ubuntu0.20.04.2 amd64Ceph distributed file system client library ii python3-ceph-argparse15.2.14-0ubuntu0.20.04.2 amd64Python 3 utility libraries for Ceph CLI ii python3-ceph-common 15.2.14-0ubuntu0.20.04.2 all Python 3 utility libraries for Ceph ii python3-cephfs 15.2.14-0ubuntu0.20.04.2 amd64Python 3 libraries for the Ceph libcephfs library ubuntu@distro-wallaby:~$ sudo add-apt-repository cloud-archive:wallaby Ubuntu Cloud Archive for OpenStack Wallaby More info: https://wiki.ubuntu.com/OpenStack/CloudArchive Press [ENTER] to continue or Ctrl-c to cancel adding it. ... ubuntu@distro-wallaby:~$ sudo apt update && sudo apt upgrade Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease Hit:3 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/wallaby InRelease Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease Reading package lists... Done Building dependency tree Reading state information... Done 44 packages can be upgraded. Run 'apt list --upgradable' to see them. Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following packages were automatically installed and are no longer required: libboost-context1.71.0 libboost-iostreams1.71.0 libboost-program-options1.71.0 libboost-thread1.71.0 libfreetype6 Use 'sudo apt autoremove' to remove them. The following NEW packages will be installed: liblua5.3-0 libpmem1 The following packages will be upgraded: bsdutils ceph-common cryptsetup cryptsetup-bin cryptsetup-initramfs cryptsetup-run fdisk initramfs-tools initramfs-tools-bin initramfs-tools-core libblkid1 libcephfs2 libcryptsetup12 libdrm-common libdrm2 libfdisk1 libmount1 librados2 libradosstriper1 librbd1 libsmartcols1 libuuid1 mount open-vm-tools python-apt-common python3-apt python3-ceph-argparse python3-ceph-common python3-cephfs python3-cryptography python3-distupgrade python3-httplib2 python3-importlib-metadata python3-openssl python3-rados python3-rbd python3-requests python3-six python3-urllib3 snapd ubuntu-advantage-tools ubuntu-release-upgrader-core util-linux uuid-runtime 44 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. 15 standard security updates Need to get 68.7 MB of archives. After this operation, 30.0 MB of additional disk space will be used. Do you want to continue? [Y/n] -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1960065 Title: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1960065] Re: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages
Stepping between UCA pockets is really akin to a distribution upgrade so I think its appropriate that only a dist-upgrade action would complete the transition correctly (or an install as detailed in #8). We could provide transitional packages but this would need to be done as patches ontop of the ceph packages from the associated source Ubuntu repository and as there is a clear path to complete an upgrade successfully I'm reticent to follow this path. ** Changed in: ceph (Ubuntu) Status: Triaged => Invalid ** Changed in: ceph (Ubuntu) Status: Invalid => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1960065 Title: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1960065] Re: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages
Thanks for mentioning this, Ed. I just checked and `apt install ceph` works correctly (removes python2 packages and upgrades ceph packages). $ sudo apt install ceph # step 8 below. ... The following packages will be REMOVED: python-cephfs python-rados python-rbd ... The following packages will be upgraded: ceph ceph-base ceph-common ceph-mgr ceph-mon ceph-osd libcephfs2 librados2 libradosstriper1 librbd1 python3-pkg-resource ... $ history | tail ... 3 sudo apt update 4 sudo apt install ceph 5 sudo add-apt-repository cloud-archive:rocky 6 sudo apt install ceph 7 sudo add-apt-repository cloud-archive:stein 8 sudo apt install ceph 9 apt list --upgradable ... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1960065 Title: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1960065] Re: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages
While I am aware that not everybody uses the charms to install ceph, I checked to see how they do this and they are in fact using apt install to perform an upgrade e.g. the ceph-mon charm does [1] using [2] which I believe will not behave the same as the "standard" way mentioned in comment #6 and is also not tested here in this bug. [1] https://github.com/openstack/charm-ceph-mon/blob/05a03bd10d885d161b07e6acf47d030549562768/lib/charms_ceph/utils.py#L2227 [2] https://github.com/juju/charm-helpers/blob/b53f741d1c6f34f26f889d79afaad838dc14fdfa/charmhelpers/fetch/ubuntu.py#L361 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1960065 Title: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1960065] Re: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages
** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1960065 Title: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1960065] Re: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages
Thanks Corey. Yes, this issue doesn't seem to be high priority. (I should have mentioned that :) It just turned out to be noticed for some other reason and I got curious enough. cheers, Mauricio -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1960065 Title: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1960065] Re: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages
Thanks for the patch Mauricio. I think it is standard to use full- upgrade or dist-upgrade on upgrades to new releases, so with that in mind I think this is low priority. Input from others is certainlly welcome though. For now I'm going to triage this. ** Changed in: ceph (Ubuntu) Status: Confirmed => Triaged ** Changed in: ceph (Ubuntu) Importance: Undecided => Low -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1960065 Title: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1960065] Re: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages
** Description changed:
- UCA Rocky has older versions than UCA Stein for some Ceph python2
- packages, e.g., python-cephfs.
+ Some ceph python2 packages have been dropped between rocky and stein.
+ (in Disco, from 13.2.1+dfsg1-0ubuntu3 to 13.2.2+dfsg1-0ubuntu1 [1]).
- This is an issue for versioned dependencies that require an exact
- version that isn't there.
+ The packages python-{cephfs,rados,rbd} have no upgrades, but have exact
+ versioned dependencies on packages that have upgrades.
- The apt solution is removing packages, but that's not allowed on `apt
- upgrade`, thus `apt` keeps back / doesn't upgrade some ceph packages
- (`apt full-upgrade` can remove/works fine.)
+ So, during `apt upgrade` the former block the latter from being
+ upgraded, which in turn block other ceph packages from being upgraded
+ (e.g., ceph-{common,mon,osd,mgr}).
- Example:
+ Note this isn't a problem on `apt full-upgrade` or `apt dist-upgrade` as
+ these allow package removals (`apt upgrade` does not), then `apt` just
+ decides to remove those 3 packages.
- $ apt-cache madison python-cephfs
- python-cephfs | 13.2.8-0ubuntu0.18.10.1~cloud0 |
http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main
amd64 Packages
- python-cephfs | 13.2.1+dfsg1-0ubuntu3~cloud0 |
http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/stein/main
amd64 Packages
- ...
+ It's possible to fix this with dummy transitional packages that remove
+ the exact versioned dependency (patch provided/tested in comment #5).
+
+ [1] https://launchpad.net/ubuntu/+source/ceph/13.2.2+dfsg1-0ubuntu1
+
+ Steps to Reproduce:
+ ---
Setup Bionic container:
- ---
- $ lxc launch ubuntu:bionic mfo-bionic-uca
- $ ssh mfo-bionic-uca.lxd
+ $ lxc launch ubuntu:bionic bionic-uca
+ $ lxc exec bionic-uca -- su - ubuntu
$ sudo apt --yes purge unattended-upgrades
$ sudo apt update
-
Install ceph from Bionic (Queens):
- ---
$ sudo apt --yes install ceph
$ dpkg -s ceph | grep Version
Version: 12.2.13-0ubuntu0.18.04.10
-
- Upgrade to ceph from UCA Rocky:
- ---
+ Upgrade to UCA Rocky:
$ sudo add-apt-repository --yes cloud-archive:rocky
$ sudo apt --dry-run upgrade
...
The following packages will be upgraded:
- ceph ceph-base ceph-common ceph-mgr ceph-mon ceph-osd libcephfs2 librados2
libradosstriper1 librbd1 python-cephfs python-pecan python-rados python-rbd
- python-webob python3-jsonpatch
+ ceph ceph-base ceph-common ceph-mgr ceph-mon ceph-osd libcephfs2 librados2
libradosstriper1 librbd1 python-cephfs python-pecan python-rados python-rbd
+ python-webob python3-jsonpatch
...
$ sudo apt --yes upgrade
$ dpkg -s ceph | grep Version
Version: 13.2.8-0ubuntu0.18.10.1~cloud0
-
- Upgrade to ceph from UCA Stein:
- ---
+ Upgrade to UCA Stein:
$ sudo add-apt-repository --yes cloud-archive:stein
$ sudo apt --dry-run upgrade
...
The following packages have been kept back:
- ceph ceph-base ceph-common ceph-mgr ceph-mon ceph-osd libcephfs2 librados2
libradosstriper1 librbd1
+ ceph ceph-base ceph-common ceph-mgr ceph-mon ceph-osd libcephfs2 librados2
libradosstriper1 librbd1
...
+
+ Issue above.
+
Check:
---
$ sudo apt -o Debug::pkgProblemResolver=1 --dry-run upgrade
- ...
- Entering ResolveByKeep 10%
+ ...
+ Entering ResolveByKeep 10%
- Dependencies are not satisfied for python-cephfs:amd64 <
13.2.8-0ubuntu0.18.10.1~cloud0 @ii mK Ib >
- Keeping package python-cephfs:amd64
- Dependencies are not satisfied for python-cephfs:amd64 <
13.2.8-0ubuntu0.18.10.1~cloud0 @ii mK Ib >
- Package python-cephfs:amd64 python-cephfs:amd64 Depends on
libcephfs2:amd64 < 13.2.8-0ubuntu0.18.10.1~cloud0 ->
13.2.9-0ubuntu0.19.04.1~cloud3 @ii umU > (= 13.2
- .8-0ubuntu0.18.10.1~cloud0)
- Keeping Package libcephfs2:amd64 due to Depends
+ Dependencies are not satisfied for python-cephfs:amd64 <
13.2.8-0ubuntu0.18.10.1~cloud0 @ii mK Ib >
+ Keeping package python-cephfs:amd64
+ Dependencies are not satisfied for python-cephfs:amd64 <
13.2.8-0ubuntu0.18.10.1~cloud0 @ii mK Ib >
+ Package python-cephfs:amd64 python-cephfs:amd64 Depends on libcephfs2:amd64
< 13.2.8-0ubuntu0.18.10.1~cloud0 -> 13.2.9-0ubuntu0.19.04.1~cloud3 @ii umU > (=
13.2
+ .8-0ubuntu0.18.10.1~cloud0)
+ Keeping Package libcephfs2:amd64 due to Depends
...
- Issue:
- -
+ Details:
+ ---
python-cephfs at version 13.2.__8__ has exact version dep on libcephfs2,
which is being upgraded to 13.2.__9__.
Why isn't python-cephfs being upgraded to 13.2.__9__ too?
AH, there's no 13.2.__9__ version for python-cephfs, only for libcephfs2.
So, it stopped on 13.2.__8__ in UCA Rocky, there's no 13.2.__9__ in UCA
Stein, as there's for libcephfs2.
-
- Why is that?
-
- OH, wait. UCA Stein does have python-cephfs, but in an _older_ version than
UCA Rocky: 13.2.__1__.
- (but it has
[Bug 1960065] Re: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages
Test Steps: --- From rocky to stein, Some packages are kept back (see '10 not upgraded'): $ sudo apt --dry-run --yes upgrade ... The following packages have been kept back: ceph ceph-base ceph-common ceph-mgr ceph-mon ceph-osd libcephfs2 librados2 libradosstriper1 librbd1 ... ... and 10 not upgraded. Local repo with the patched ceph packages from bionic-stein: $ dpkg-scanpackages . > Packages $ echo 'deb [trusted=yes] file:/home/ubuntu ./' | sudo tee /etc/apt/sources.list.d/local.list $ sudo apt update Here they are, see first line: $ apt-cache madison python-cephfs python-cephfs | 13.2.9-0ubuntu0.19.04.1~cloud3+lp19600651 | file:/home/ubuntu ./ Packages python-cephfs | 13.2.8-0ubuntu0.18.10.1~cloud0 | http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 Packages python-cephfs | 12.2.13-0ubuntu0.18.04.10 | http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages python-cephfs | 12.2.13-0ubuntu0.18.04.10 | http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages python-cephfs | 12.2.4-0ubuntu1 | http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages No packages are kept back now (see '0 not upgraded'): $ sudo apt --dry-run upgrade ... The following packages were automatically installed and are no longer required: ... python-cephfs ... python-rados python-rbd ... Use 'sudo apt autoremove' to remove them. ... The following packages will be upgraded: ceph ceph-base ceph-common ceph-mgr ceph-mon ceph-osd ibverbs-providers libcephfs2 libibverbs1 libnuma1 librados2 libradosstriper1 librbd1 python-cephfs python-pkg-resources python-rados python-rbd python-setuptools python-webob python3-jwt python3-pkg-resources ... and 0 not upgraded. And autoremove clears all the python2 stuff up. ubuntu@mfo-b-uca2:~/ceph-debs$ sudo apt autoremove ... Reading state information... Done The following packages will be REMOVED: formencode-i18n libjs-sphinxdoc libjs-underscore libpython-stdlib libpython2.7 libpython2.7-minimal libpython2.7-stdlib python python-asn1crypto python-bcrypt python-bs4 python-cephfs python-certifi python-cffi-backend python-chardet python-cherrypy3 python-cryptography python-dnspython python-enum34 python-formencode python-html5lib python-idna python-ipaddress python-jinja2 python-logutils python-lxml python-mako python-markupsafe python-minimal python-openid python-openssl python-paste python-pastedeploy python-pastescript python-pecan python-pkg-resources python-prettytable python-pyinotify python-rados python-rbd python-repoze.lru python-requests python-routes python-scgi python-setuptools python-simplegeneric python-simplejson python-singledispatch python-six python-tempita python-urllib3 python-waitress python-webencodings python-webob python-webtest python-werkzeug python2.7 python2.7-minimal 0 upgraded, 0 newly installed, 58 to remove and 0 not upgraded. ... ** Patch added: "lp1960065_ceph_bionic-stein.debdiff" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/+attachment/5559214/+files/lp1960065_ceph_bionic-stein.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1960065 Title: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1960065] Re: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages
Hi Corey, Yes, this is limited to `apt upgrade` per the restriction it has on package removal. I just tested the following debdiff successfully, if that helps. cheers, Mauricio -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1960065 Title: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1960065] Re: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages
Hi Mauricio, I just stepped through the following successfully. Is this limited to using 'apt upgrade'? root@bionic-uca:~# history 1 sudo apt --yes purge unattended-upgrades 2 apt update 3 sudo apt --yes install ceph 4 dpkg -s ceph | grep Version 5 sudo add-apt-repository --yes cloud-archive:rocky 6 man apt 7 sudo apt dist-upgrade 8 dpkg -s ceph | grep Version 9 sudo add-apt-repository --yes cloud-archive:stein 10 dpkg -s ceph | grep Version 11 sudo apt dist-upgrade 12 apt policy python3-ceph 13 apt policy python3-cephfs 14 apt policy python-cephfs And the last 2 steps show: root@bionic-uca:~# apt policy python3-cephfs python3-cephfs: Installed: 13.2.9-0ubuntu0.19.04.1~cloud3 Candidate: 13.2.9-0ubuntu0.19.04.1~cloud3 root@bionic-uca:~# apt policy python-cephfs python-cephfs: Installed: (none) Candidate: 13.2.8-0ubuntu0.18.10.1~cloud0 Thanks, Corey -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1960065 Title: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1960065] Re: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: ceph (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1960065 Title: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1960065] Re: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages
Thanks to ~pponnuvel for reporting the issue with `apt upgrade` internally. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1960065 Title: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
