Bug#858331: dpkg -S doesn't work properly with usrmerge
Control: tags -1 - patch On Sun, 2022-07-17 at 00:26:46 +0100, Luca Boccassi wrote: > Control: tags -1 patch > The attached patch from user uau is a working solution that would allow > to lift this moratorium and also solve this bug. It is shared as-is, > with permission and under the same license as src:dpkg. As I mentioned at the time, this patch is a major hack, it is conceptually broken as it hardcodes pathnames and remaps them behind the involved packages back (action at a distance), it breaks interfaces, it's incomplete, etc. Even the included README says as much. So, no, it's not a "working solution"… Guillem
Bug#858331: dpkg -S doesn't work properly with usrmerge
Control: tags -1 patch Hello, On top of the issue described in this bug, the Debian CTTE in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#110 also ruled as follow because of the same missing functionality in dpkg: "Moratorium on moving files' logical locations into /usr <...> The Technical Committee recommends that during the Debian 12 development cycle, the maintainers of individual packages should not proactively move files from the root filesystem to the corresponding locations in /usr in the data.tar.* of packages. Files that were in /usr in the Debian 11 release should remain in /usr, while files that were in /bin, /lib* or /sbin in the Debian 11 release should remain in those directories. If files were moved from /bin, /lib* or /sbin into /usr since the Debian 11 release, they should be moved back to their Debian 11 locations." The attached patch from user uau is a working solution that would allow to lift this moratorium and also solve this bug. It is shared as-is, with permission and under the same license as src:dpkg. -- Kind regards, Luca Boccassi diff --git a/README.usrmerge b/README.usrmerge new file mode 100644 index 0..ee8b1646d --- /dev/null +++ b/README.usrmerge @@ -0,0 +1,67 @@ +This is a proof of concept version of dpkg with explicit usrmerge +support. This means the following file path conversions are done in +various places: +/lib -> /usr/lib +/bin ->/ /usr/bin +/sbin -> /usr/sbin + +Usrmerge also includes other links like "/libx32 -> /usr/libx32" +depending on platform, but those are not handled. This shouldn't +matter much for testing since few packages contain those anyway, and +it does not matter much whether dpkg is aware of aliasing for those +paths. + +The conversion is done at least in the following cases: +- installing packages: any install path contained in .deb +- reading list of existing installed files (/var/lib/dpkg/info/*.list) +- dpkg-query (or dpkg) -S patterns beginning with '/' +- reading existing statoverride locations, adding/removing +- dpkg-statoverride --list glob patterns beginning with '/' +- reading existing divert locations, adding/removing +- dpkg-divert --list glob patterns beginning with '/' +- file trigger locations +- .md5sums file targets +- .conffiles (should any exist with such paths) + +These conversions mean that dpkg should notice any file conflicts +between /bin/x and /usr/bin/x, since both are now converted to the +same literal path /usr/bin/x. Most maintainer scripts and tools should +continue working, as the paths they use are automatically converted. +However, there is some potential for maintainer scripts or other tools +to break if they care about the exact paths returned from queries like +"dpkg-divert --truename" or "dpkg-query -S" (since those will now +refer to the real location under /usr). + +No separate database conversion step is required. Existing paths are +converted as they are read from disk. Any files generated and written +by dpkg will have the new paths, since they will have been converted +before writing. This means newly created .list files use converted +locations, but .triggers files do not (they are copied verbatim from +the package, not generated from already parsed data). The +dpkg-generated triggers/File file will contain converted locations +when it is written. (Applies at least to udev /lib/udev/hwdb.d). + +The query conversions mean that for example "dpkg -S '/bin/ba*'" +output will include "/usr/bin/bash" (as the query will be converted to +"/usr/bin/ba*"), but "dpkg -S '/???/bash'" will find no matches since +the query itself can not be converted and the pattern does not match +the actual path "/usr/bin/bash". + +There's a minor cosmetic issue where the conversion does not add the +"/usr" directory itself, so theoretically a package containing files +only under /lib would be listed as containing the directory "/usr/lib" +but not "/usr", but this shouldn't matter in practice (Debian packages +contain /usr/share/doc/* anyway). Maybe it could cause issues when +bootstrapping a new system in case this means trying to create +/usr/bin before /usr? Could be worked around for example by just +hardcoding "/usr" as contents of every package (in dpkg), more +"correct" (and complex) fix, or disabling usrmerge logic for initial +bootstrap as below. + +The usrmerge changes can be disabled by creating the file +"/var/lib/dpkg/DISABLE_USRMERGE_LOGIC". There's no logic to create +such a file automatically or to sanity check the existence of usrmerge +links if the file does not exist. Expect things to break if you run +this dpkg version on a system without the usrmerge links and without +creating the file, or if you create the file on an usrmerged system +after already running this dpkg without it existing. diff --git a/debian/dpkg.docs b/debian/dpkg.docs index 308db3568..ebdbd8139 100644 --- a/debian/dpkg.docs +++ b/debian/dpkg.docs @@ -1,5 +1,6 @@ AUTHORS THANKS +README.usrmerge
Bug#858331: dpkg -S doesn't work properly with usrmerge
Control: unmerge -1 Control: severity -1 wishlist Control: retitle -1 dpkg-query: -S does not work with reverse symlinks from /usr-merge On Wed, 2017-03-22 at 16:38:50 +1100, Russell Coker wrote: > On Wednesday, 22 March 2017 4:41:32 AM AEDT Guillem Jover wrote: > > Not really, this case is the reverse, which is searching for the > > canonical path on the filesystem, but which is not the canonical > > one on the dpkg db. > > > > Solving this from the dpkg side, is non-trivial if not impractical. > > It would require either scanning the whole filesystem to find aliased > > pathnames. Or hardcoding the mappings, which would be encoding local > > policy into dpkg somehow, which it might get out-of-sync with the > > filesystem contents. Or possibly other ugly things. > > /etc/selinux/default/contexts/files/file_contexts.subs_dist has the following > contents in the next policy for Unstable: > /bin /usr/bin > /lib /usr/lib > /lib32 /usr/lib > /lib64 /usr/lib > /libx32 /usr/lib > /sbin /usr/sbin > /etc/init.d /etc/rc.d/init.d > /lib/systemd /usr/lib/systemd > /run/lock /var/lock > /usr/lib32 /usr/lib > /usr/lib64 /usr/lib > /usr/libx32 /usr/lib > /usr/local/lib32 /usr/lib > /usr/local/lib64 /usr/lib > /usr/local/lib /usr/lib > > Why not have a directory for config files listing equivalent paths for the > purpose of dpkg -S etc? Above is the equivalency file for SE Linux, this is > needed because SE Linux uses the canonical path name for the initial > labelling > of security contexts. To do that we need to know all the path names. Now > the > above isn't going to suit dpkg -S exactly. With SE Linux labelling we > consider lib, lib32, lib64, and libx32 to be equivalent as the same file name > in multiple directories has the same security implications but obviously not > the same dpkg use. > > If we had a directory for subs files for the purpose of dpkg then usrmerge > and > anything else with similar operations could install a file to make dpkg -S > work. Hmmm, ok I guess, a "solution" could be: * Add a new option "--map-pathname a b" or similar. * When parsing the command-line or the config file fragments, check whether the mapping corresponds to an existing reverse symlink, and if not either warn or error out, so that these never get out of sync. * For each file in the db, match normally first, and otherwise try each mapped pathname; if a mapped pathname matches probably warn the user, and print the original pathname from the db (not the mapped one). This could go into /etc/dpkg/dpkg.cfg.d/usrmerge or whaterver, and more importantly it should not affect users that do not use any such hack. It still feels somehow wrong, and most probably it should not be enabled by default, because if this is done transparently these paths might end up on packages and similar, which would break on systems not /usr-merged. And if it is not enabled by default, people will still complain that it does not work out of the box anyway, so, as stated before it's still impractical. > > I covered this in #848622. So, unfortunately, this part, I don't think > > is fixable. And as I've stated before, while I think the idea behind > > merged-/usr is fine, the way it's been deployed is not, but this was > > ignored as a non-issue, so… > > > > I think this should be wontfixed, and closed. But if someone can come > > up with a clean way to handle this, I'm open to suggestions. > > It should not be closed. Set it to wontfix for the moment if you must, but > please leave it open so it's a known issue (otherwise it will keep getting > reported) and so that anyone else who has any ideas can add them. Well the usrmerge bug could always be marked as affecting dpkg, but ok, unmerged and massaged the metadata. Thanks, Guillem
Bug#858331: dpkg -S doesn't work properly with usrmerge
On Wednesday, 22 March 2017 4:41:32 AM AEDT Guillem Jover wrote: > > > # dpkg -S /usr/lib/systemd/libsystemd-shared-232.so > > > dpkg-query: no path found matching pattern > > > /usr/lib/systemd/libsystemd-shared-232.so # dpkg -S > > > /lib/systemd/libsystemd-shared-232.so > > > systemd: /lib/systemd/libsystemd-shared-232.so > > > # ls -l /lib > > > lrwxrwxrwx. 1 root root 7 Jan 5 15:04 /lib -> usr/lib > > > > > > I think that dpkg should know that those 2 filenames are the same and > > > return the package systemd in both cases. > > > > That's a long-standing problem, first reported 15 years ago. > > Not really, this case is the reverse, which is searching for the > canonical path on the filesystem, but which is not the canonical > one on the dpkg db. > > Solving this from the dpkg side, is non-trivial if not impractical. > It would require either scanning the whole filesystem to find aliased > pathnames. Or hardcoding the mappings, which would be encoding local > policy into dpkg somehow, which it might get out-of-sync with the > filesystem contents. Or possibly other ugly things. /etc/selinux/default/contexts/files/file_contexts.subs_dist has the following contents in the next policy for Unstable: /bin /usr/bin /lib /usr/lib /lib32 /usr/lib /lib64 /usr/lib /libx32 /usr/lib /sbin /usr/sbin /etc/init.d /etc/rc.d/init.d /lib/systemd /usr/lib/systemd /run/lock /var/lock /usr/lib32 /usr/lib /usr/lib64 /usr/lib /usr/libx32 /usr/lib /usr/local/lib32 /usr/lib /usr/local/lib64 /usr/lib /usr/local/lib /usr/lib Why not have a directory for config files listing equivalent paths for the purpose of dpkg -S etc? Above is the equivalency file for SE Linux, this is needed because SE Linux uses the canonical path name for the initial labelling of security contexts. To do that we need to know all the path names. Now the above isn't going to suit dpkg -S exactly. With SE Linux labelling we consider lib, lib32, lib64, and libx32 to be equivalent as the same file name in multiple directories has the same security implications but obviously not the same dpkg use. If we had a directory for subs files for the purpose of dpkg then usrmerge and anything else with similar operations could install a file to make dpkg -S work. > I covered this in #848622. So, unfortunately, this part, I don't think > is fixable. And as I've stated before, while I think the idea behind > merged-/usr is fine, the way it's been deployed is not, but this was > ignored as a non-issue, so… > > I think this should be wontfixed, and closed. But if someone can come > up with a clean way to handle this, I'm open to suggestions. It should not be closed. Set it to wontfix for the moment if you must, but please leave it open so it's a known issue (otherwise it will keep getting reported) and so that anyone else who has any ideas can add them. -- My Main Blog http://etbe.coker.com.au/ My Documents Bloghttp://doc.coker.com.au/
Bug#858331: dpkg -S doesn't work properly with usrmerge
Hi! On Tue, 2017-03-21 at 17:11:58 +0100, Sven Joachim wrote: > Control: forcemerge 134758 -1 > > On 2017-03-21 12:11 +, Russell Coker wrote: > > Package: dpkg > > Version: 1.18.23 > > Severity: normal > > > > # dpkg -S /usr/lib/systemd/libsystemd-shared-232.so > > dpkg-query: no path found matching pattern > > /usr/lib/systemd/libsystemd-shared-232.so > > # dpkg -S /lib/systemd/libsystemd-shared-232.so > > systemd: /lib/systemd/libsystemd-shared-232.so > > # ls -l /lib > > lrwxrwxrwx. 1 root root 7 Jan 5 15:04 /lib -> usr/lib > > > > I think that dpkg should know that those 2 filenames are the same and return > > the package systemd in both cases. > > That's a long-standing problem, first reported 15 years ago. Not really, this case is the reverse, which is searching for the canonical path on the filesystem, but which is not the canonical one on the dpkg db. Solving this from the dpkg side, is non-trivial if not impractical. It would require either scanning the whole filesystem to find aliased pathnames. Or hardcoding the mappings, which would be encoding local policy into dpkg somehow, which it might get out-of-sync with the filesystem contents. Or possibly other ugly things. I covered this in #848622. So, unfortunately, this part, I don't think is fixable. And as I've stated before, while I think the idea behind merged-/usr is fine, the way it's been deployed is not, but this was ignored as a non-issue, so… I think this should be wontfixed, and closed. But if someone can come up with a clean way to handle this, I'm open to suggestions. Thanks, Guillem
Bug#858331: dpkg -S doesn't work properly with usrmerge
Control: forcemerge 134758 -1 On 2017-03-21 12:11 +, Russell Coker wrote: > Package: dpkg > Version: 1.18.23 > Severity: normal > > # dpkg -S /usr/lib/systemd/libsystemd-shared-232.so > dpkg-query: no path found matching pattern > /usr/lib/systemd/libsystemd-shared-232.so > # dpkg -S /lib/systemd/libsystemd-shared-232.so > systemd: /lib/systemd/libsystemd-shared-232.so > # ls -l /lib > lrwxrwxrwx. 1 root root 7 Jan 5 15:04 /lib -> usr/lib > > I think that dpkg should know that those 2 filenames are the same and return > the package systemd in both cases. That's a long-standing problem, first reported 15 years ago. Cheers, Sven
Bug#858331: dpkg -S doesn't work properly with usrmerge
Package: dpkg Version: 1.18.23 Severity: normal # dpkg -S /usr/lib/systemd/libsystemd-shared-232.so dpkg-query: no path found matching pattern /usr/lib/systemd/libsystemd-shared-232.so # dpkg -S /lib/systemd/libsystemd-shared-232.so systemd: /lib/systemd/libsystemd-shared-232.so # ls -l /lib lrwxrwxrwx. 1 root root 7 Jan 5 15:04 /lib -> usr/lib I think that dpkg should know that those 2 filenames are the same and return the package systemd in both cases. -- System Information: Debian Release: 9.0 APT prefers unstable APT policy: (500, 'unstable') Architecture: amd64 (x86_64) Kernel: Linux 4.9.0-2-amd64 (SMP w/2 CPU cores) Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) Versions of packages dpkg depends on: ii libbz2-1.0 1.0.6-8.1 ii libc62.24-9 ii liblzma5 5.2.2-1.2+b1 ii libselinux1 2.6-3 ii tar 1.29b-1.1 ii zlib1g 1:1.2.8.dfsg-5 dpkg recommends no packages. Versions of packages dpkg suggests: ii apt1.4~rc2 pn debsig-verify -- no debconf information
