Re: usrmerge on root NFS will not be run automatically
Very unimpressed with the so called "fix" for #842145 of just blocking running the script on nfs mount rather than fixing the script to work properly with nfs. The problem with the script is that it does not ignore the .nfs* files. An explanation of these files is available here: https://unix.stackexchange.com/questions/348315/nfs-mount-device-or-resource-busy I am not a programmer so will not share my crappy attempt at getting the script to work. I will describe a workaround I did and maybe that may help someone come up with their own workaround for system with nfs root (where the nfs server is not running linux). Or even better maybe the package maintainer might adjust the script within usrmerge to work properly with nfs using these ideas. - Starting from bullseye (I rolled back to a snapshot pre-install of usrmerge), downloaded src of usrmerge to get access to the convert-usrmerge script. - modify script to try and get it to ignore/avoid changing any objects with ".nfs000" in its name - run the script so it hopefully completes successfully (In my case it did not fully complete. It made most of the changes but I still had some directories in root such as /lib - probably from mistakes in my code changes) - rebooted nfs client to clear open file handles on nfs which would remove .nfs000* files - ran the original unmodified script which this time completed successfully including converting eg /lib to a symlink. I think this worked as most of the files had moved and symlinks created so there was not much left for it to do except sort out the remaining few moves/symlinks. - installed usrmerge package which this time completed (it detected the merge completed and did not complain about nfs) >From there I expect it should be safe to upgrade to bookworm without usrmerge breaking the upgrade (not tested yet). good luck
Re: usrmerge on root NFS will not be run automatically
On Thu, 14 Sep 2023 22:17:35 +0200 Marco wrote: > If I screw with this I'd prefer to do it at night or on a weekend to > keep the systems running during business hours. Followup: I went through the list and resolved each conflict manually. I launched usrmerge after every change and deleted/merged the offending files. Note that I ran usrmerge on the individual hosts themselves, on NFS root. Although usrmerge complained that this won't work, it somehow did. Systems rebooted, all came up fine, no broken packages and the programs are working. Thanks for all the support. Case solved. Marco
Re: usrmerge on root NFS will not be run automatically
On Fri, 15 Sep 2023 17:55:06 + Andy Smith wrote: > I haven't followed this thread closely, but is my understanding > correct: > > - You have a FreeBSD NFS server with an export that is a root > filesystem of a Debian 11 install shared by multiple clients Almost. It's not *one* Debian installation, it's many (diskless workstations PXE boot). Each host has it's own root on the NFS. Some stuff is shared, but that's not relevant here. > - You're trying to do an upgrade to Debian 12 running on one of the > clients. Not on one on *all* clients. > - It tries to do a usrmerge but aborts because NFS is not supported > by that script? Correct. Strangely the usrmerge script succeeded on one host. But on all others it throws errors. Either relating to NFS being not supported or because of duplicate files. > If so, have you tried reporting a bug on this yet? No I haven't. As far as I understand it's a known issue and the developer has decided to just have the script fail on NFS. > If you don't get anywhere with that, I don't think you have much > choice except to take away the root directory tree to a Linux host, > chroot into it and complete the merge there, then pack it up again > and bring it back to your NFS server. Which is very far from ideal. I'll try to solve the conflicts manually. If that fails, that's what I have to do, I guess. I didn't expect that level of fiddling with system files for a simple upgrade. But hey, here we are now. > The suggestions about running a VM on the NFS server probably aren't > going to work as you won't be able to take the directory tree out of > use and export it as a block device to the VM. Indeed. > The option of making the usrmerge script work from FreeBSD might not > be too technically challenging but I wouldn't want to do it without > assistance from the Debian developers responsible for the script. I won't do that. I don't speak Perl and will not rewrite the usrmerge script. Marco
Re: usrmerge on root NFS will not be run automatically
Hello, On Fri, Sep 15, 2023 at 01:52:27PM +0200, Marco wrote: > On Thu, 14 Sep 2023 16:43:09 -0400 > Dan Ritter wrote: > > Each of these things could be rewritten to be compatible with > > FreeBSD; I suspect it would take about twenty minutes to an hour, > > most of it testing, for someone who was familiar with FreeBSD's > > userland > > I'm not going down that route. I haven't followed this thread closely, but is my understanding correct: - You have a FreeBSD NFS server with an export that is a root filesystem of a Debian 11 install shared by multiple clients - You're trying to do an upgrade to Debian 12 running on one of the clients. - It tries to do a usrmerge but aborts because NFS is not supported by that script? If so, have you tried reporting a bug on this yet? It seems like an interesting problem which although being quite a corner case, might spark the interest of the relevant Debian developers. If you don't get anywhere with that, I don't think you have much choice except to take away the root directory tree to a Linux host, chroot into it and complete the merge there, then pack it up again and bring it back to your NFS server. Which is very far from ideal. The suggestions about running a VM on the NFS server probably aren't going to work as you won't be able to take the directory tree out of use and export it as a block device to the VM. Or rather, you could do that, but it's probably not quicker/easier than the method of taking a copy of it elsewhere then bringing it back. The option of making the usrmerge script work from FreeBSD might not be too technically challenging but I wouldn't want to do it without assistance from the Debian developers responsible for the script. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: usrmerge on root NFS will not be run automatically
> So the file in /lib appears to be newer. So what to do? Can I delete > the one in /usr/lib ? Yes. Stefan
Re: usrmerge on root NFS will not be run automatically
On Thu, 14 Sep 2023 16:54:27 -0400 Stefan Monnier wrote: > Still going on with this? I am. > Have you actually looked at those two files: > > /lib/udev/rules.d/60-libsane1.rules and > /usr/lib/udev/rules.d/60-libsane1.rules > > to see if they're identical or not and to see if you might have an > idea how to merge them? Yes, I did. On some hosts they are identical, on others they're different. That's why I asked how to handle that. > `usrmerge` did give you a pretty clear explanation of the problem it's > facing (AFAIC) It does indeed. > and I believe it should be very easy to address it Everything is easy if you only know how to do it. As I said, on some hosts they are identical. So what to do? Can I delete one of them? If yes, which one? On other hosts they differ, here the first lines: /lib/ # This file was generated from description files (*.desc) # by sane-desc 3.6 from sane-backends 1.1.1-debian # # udev rules file for supported USB and SCSI devices # # For the list of supported USB devices see /lib/udev/hwdb.d/20-sane.hwdb # # The SCSI device support is very basic and includes only # scanners that mark themselves as type "scanner" or # SCSI-scanners from HP and other vendors that are entitled "processor" # but are treated accordingly. # # If your SCSI scanner isn't listed below, you can add it to a new rules # file under /etc/udev/rules.d/. # # If your scanner is supported by some external backend (brother, epkowa, # hpaio, etc) please ask the author of the backend to provide proper # device detection support for your OS # # If the scanner is supported by sane-backends, please mail the entry to # the sane-devel mailing list (sane-de...@alioth-lists.debian.net). # ACTION=="remove", GOTO="libsane_rules_end" … /usr/lib/ # This file was generated from description files (*.desc) # by sane-desc 3.6 from sane-backends 1.0.31-debian # # udev rules file for supported USB and SCSI devices # # For the list of supported USB devices see /lib/udev/hwdb.d/20-sane.hwdb # # The SCSI device support is very basic and includes only # scanners that mark themselves as type "scanner" or # SCSI-scanners from HP and other vendors that are entitled "processor" # but are treated accordingly. # # If your SCSI scanner isn't listed below, you can add it to a new rules # file under /etc/udev/rules.d/. # # If your scanner is supported by some external backend (brother, epkowa, # hpaio, etc) please ask the author of the backend to provide proper # device detection support for your OS # # If the scanner is supported by sane-backends, please mail the entry to # the sane-devel mailing list (sane-de...@alioth-lists.debian.net). # ACTION!="add", GOTO="libsane_rules_end" … So the file in /lib appears to be newer. So what to do? Can I delete the one in /usr/lib ? > (no need to play with anything funny like setting up a VM or > mounting the disk from some other system). Which is good because that's not that easy, apparently. Thank you for your replies and support regarding this matter. Marco
Re: usrmerge on root NFS will not be run automatically
On Thu, 14 Sep 2023 16:43:09 -0400 Dan Ritter wrote: > The heart of the convert-usrmerge perl script is pretty > reasonable. However: > > […] > > Similarly, there are calls to stat and du which probably have > some incompatibilities. > > The effect of running this would be fairly safe, but also not do > anything: you would get some errors and then it would die. Ok, then I'll not try that. Would be a waste of time. > Each of these things could be rewritten to be compatible with > FreeBSD; I suspect it would take about twenty minutes to an hour, > most of it testing, for someone who was familiar with FreeBSD's > userland I'm not going down that route. Marco
Re: usrmerge on root NFS will not be run automatically
Still going on with this? Have you actually looked at those two files: /lib/udev/rules.d/60-libsane1.rules and /usr/lib/udev/rules.d/60-libsane1.rules to see if they're identical or not and to see if you might have an idea how to merge them? [ as I suggested a week ago. ] `usrmerge` did give you a pretty clear explanation of the problem it's facing (AFAIC) and I believe it should be very easy to address it (no need to play with anything funny like setting up a VM or mounting the disk from some other system). If you're not sure what to do with those two files, show them to us. Stefan Marco [2023-09-14 20:28:59] wrote: > On Thu, 14 Sep 2023 13:20:09 -0400 > Dan Ritter wrote: > >> > FreeBSD (actually a TrueNAS appliance) >> >> If it supports the 9P share system, Debian can mount that with >> -t 9p. >> >> I don't know whether TrueNAS enabled that. > > No it does not. I just confirmed, the only choices are raw disk > access (ZVOL), NFS and Samba. > > However, usrmerge is a perl script. Can I run it on the server > (after chroot'ing) in a jail (under FreeBSD)? Or does this mess > things up? Just a thought. > > Marco
Re: usrmerge on root NFS will not be run automatically
On Thu, 14 Sep 2023 15:01:50 -0400 Dan Ritter wrote: > Is this a mission-critical server? I'd say so, yes. It's not one single server. It's *all* workstations. > i.e. will screwing it up for a day cause other people to be upset Yes, because no one can use their computer. > or you to lose money? Yes. If I screw with this I'd prefer to do it at night or on a weekend to keep the systems running during business hours. > Do you have a good, current backup? Yes. > Since it's TrueNAS, I assume you are using ZFS, so: have you sent > snapshots to some other device recently? Yes, every three hours. And one before every system upgrade.
Re: usrmerge on root NFS will not be run automatically
On Thu, 14 Sep 2023 13:20:09 -0400 Dan Ritter wrote: > > FreeBSD (actually a TrueNAS appliance) > > If it supports the 9P share system, Debian can mount that with > -t 9p. > > I don't know whether TrueNAS enabled that. No it does not. I just confirmed, the only choices are raw disk access (ZVOL), NFS and Samba. However, usrmerge is a perl script. Can I run it on the server (after chroot'ing) in a jail (under FreeBSD)? Or does this mess things up? Just a thought. Marco
Re: usrmerge on root NFS will not be run automatically
On Thu, 14 Sep 2023 11:00:17 -0400 Dan Ritter wrote: > What VM software are you using bhyve …which I know very little about. It's supported on the server, I've tried it, set up a VM, it works. But the server is mainly serving NFS shares to various clients. > and what's the OS on which that runs? FreeBSD (actually a TrueNAS appliance) Marco
Re: usrmerge on root NFS will not be run automatically
Marco wrote: > On Fri, 8 Sep 2023 12:26:38 -0400 > Dan Ritter wrote: > > * have the VM mount the filesystem directly > > How? I can only attach devices (=whole disks) to the VM or mount the > FS via NFS. I can't attach it as a device because it's not a device, > rather than a directory with the root file systems of several hosts > directly on the server. So that doesn't work. What VM software are you using, and what's the OS on which that runs? -dsr-
Re: usrmerge on root NFS will not be run automatically
On Fri, 8 Sep 2023 12:26:38 -0400 Dan Ritter wrote: > Can you start a temporary VM directly on the server? I just checked. I can, yes. > If so, you can > * stop your remote Debian machine Ok, no problem. > * run a Debian rescue image in the VM on the NFS server No problem. > * have the VM mount the filesystem directly How? I can only attach devices (=whole disks) to the VM or mount the FS via NFS. I can't attach it as a device because it's not a device, rather than a directory with the root file systems of several hosts directly on the server. So that doesn't work. This leaves me with an NFS mount in the VM. But NFS mounts are not supported by usrmerge, that's the whole issue I'm facing here. So this VM-on-the-server idea doesn't work in my case or am I missing something here? Another question: if usrmerge complains that the file is present in /lib as well as /usr/lib, what's the correct thing to do if i) the files are identical ii) the files are different ? Regards Marco
Re: usrmerge on root NFS will not be run automatically
Hello, El dom., 10 sept. 2023 21:55, Marco escribió: > On Fri, 8 Sep 2023 12:26:38 -0400 > Dan Ritter wrote: > > > > That is quite an involved task. I didn't expect such fiddling for a > > > simple OS update. I'm a bit worried that the permissions and owners > > > go haywire when I copy stuff directly off the server onto a VM and > > > back onto the server. Is there a recommended procedure or > > > documentation available? > > > > Can you start a temporary VM directly on the server? > > I might actually. I'll have to check the following days. > > > If so, you can > > * stop your remote Debian machine > > * run a Debian rescue image in the VM on the NFS server > > * have the VM mount the filesystem directly > > * chroot, run usrmerge > > * unmount > > Ok, that's also quite a task, but it seems less error-prone than > copying a bunch of system files across the network and hope for the > best. I'll try. > > Marco > Maybe you can open a new bug asking for a better documentation or what should be done in this case. Maybe dpkg -L with both files can help to clarify what should be done Regards >
Re: usrmerge on root NFS will not be run automatically
On Fri, 8 Sep 2023 12:26:38 -0400 Dan Ritter wrote: > > That is quite an involved task. I didn't expect such fiddling for a > > simple OS update. I'm a bit worried that the permissions and owners > > go haywire when I copy stuff directly off the server onto a VM and > > back onto the server. Is there a recommended procedure or > > documentation available? > > Can you start a temporary VM directly on the server? I might actually. I'll have to check the following days. > If so, you can > * stop your remote Debian machine > * run a Debian rescue image in the VM on the NFS server > * have the VM mount the filesystem directly > * chroot, run usrmerge > * unmount Ok, that's also quite a task, but it seems less error-prone than copying a bunch of system files across the network and hope for the best. I'll try. Marco
Re: usrmerge on root NFS will not be run automatically
> root@foobar:~# /usr/lib/usrmerge/convert-usrmerge > > FATAL ERROR: > Both /lib/udev/rules.d/60-libsane1.rules and > /usr/lib/udev/rules.d/60-libsane1.rules exist. The problem is that "usrmerge" needs to unify those two and doesn't know how. So you need to do it by hand. E.g. get rid of one of those two (or maybe if you can make them 100% identical `usrmerge` will be happy as well). Stefan
Re: usrmerge on root NFS will not be run automatically
On Fri, 8 Sep 2023 16:55:23 +0200 zithro wrote: > On 08 Sep 2023 12:54, Marco wrote: > >Warning: NFS detected, /usr/lib/usrmerge/convert-usrmerge will > > not be run automatically. See #842145 for details. > > Read : > - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=842145 “I repeated it a few times. I had to restart various services in between retries (I think I restarted everything by the end). Eventually it succeeded.” I tried it 30 times to no avail. The report doesn't offer another solution. > - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1039522 “instead of converting the client, convert the server first.” I don't want to convert the server. The server is running fine and has no issues. I don't have a clue what this has to do with the server. “So there is a workaround when your NFS server is a Linux machine and you may use chroot on it, at least.” The server doesn't run Linux. So also no solution there. > carefully copy the files over a Linux machine, chroot+convert > there, then move back to the NFS server. That is quite an involved task. I didn't expect such fiddling for a simple OS update. I'm a bit worried that the permissions and owners go haywire when I copy stuff directly off the server onto a VM and back onto the server. Is there a recommended procedure or documentation available? > Can help: > https://unix.stackexchange.com/questions/312218/chroot-from-freebsd-to-linux I cannot install stuff on the server unfortunately. Thanks for your quick reply. Marco
Re: usrmerge on root NFS will not be run automatically
On 08 Sep 2023 12:54, Marco wrote:
Warning: NFS detected, /usr/lib/usrmerge/convert-usrmerge will not be run
automatically. See #842145 for details.
Read :
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=842145
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1039522
The solution would be :
"convert-usrmerge can be run in a chroot on the NFS server"
If your NFS server is not Linux-based, carefully copy the files over a
Linux machine, chroot+convert there, then move back to the NFS server.
Make backups ;)
Can help:
https://unix.stackexchange.com/questions/312218/chroot-from-freebsd-to-linux
As per the why, I don't know.
Maybe the symlink handling by NFS ?
Reading the script "/usr/lib/usrmerge/convert-usrmerge", you get the
reasons why it fails :
142 # The other cases are more complex and there are some corner
cases that
143 # we do not try to resolve automatically.
144
145 # both source and dest are links
[...]
168 # the source is a link
[...]
175 # the destination is a link
[...]
191 # both source and dest are directories
192 # this is the second most common case
You may change the script to detect where/why it fails, so edit
fatal("Both $n and /usr$n exist");
to
fatal("Both $n and /usr$n exist - err line 145");
fatal("Both $n and /usr$n exist - err line 168");
Also found this on
"https://en.opensuse.org/openSUSE:Usr_merge#Known_Problems; :
- "File systems that do not support RENAME_EXCHANGE such as ZFS or NFS
cannot perform live conversion (Bug 1186637)"
- "Conversion fails if there's a mount point below
(/usr)/{bin,sbin,lib,lib64}"
Good luck
--
++
zithro / Cyril
