Re: libdmx removal incomplete?
On 2021/12/01 02:05, Alexander wrote: > On 2021/11/30 8:14, Stuart Henderson wrote: > > On 2021-11-29, Amit Kulkarni wrote: > > > On Sun, Nov 28, 2021 at 5:17 PM Alexander wrote: > > >> Just to gauge what to expect from this and whether I did this wrong: > > >> After configuring /etc/sysclean.ignore I get 3382 files of which 3274 > > >> are in /usr/X11R6/lib/X11/fonts/. Are numbers this large to be expected? > > > > > > 3382 files is too large. > > > > That seems about right for the removed font variants to me. You can't > > judge by the number of files, only the filenames. > > > > *If* you don't compile your own software from outside ports/packages, the > > files under /usr listed in sysclean's default output (no -a flag) is good. > > I do review manually before rm'ing but I have *never* had it suggesT > > removing something under /usr that is required. Files outside /usr > > need more care. > > > This is probably a stupid question but how do you review them manually? > I have a couple files that are manpages, that's easy. signify-keys, too. > There is some sgi stuff, also easy, retirement is known. > Same goes for switchd-related things. > But what about the rest? Assuming you don't just know everything about > those files already, do you find(1)/grep(1) through the source tree and > commit messages or is there a different way? I just scan through the filenames quickly and see if I think they're likely to cause a problem. I'm doing that anyway for the files outside of /usr and I read source-changes so I know what to expect.
Re: libdmx removal incomplete?
On 2021/11/30 8:14, Stuart Henderson wrote: > On 2021-11-29, Amit Kulkarni wrote: > > On Sun, Nov 28, 2021 at 5:17 PM Alexander wrote: > >> Just to gauge what to expect from this and whether I did this wrong: > >> After configuring /etc/sysclean.ignore I get 3382 files of which 3274 > >> are in /usr/X11R6/lib/X11/fonts/. Are numbers this large to be expected? > > > > 3382 files is too large. > > That seems about right for the removed font variants to me. You can't > judge by the number of files, only the filenames. > > *If* you don't compile your own software from outside ports/packages, the > files under /usr listed in sysclean's default output (no -a flag) is good. > I do review manually before rm'ing but I have *never* had it suggesT > removing something under /usr that is required. Files outside /usr > need more care. > This is probably a stupid question but how do you review them manually? I have a couple files that are manpages, that's easy. signify-keys, too. There is some sgi stuff, also easy, retirement is known. Same goes for switchd-related things. But what about the rest? Assuming you don't just know everything about those files already, do you find(1)/grep(1) through the source tree and commit messages or is there a different way? Best regards, Alexander
Re: libdmx removal incomplete?
> Date: Mon, 29 Nov 2021 08:31:15 -0500 > From: Nick Holland > > On 11/28/21 6:17 PM, Alexander wrote: > ... > > Lastly: From your emails it seems to me that the use of sysclean after > > upgrading is very much encouraged if not necessary. Then why is it not > > included in base (especially when it's developed by OpenBSD developers)? > > Or am I misunderstanding the requirements for inclusion of packages in > > base? > > VERY WRONG (as others have said). > > I've been using OpenBSD since v2.4, I have never run a "clean up" tool of > any kind. I reinstall only when replacing hardware, the rest of the time, > I run upgrades, I run snapshots and update frequently so I get a lot of > old files piling up at times. And they just don't matter. > > Occasionally, I have manually deleted old libraries when I have > run a system too long and an old HD starts getting tight on space > > [...] > > Using an automatic cleanup tool is far more likely to CAUSE problems > than to fix problems. I'm not saying they /often/ cause problems, > but since old files laying around basically never cause problems other > than a small amount of space, there's some risk and almost no gain. > Thanks Nick. That makes sense to me and is/was already my approach. I was mainly just curious to double-check with find(1) when I saw that notice on current.html. I was not actively looking to free up space, as me previously not even knowing that sysclean even exists might also suggest ;) But good to read another account on the OS's stability, thank you. > > -- > > Date: Sun, 28 Nov 2021 22:58:38 -0700 (MST) > From: Theo de Raadt > > >These files are still part of xshare70 set, and should not be > >removed. There are part of xorgproto (xenocara/proto/xorgproto). > > > >> Lastly: From your emails it seems to me that the use of sysclean after > >> upgrading is very much encouraged if not necessary. Then why is it not > >> included in base (especially when it's developed by OpenBSD developers)? > >> Or am I misunderstanding the requirements for inclusion of packages in > >> base? > > ^^^ > WRONG. Deleting old files is DISCOURAGED -- because we do > not have tooling to discover if a user has built their own > private programs which require those files. I am actually > getting a bit tired of (1) people overly worried about old > files (2) who don't recognize they can always reinstall and > (3) that we (OpenBSD) are not able to determine what to delete > any better than you the user. Thanks for making this very clear, makes sense that you can't deal with every weird non-standard installation. I hadn't thought of that before. But again not worried here, I was just a bit surprised when I stumbled over those original *dmx* files. As for reinstalling: Sure, I'm not really worried about my system, my (tested) backup scheme is working beautifully and reinstalling takes about as long as brewing a cup of coffee. But I would always like to avoid it when possible, especially when I can learn something about my system and how it works/is designed in the process. Best regards, Alexander
Re: libdmx removal incomplete?
On 2021/11/29 6:45, Sebastien Marie wrote: > On Sun, Nov 28, 2021 at 11:17:01PM +, Alexander wrote: > > > > Just to gauge what to expect from this and whether I did this wrong: > > After configuring /etc/sysclean.ignore I get 3382 files of which 3274 > > are in /usr/X11R6/lib/X11/fonts/. Are numbers this large to be expected? > > There are a bunch of files from /usr/X11R6/lib/X11/fonts/ which were > removed. On Sept 3, 3274 files were removed. > > https://github.com/openbsd/xenocara/commit/65ebc3c6dcf6461818fcc3917f443b4ab5b1ce1c > > So it is expected if your install was done before Sept 3, and your > current version is after Sept 3. That is the case, the install was a 6.9 continuing in -current. The diff you pointed to actually turns out to account for the majority of the files listed. So that cuts down the output significantly already. > > > Also: The above mentioned dmx files are not listed. Does that mean my > > assumption that they are related to the removed libdmx is false or did I > > screw something else up? > > $ pkg_locate dmx | grep X11R6 > xshare70:/usr/X11R6/include/X11/extensions/dmx.h > xshare70:/usr/X11R6/include/X11/extensions/dmxproto.h > xshare70:/usr/X11R6/lib/pkgconfig/dmxproto.pc > > These files are still part of xshare70 set, and should not be > removed. There are part of xorgproto (xenocara/proto/xorgproto). Thanks a lot, this really confused me. > > > Lastly: From your emails it seems to me that the use of sysclean after > > upgrading is very much encouraged if not necessary. Then why is it not > > included in base (especially when it's developed by OpenBSD developers)? > > Or am I misunderstanding the requirements for inclusion of packages in > > base? > > If removal of files is required, it is explicitly mentioned in > upgradeXX.html or current.html. Very few files will broke your system > if present. > > In the other side, removing files that are used will broke your system > (for example, if you compile a program yourself, it will use system > libraries like libc, libm...). > Thanks for the explanation, that makes sense. Best regards, Alexander
Re: libdmx removal incomplete?
On 2021-11-29, Nick Holland wrote: > I've been using OpenBSD since v2.4, I have never run a "clean up" tool of > any kind. I reinstall only when replacing hardware, the rest of the time, > I run upgrades, I run snapshots and update frequently so I get a lot of > old files piling up at times. And they just don't matter. FWIW I've been running it since whichever version had picture with a dark background and a red daemon head, and I use sysclean frequently. Most of the machines I maintain are remote and the upgrade process copes poorly or very poorly when running out of space. Before sysclean I usually did things like "cd /usr; rm -r share libdata X11R6 include" and looking at file dates/sizes in /usr/lib so it's a big improvement. Different people have worked out different ways to maintain their systems, that's fine.
Re: libdmx removal incomplete?
On 2021-11-29, Amit Kulkarni wrote: > On Sun, Nov 28, 2021 at 5:17 PM Alexander wrote: >> Just to gauge what to expect from this and whether I did this wrong: >> After configuring /etc/sysclean.ignore I get 3382 files of which 3274 >> are in /usr/X11R6/lib/X11/fonts/. Are numbers this large to be expected? > > 3382 files is too large. That seems about right for the removed font variants to me. You can't judge by the number of files, only the filenames. *If* you don't compile your own software from outside ports/packages, the files under /usr listed in sysclean's default output (no -a flag) is good. I do review manually before rm'ing but I have *never* had it suggesT removing something under /usr that is required. Files outside /usr need more care. (And, if you do compile your own software, it's going to break every few releases on OpenBSD anyway, system call ABIs change fairly frequently).
Re: libdmx removal incomplete?
On 11/28/21 6:17 PM, Alexander wrote: ... Lastly: From your emails it seems to me that the use of sysclean after upgrading is very much encouraged if not necessary. Then why is it not included in base (especially when it's developed by OpenBSD developers)? Or am I misunderstanding the requirements for inclusion of packages in base? VERY WRONG (as others have said). I've been using OpenBSD since v2.4, I have never run a "clean up" tool of any kind. I reinstall only when replacing hardware, the rest of the time, I run upgrades, I run snapshots and update frequently so I get a lot of old files piling up at times. And they just don't matter. Occasionally, I have manually deleted old libraries when I have run a system too long and an old HD starts getting tight on space, but that is usually an indicator that I should probably be looking at swapping out the hardware because it has done its time and I've probably got something better. And often not even then: $ ls -lt /usr/lib/|tail -4 -r--r--r-- 1 root bin274965 Feb 9 2012 libpcap.so.6.0 -r--r--r-- 1 root bin240930 Feb 9 2012 libkvm.so.12.0 -r--r--r-- 1 root bin323995 Feb 9 2012 libexpat.so.9.0 -r--r--r-- 1 root bin 2593417 Feb 9 2012 libc.so.62.0 (wow. that's an old machine.) Using an automatic cleanup tool is far more likely to CAUSE problems than to fix problems. I'm not saying they /often/ cause problems, but since old files laying around basically never cause problems other than a small amount of space, there's some risk and almost no gain. That machine with files left over from 2012? It's got a 40G hard disk. You will have trouble convincing me in 2021 that you are running out of disk space and thus need to "clean" your system. $ dmesg|grep ^wd wd0 at pciide0 channel 0 drive 0: wd0: 16-sector PIO, LBA48, 38146MB, 78125000 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 4 (to be fair, that machine fell off the 'net for a few years, I assumed it had died. Then it suddenly came back on line, so I brought it up to -current, so it skipped a lot of releases. But it's /usr partition is well under 50% full, so it has some life left...) Nick.
Re: libdmx removal incomplete?
On Mon, Nov 29, 2021 at 08:36:42AM +0100, Sebastien Marie wrote: > On Sun, Nov 28, 2021 at 10:58:38PM -0700, Theo de Raadt wrote: > > (2) who don't recognize they can always reinstall and > > Reinstalling means "choose the files you want to keep" vs "choose the > files you want to remove". Both have pros and cons. If you're in a situation where restoring configuration and user data after a re-install, either of the same or a more recent OpenBSD version, is a significant burden then you've already got a potential problem looming in the background. 100% of our production machines and servers are updated to each new OpenBSD version by re-install. This includes compiling any and all required ports from source. I can't remember the last time any particular machine required more than six hours, including time to either image the main system disk or physically replace it with another unit. If you keep the OpenBSD installation separate from user data, I.E. on a different physical disk, upgrade by re-installation becomes very easy. Just backup the entire installation to a partition on the user data disk, and do a fresh install on the system disk. Then mount the backup that you just made, copy and manually update any custom configuration that you had previously. Since we build all of our packages from source, often with local modifications to the makefiles, we tend to download the relevant source files first and check that the custom builds complete successfully on a scratch machine the day before we start the real updates. This also has the advantage that we can easily downgrade back to a clean install of a previous version if it ever became necessary for testing or other reasons.
Re: libdmx removal incomplete?
On Sun, Nov 28, 2021 at 10:58:38PM -0700, Theo de Raadt wrote: > >These files are still part of xshare70 set, and should not be > >removed. There are part of xorgproto (xenocara/proto/xorgproto). > > > >> Lastly: From your emails it seems to me that the use of sysclean after > >> upgrading is very much encouraged if not necessary. Then why is it not > >> included in base (especially when it's developed by OpenBSD developers)? > >> Or am I misunderstanding the requirements for inclusion of packages in > >> base? > > ^^^ > WRONG. Deleting old files is DISCOURAGED -- because we do > not have tooling to discover if a user has built their own > private programs which require those files. I agree. > I am actually > getting a bit tired of (1) people overly worried about old > files It is useful in certain circumstances. > (2) who don't recognize they can always reinstall and Reinstalling means "choose the files you want to keep" vs "choose the files you want to remove". Both have pros and cons. > (3) that we (OpenBSD) are not able to determine what to delete > any better than you the user. I agree. But for me it doesn't mean no tooling should exist to help the user in the task of file deletion. It is about help user to choose the files he/she want to delete from the two following lists: # find / -type f | wc -l 109221 # sysclean | wc -l 28 (OpenBSD aarch64 upgraded from Nov 1 to Nov 28 snapshot). Thanks. -- Sebastien Marie
Re: libdmx removal incomplete?
>These files are still part of xshare70 set, and should not be >removed. There are part of xorgproto (xenocara/proto/xorgproto). > >> Lastly: From your emails it seems to me that the use of sysclean after >> upgrading is very much encouraged if not necessary. Then why is it not >> included in base (especially when it's developed by OpenBSD developers)? >> Or am I misunderstanding the requirements for inclusion of packages in >> base? ^^^ WRONG. Deleting old files is DISCOURAGED -- because we do not have tooling to discover if a user has built their own private programs which require those files. I am actually getting a bit tired of (1) people overly worried about old files (2) who don't recognize they can always reinstall and (3) that we (OpenBSD) are not able to determine what to delete any better than you the user. >If removal of files is required, it is explicitly mentioned in >upgradeXX.html or current.html. Very few files will broke your system >if present. > >In the other side, removing files that are used will broke your system >(for example, if you compile a program yourself, it will use system >libraries like libc, libm...). > >Thanks. >-- >Sebastien Marie > >
Re: libdmx removal incomplete?
On Sun, Nov 28, 2021 at 11:17:01PM +, Alexander wrote: > > Just to gauge what to expect from this and whether I did this wrong: > After configuring /etc/sysclean.ignore I get 3382 files of which 3274 > are in /usr/X11R6/lib/X11/fonts/. Are numbers this large to be expected? There are a bunch of files from /usr/X11R6/lib/X11/fonts/ which were removed. On Sept 3, 3274 files were removed. https://github.com/openbsd/xenocara/commit/65ebc3c6dcf6461818fcc3917f443b4ab5b1ce1c So it is expected if your install was done before Sept 3, and your current version is after Sept 3. > Also: The above mentioned dmx files are not listed. Does that mean my > assumption that they are related to the removed libdmx is false or did I > screw something else up? $ pkg_locate dmx | grep X11R6 xshare70:/usr/X11R6/include/X11/extensions/dmx.h xshare70:/usr/X11R6/include/X11/extensions/dmxproto.h xshare70:/usr/X11R6/lib/pkgconfig/dmxproto.pc These files are still part of xshare70 set, and should not be removed. There are part of xorgproto (xenocara/proto/xorgproto). > Lastly: From your emails it seems to me that the use of sysclean after > upgrading is very much encouraged if not necessary. Then why is it not > included in base (especially when it's developed by OpenBSD developers)? > Or am I misunderstanding the requirements for inclusion of packages in > base? If removal of files is required, it is explicitly mentioned in upgradeXX.html or current.html. Very few files will broke your system if present. In the other side, removing files that are used will broke your system (for example, if you compile a program yourself, it will use system libraries like libc, libm...). Thanks. -- Sebastien Marie
Re: libdmx removal incomplete?
On Sun, Nov 28, 2021 at 5:17 PM Alexander wrote: > > Hi, > thanks to both of you. > > On 2021/11/26 6:51, Sebastien Marie wrote: > > On Thu, Nov 25, 2021 at 06:16:11PM -0600, Amit Kulkarni wrote: > > > > I'm aware that I'm pretty late with this, still I'd like to ask in case > > > > this is not completely irrelevant. > > > > > > > > The last entry on https://www.openbsd.org/faq/current.html before > > > > 'Roll current' was the libdmx removal: > > > > https://cvsweb.openbsd.org/www/faq/current.html?rev=1.1077 > > > > > > > > After the suggested 'rm -f' commands there are still some files around > > > > on my system that to me seem to be related: > > > > > > > > $ find /usr/X11R6/ -iname *dmx* > > > > /usr/X11R6/lib/pkgconfig/dmxproto.pc > > > > /usr/X11R6/include/X11/extensions/dmx.h > > > > /usr/X11R6/include/X11/extensions/dmxproto.h > > > > > > > > dmx.h and dmxproto.h for example reference the deleted dmxext.h > > > > Does that mean this libdmx removal is incomplete or am I just > > > > misunderstanding something? > > > > Thanks in advance. > > > > > > > > > try to install sysclean, configure /etc/sysclean.ignore > > > > > > and do a 'doas sysclean -a', all these files will be gone then. > > > > > > > removing files based on `sysclean -a` output might be dangerous. it > > will list all files, even the one still used by packages. it could > > result in not working packages. > > > > `sysclean` (without option) is safer. > > > > please note that the stage 'configure /etc/sysclean.ignore' is > > important to exclude from the output configuration files (in /etc) you > > manually created. > > Just to gauge what to expect from this and whether I did this wrong: > After configuring /etc/sysclean.ignore I get 3382 files of which 3274 > are in /usr/X11R6/lib/X11/fonts/. Are numbers this large to be expected? > > Also: The above mentioned dmx files are not listed. Does that mean my > assumption that they are related to the removed libdmx is false or did I > screw something else up? > > Lastly: From your emails it seems to me that the use of sysclean after > upgrading is very much encouraged if not necessary. Then why is it not > included in base (especially when it's developed by OpenBSD developers)? > Or am I misunderstanding the requirements for inclusion of packages in > base? > Hi Alexander, 3382 files is too large. Post the contents of /etc/sysclean.ignore please. Something weird is happening. If you go ahead, you will have unusable system. Here is the upgrade 70 guide, which lists which exact files need to be deleted. https://www.openbsd.org/faq/upgrade70.html Thanks
Re: libdmx removal incomplete?
Hi, thanks to both of you. On 2021/11/26 6:51, Sebastien Marie wrote: > On Thu, Nov 25, 2021 at 06:16:11PM -0600, Amit Kulkarni wrote: > > > I'm aware that I'm pretty late with this, still I'd like to ask in case > > > this is not completely irrelevant. > > > > > > The last entry on https://www.openbsd.org/faq/current.html before > > > 'Roll current' was the libdmx removal: > > > https://cvsweb.openbsd.org/www/faq/current.html?rev=1.1077 > > > > > > After the suggested 'rm -f' commands there are still some files around > > > on my system that to me seem to be related: > > > > > > $ find /usr/X11R6/ -iname *dmx* > > > /usr/X11R6/lib/pkgconfig/dmxproto.pc > > > /usr/X11R6/include/X11/extensions/dmx.h > > > /usr/X11R6/include/X11/extensions/dmxproto.h > > > > > > dmx.h and dmxproto.h for example reference the deleted dmxext.h > > > Does that mean this libdmx removal is incomplete or am I just > > > misunderstanding something? > > > Thanks in advance. > > > > > > try to install sysclean, configure /etc/sysclean.ignore > > > > and do a 'doas sysclean -a', all these files will be gone then. > > > > removing files based on `sysclean -a` output might be dangerous. it > will list all files, even the one still used by packages. it could > result in not working packages. > > `sysclean` (without option) is safer. > > please note that the stage 'configure /etc/sysclean.ignore' is > important to exclude from the output configuration files (in /etc) you > manually created. Just to gauge what to expect from this and whether I did this wrong: After configuring /etc/sysclean.ignore I get 3382 files of which 3274 are in /usr/X11R6/lib/X11/fonts/. Are numbers this large to be expected? Also: The above mentioned dmx files are not listed. Does that mean my assumption that they are related to the removed libdmx is false or did I screw something else up? Lastly: From your emails it seems to me that the use of sysclean after upgrading is very much encouraged if not necessary. Then why is it not included in base (especially when it's developed by OpenBSD developers)? Or am I misunderstanding the requirements for inclusion of packages in base? Best regards, Alexander
Re: libdmx removal incomplete?
On Thu, Nov 25, 2021 at 06:16:11PM -0600, Amit Kulkarni wrote: > > I'm aware that I'm pretty late with this, still I'd like to ask in case > > this is not completely irrelevant. > > > > The last entry on https://www.openbsd.org/faq/current.html before > > 'Roll current' was the libdmx removal: > > https://cvsweb.openbsd.org/www/faq/current.html?rev=1.1077 > > > > After the suggested 'rm -f' commands there are still some files around > > on my system that to me seem to be related: > > > > $ find /usr/X11R6/ -iname *dmx* > > /usr/X11R6/lib/pkgconfig/dmxproto.pc > > /usr/X11R6/include/X11/extensions/dmx.h > > /usr/X11R6/include/X11/extensions/dmxproto.h > > > > dmx.h and dmxproto.h for example reference the deleted dmxext.h > > Does that mean this libdmx removal is incomplete or am I just > > misunderstanding something? > > Thanks in advance. > > > try to install sysclean, configure /etc/sysclean.ignore > > and do a 'doas sysclean -a', all these files will be gone then. > removing files based on `sysclean -a` output might be dangerous. it will list all files, even the one still used by packages. it could result in not working packages. `sysclean` (without option) is safer. please note that the stage 'configure /etc/sysclean.ignore' is important to exclude from the output configuration files (in /etc) you manually created. as a remainder, sysclean will only *LIST* files (it is pledged read-only), and doesn't remove anything itself. thanks. -- Sebastien Marie
Re: libdmx removal incomplete?
> I'm aware that I'm pretty late with this, still I'd like to ask in case > this is not completely irrelevant. > > The last entry on https://www.openbsd.org/faq/current.html before > 'Roll current' was the libdmx removal: > https://cvsweb.openbsd.org/www/faq/current.html?rev=1.1077 > > After the suggested 'rm -f' commands there are still some files around > on my system that to me seem to be related: > > $ find /usr/X11R6/ -iname *dmx* > /usr/X11R6/lib/pkgconfig/dmxproto.pc > /usr/X11R6/include/X11/extensions/dmx.h > /usr/X11R6/include/X11/extensions/dmxproto.h > > dmx.h and dmxproto.h for example reference the deleted dmxext.h > Does that mean this libdmx removal is incomplete or am I just > misunderstanding something? > Thanks in advance. try to install sysclean, configure /etc/sysclean.ignore and do a 'doas sysclean -a', all these files will be gone then.
