bug#37702: Suggestion for 'df' utility
On 10/13/19 3:00 PM, Assaf Gordon wrote: I'm not sure if it's easy to find a set of criteria that would work well while having minimal unexpected side effects of hiding entries people in other systems do expect to see. No matter what we do (even if we do nothing), there will be problems. But doing nothing is clearly a bad idea, as the output of plain df is quite bad right now in typical use. We can do better than that, even if we cannot be perfect and we cause problems by changing defaults. Out of curiosity, can you share the output of the following commands on the same system? Sure, here it is: $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:00 140.7M 1 loop /snap/gnome-3-26-1604/90 loop1 7:10 44.2M 1 loop /snap/gtk-common-themes/1353 loop2 7:20 149.9M 1 loop /snap/gnome-3-28-1804/71 loop3 7:30 3.7M 1 loop /snap/gnome-system-monitor/100 loop4 7:40 14.8M 1 loop /snap/gnome-characters/317 loop5 7:50 956K 1 loop /snap/gnome-logs/73 loop6 7:60 149.9M 1 loop /snap/gnome-3-28-1804/67 loop7 7:70 3.7M 1 loop /snap/gnome-system-monitor/95 loop8 7:80 4M 1 loop /snap/gnome-calculator/406 loop9 7:90 54.5M 1 loop /snap/core18/1192 loop10 7:10 089M 1 loop /snap/core/7713 loop11 7:11 0 42.8M 1 loop /snap/gtk-common-themes/1313 loop12 7:12 0 14.8M 1 loop /snap/gnome-characters/296 loop13 7:13 0 140.7M 1 loop /snap/gnome-3-26-1604/92 loop14 7:14 0 89.1M 1 loop /snap/core/7917 loop15 7:15 0 956K 1 loop /snap/gnome-logs/81 loop16 7:16 0 4.2M 1 loop /snap/gnome-calculator/501 loop17 7:17 0 54.4M 1 loop /snap/core18/1144 sda 8:00 111.8G 0 disk ├─sda1 8:10 96.9G 0 part / ├─sda2 8:20 1K 0 part └─sda5 8:5015G 0 part [SWAP] sdb 8:16 0 2.7T 0 disk └─sdb1 8:17 0 2.7T 0 part └─md127 9:127 0 2.7T 0 raid1 └─md127p1 259:00 2.7T 0 md/home sdc 8:32 0 2.7T 0 disk └─sdc1 8:33 0 2.7T 0 part └─md127 9:127 0 2.7T 0 raid1 └─md127p1 259:00 2.7T 0 md/home sdd 8:48 1 7.5G 0 disk └─sdd1 8:49 1 7.5G 0 part /media/eggert/B827-D456 sr011:01 1024M 0 rom $ df -x tmpfs -x devtmpfs -x squashfs Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda199431552 11740452 82597212 13% / /dev/md127p1 2884021472 1326329584 1411168908 49% /home /dev/sdd1 781286447051363107728 61% /media/eggert/B827-D456
bug#37702: Suggestion for 'df' utility
On 2019-10-14 00:13, Assaf Gordon wrote: > Also in other systems where "/tmp" is a "tmpfs", > users might want to see how much space is available. > > If we hide it by default, they can of course use "df /tmp" > or "df --all" - it's not about removing this option, > it is just about making users' life harder or easier, > and making unexpected changes. I see the point ... and agree. >> This seems to be a small change, and not satisfying the snap case. > > Possibly hiding "squashfs" of readonly-mounts could get rid of those snaps? Possibly. But still the same question: how many users would like to see them per default, and how many do not? I would not like to argue for something like a ~/.dfrc, but it seems there are a lot of different flavors of wishes out there. Yet leaving the default seems to be a wise choice to me. FWIW: df has the -x option, but if someone would use it in an alias like "df -x sqashfs", then there's currently no chance to override it again: $ alias df='/usr/bin/df -x tmpfs' $ df -t tmpfs df: file system type 'tmpfs' both selected and excluded Have a nice day, Berny
bug#37702: Suggestion for 'df' utility
Hello Bernhard, On 2019-10-13 3:57 p.m., Bernhard Voelker wrote: On 2019-10-13 23:28, Paul Eggert wrote: In any sane system there would be only four lines of non-header output (for tmpfs etc, /, /home, and /media/eggert/B827-D456), but df is outputting 28 lines. What is so special about tmpfs so that you would like to see it? As an interesting use-case (though not common), I recently configured a raspberry PI device, and wanted to mount as many locations on tmpfs as possible, e.g. "/tmp" "/var/tmp", "/var/log" etc. In was very useful in those cases to be able to see separate tmpfs file system listed, with information about how big they are and how much space was used. Also in other systems where "/tmp" is a "tmpfs", users might want to see how much space is available. If we hide it by default, they can of course use "df /tmp" or "df --all" - it's not about removing this option, it is just about making users' life harder or easier, and making unexpected changes. I recently also encountered a change in a default behavior of a program which I've been using a very long time - and it is *very* frustrating to have something that worked "just fine" for so long being changed. Here on my openSUSE:Tumbleweed system, I see the following: $ df -T Filesystem Type 1K-blocks Used Available Use% Mounted on [...] /dev/loop0 ext2 31729 31729 0 100% /FULL_PARTITION_TMPDIR [...] (The /FULL_PARTITION_TMPDIR is used by a special coreutils test.) That's an interesting case, where I would think you'd want to see it, because you explicitly mounted it. I think I could well live with adding 'devtmpfs' and 'tmpfs' to the pseudo file systems in gnulib's "mountlist.c". I agree, but think this needs to be communicated very well, and in advance - perhaps announce this change ahead of time to the respective package maintainers of each distribution - just so they'll know it's coming (and also have a way to revert it if they don't like it). This seems to be a small change, and not satisfying the snap case. Possibly hiding "squashfs" of readonly-mounts could get rid of those snaps? regards, -assaf
bug#37702: Suggestion for 'df' utility
On 2019-10-13 3:28 p.m., Paul Eggert wrote: [..] I mean c'mon, here's the output of 'df' on the Ubuntu 18.04.3 LTS workstation I'm typing this particular message on. In any sane system there would be only four lines of non-header output (for tmpfs etc, /, /home, and /media/eggert/B827-D456), but df is outputting 28 lines. This is ridiculous. It is certainly inconvenient if that's not what you are looking for (and certainly most desktop users aren't). But I'm not sure if it's easy to find a set of criteria that would work well while having minimal unexpected side effects of hiding entries people in other systems do expect to see. Out of curiosity, can you share the output of the following commands on the same system? lsblk df -x tmpfs -x devtmpfs -x squashfs Thanks, - assaf
bug#37702: Suggestion for 'df' utility
On 2019-10-13 23:28, Paul Eggert wrote: > In any sane system there would be only > four lines of non-header output (for tmpfs etc, /, /home, and > /media/eggert/B827-D456), but df is outputting 28 lines. What is so special about tmpfs so that you would like to see it? Here on my openSUSE:Tumbleweed system, I see the following: $ df -T Filesystem Type 1K-blocks Used Available Use% Mounted on devtmpfs devtmpfs 10187924 8 10187916 1% /dev tmpfs tmpfs 10199048 45788 10153260 1% /dev/shm tmpfs tmpfs 10199048 18036 10181012 1% /run tmpfs tmpfs 10199048 0 10199048 0% /sys/fs/cgroup /dev/sda2 ext4 20511312 12141864 7304488 63% / /dev/loop0 ext2 31729 31729 0 100% /FULL_PARTITION_TMPDIR /dev/sda5 ext4 619142920 390088908 229037628 64% /media/big_data /dev/sda3 ext3 103085876 90714416 7128248 93% /home tmpfs tmpfs 203980820 2039788 1% /run/user/1000 (The /FULL_PARTITION_TMPDIR is used by a special coreutils test.) I think I could well live with adding 'devtmpfs' and 'tmpfs' to the pseudo file systems in gnulib's "mountlist.c". This seems to be a small change, and not satisfying the snap case. Yet, I agree with Assaf that changing the defaults has to be done with caution. Eliding r/o filesystems or where usage<1% doesn't look like such. Have a nice day, Berny
bug#37702: Suggestion for 'df' utility
On 10/13/19 2:11 PM, Assaf Gordon wrote: This thread originated by a request to "clean up" the output on newer ubuntu machines which use "snap" packages as /dev/loopN . Let's not turn that into a drastic change It could certainly be multiple sets of patches. But let's face it, df's utility for ordinary interactive use has degraded significantly with time due to all the random filesystems people have been adding, and we shouldn't keep our heads in the sands about this. df's default needs to change, one way or another. I mean c'mon, here's the output of 'df' on the Ubuntu 18.04.3 LTS workstation I'm typing this particular message on. In any sane system there would be only four lines of non-header output (for tmpfs etc, /, /home, and /media/eggert/B827-D456), but df is outputting 28 lines. This is ridiculous. Filesystem 1K-blocks Used Available Use% Mounted on udev 7644704 07644704 0% /dev tmpfs 1533620 19241531696 1% /run /dev/sda199431552 11740340 82597324 13% / tmpfs 7668096 332127634884 1% /dev/shm tmpfs5120 8 5112 1% /run/lock tmpfs 7668096 07668096 0% /sys/fs/cgroup /dev/loop0 144128 144128 0 100% /snap/gnome-3-26-1604/90 /dev/loop1 45312 45312 0 100% /snap/gtk-common-themes/1353 /dev/loop2 153600 153600 0 100% /snap/gnome-3-28-1804/71 /dev/loop4 15104 15104 0 100% /snap/gnome-characters/317 /dev/loop6 153600 153600 0 100% /snap/gnome-3-28-1804/67 /dev/loop5 1024 1024 0 100% /snap/gnome-logs/73 /dev/loop3 3840 3840 0 100% /snap/gnome-system-monitor/100 /dev/loop10 91264 91264 0 100% /snap/core/7713 /dev/loop7 3840 3840 0 100% /snap/gnome-system-monitor/95 /dev/loop8 4224 4224 0 100% /snap/gnome-calculator/406 /dev/loop9 55808 55808 0 100% /snap/core18/1192 /dev/loop11 43904 43904 0 100% /snap/gtk-common-themes/1313 /dev/loop12 15104 15104 0 100% /snap/gnome-characters/296 /dev/loop13144128 144128 0 100% /snap/gnome-3-26-1604/92 /dev/loop14 91264 91264 0 100% /snap/core/7917 /dev/loop15 1024 1024 0 100% /snap/gnome-logs/81 /dev/loop17 55808 55808 0 100% /snap/core18/1144 /dev/loop16 4352 4352 0 100% /snap/gnome-calculator/501 /dev/md127p1 2884021472 1326255744 1411242748 49% /home tmpfs 1533616 161533600 1% /run/user/121 tmpfs 1533616 601533556 1% /run/user/1000 /dev/sdd1 781286447051363107728 61% /media/eggert/B827-D456
bug#37702: Suggestion for 'df' utility
Hi all, On 2019-10-13 2:27 p.m., Paul Eggert wrote: On 10/13/19 2:41 AM, Pádraig Brady wrote: I wonder could we key (also) on used==0||available==0. Yes, looking at the sample output I gave earlier, I'd say we could by default drop filesystems where usage is 1% or less. That would solve the problem for my workstation. This is roughly akin to the "used==0" test you're suggesting. I would humbly suggest caution with such unexpected user-facing changes to the default output of 'df' - learning the lessons from changing the quotes in 'ls'. Countless users have been using 'df' in their own ways, and have gotten used to certain outputs. This thread originated by a request to "clean up" the output on newer ubuntu machines which use "snap" packages as /dev/loopN . Let's not turn that into a drastic change that will affect many other existing systems - the users on other systems did not ask for any changes. --- Specifically for "default drop filesystems where usage is 1% or less" - I can think of few cases off the top of my head where this would be extremely confusing: - I recently installed a 33TB raid file system. The usage on that system is at %1 and will stay like so for at least several days. - Amazon cloud services (AWS) offers an NFS4 service (they call it "EFS") that has reported size of 8 exabytes. There too usage could be at %1 for a long long time. --- For cases where I want to list only the "real" storage, I typically use an alias such as: alias dff='df -h -x tmpfs -x devtmpfs' And it would be very easy and least disruptive to recommend to ubuntu users to add "-x squashfs" or another file system to ignore. Perhaps we can come up with a recommended list of "lesser" file systems to ignore (or conditions such as read-only file systems) and add it as a new option, but please let's not make it the default. My two cents, - assaf
bug#37702: Suggestion for 'df' utility
On 10/13/19 2:41 AM, Pádraig Brady wrote: I wonder could we key (also) on used==0||available==0. Yes, looking at the sample output I gave earlier, I'd say we could by default drop filesystems where usage is 1% or less. That would solve the problem for my workstation. This is roughly akin to the "used==0" test you're suggesting. (I don't know if this would address the problem of small snap loop devices. I haven't seen sample df output from that.) What you mean by "available==0"? Actual zero-size filesystems, or filesystems whose size is less than some epsilon? What would the epsilon be? Can you give an example?
bug#37702: Suggestion for 'df' utility
On 11/10/2019 20:56, Paul Eggert wrote: On 10/11/19 11:20 AM, Pádraig Brady wrote: if you want to exclude nested file systems like that, you could try: alias df='df -x squashfs' On my Fedora 30 workstation that option doesn't make any difference. Regardless of whether '-x squashfs' is used, I see this output from 'df': Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 4065704 04065704 0% /dev tmpfs 4081560 366164044944 1% /dev/shm tmpfs 4081560 16964079864 1% /run tmpfs 4081560 04081560 0% /sys/fs/cgroup /dev/sda559614116 16910684 39645412 30% / tmpfs 4081560 1244081436 1% /tmp /dev/sda2 1849433716 207781976 1547682948 12% /home /dev/sda1 50950402444684572044 6% /boot tmpfs 81631260 816252 1% /run/user/1000 and most of these lines are useless. For many years we've put up with the problem of too many filesystems in the default plain 'df' output, and now's as good a time as any to fix that. On my workstation there should be only four lines of information, one each for /, /home, /boot, and the shared tmpfs area. Presumably readonly filesystems should also be omitted by default, since they're not something people ordinarily care about. We can add a flag or two for the rare people who want to see these normally-useless lines. What df tries to show now is a deduplicated list of file systems that have consumable storage. Maybe read-only could be valid to drop by default, but I've worked on systems where certain file-systems were read-only most of the time, but they were consumable. I wonder could we key (also) on used==0||available==0. cheers, Pádraig
