bug#37702: Suggestion for 'df' utility

[Julian Andres Klode]

On Fri, Jul 10, 2020 at 04:29:21PM -0700, Bryce Harrington wrote:
> On Mon, Jun 01, 2020 at 09:26:14AM -0700, Bryce Harrington wrote:
> > On Mon, Jun 01, 2020 at 09:04:26AM -0700, Bryce Harrington wrote:
> > > On Sun, May 31, 2020 at 01:49:24PM +0100, Pádraig Brady wrote:
> > > > On 31/05/2020 10:36, Bernhard Voelker wrote:
> > > > > What about to start with this?
> > > > > 
> > > > >$ GIT_PAGER= git -C gnulib diff
> > > > >diff --git a/lib/mountlist.c b/lib/mountlist.c
> > > > >index 7abe0248e..5f6249dec 100644
> > > > >--- a/lib/mountlist.c
> > > > >+++ b/lib/mountlist.c
> > > > >@@ -164,6 +164,9 @@
> > > > > 
> > > > > #define ME_DUMMY_0(Fs_name, Fs_type)\
> > > > >   (strcmp (Fs_type, "autofs") == 0  \
> > > > >+   || strcmp (Fs_type, "tmpfs") == 0\
> > > > 
> > > > tmpfs is consumable, so wouldn't be appropriate to add I think
> > > > 
> > > > >+   || strcmp (Fs_type, "devtmpfs") == 0 \
> > > > >+   || strcmp (Fs_type, "squashfs") == 0 \
> 
> FYI, I've landed your suggested change into Ubuntu's coreutils today.
> Only difference from this is I interleaved these additions
> alphabetically.

I'm now adding fuse.portal to the list, see




https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/1905623
-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

bug#37702: Suggestion for 'df' utility

[Bryce Harrington]

On Mon, Jun 01, 2020 at 09:26:14AM -0700, Bryce Harrington wrote:
> On Mon, Jun 01, 2020 at 09:04:26AM -0700, Bryce Harrington wrote:
> > On Sun, May 31, 2020 at 01:49:24PM +0100, Pádraig Brady wrote:
> > > On 31/05/2020 10:36, Bernhard Voelker wrote:
> > > > What about to start with this?
> > > > 
> > > >$ GIT_PAGER= git -C gnulib diff
> > > >diff --git a/lib/mountlist.c b/lib/mountlist.c
> > > >index 7abe0248e..5f6249dec 100644
> > > >--- a/lib/mountlist.c
> > > >+++ b/lib/mountlist.c
> > > >@@ -164,6 +164,9 @@
> > > > 
> > > > #define ME_DUMMY_0(Fs_name, Fs_type)\
> > > >   (strcmp (Fs_type, "autofs") == 0  \
> > > >+   || strcmp (Fs_type, "tmpfs") == 0\
> > > 
> > > tmpfs is consumable, so wouldn't be appropriate to add I think
> > > 
> > > >+   || strcmp (Fs_type, "devtmpfs") == 0 \
> > > >+   || strcmp (Fs_type, "squashfs") == 0 \

FYI, I've landed your suggested change into Ubuntu's coreutils today.
Only difference from this is I interleaved these additions
alphabetically.

I'd still be interested in if there's a way to differentiate consumable
vs. non-consumable tmpfs mounts such that we can hide the latter.

> By the way, I'm told that the snap developers tag their squashfs mounts
> with "x-gdu.hide", so if we want to be more precise in what types of
> squashfs mount points to exclude filtering on that property might be
> one way.

I dug into this a bit more.  It looks like the tag is exposed through
libmount, however gnulib explicitly does not want to depend on
libmount.  If there is a different way to access this info on a mount
point that can be filtered on, it might be a helpful refinement.  I
don't know if there are many use cases for non-snap squashfs that would
need displayed in df output, but I imagine there must be some out there.

Bryce

>   $ mount | grep squashfs | head -n1
>   /var/lib/snapd/snaps/chromium_1143.snap on /snap/chromium/1143 type 
> squashfs (ro,nodev,relatime,x-gdu.hide)
> 
> 
> Bryce
> 
> > > >|| strcmp (Fs_type, "proc") == 0 \
> > > >|| strcmp (Fs_type, "subfs") == 0\
> > > >/* for Linux 2.6/3.x */  \
> > 
> > This code appears to be part of gnulib, so I take it the patch would
> > need to be added there?  Would it also need updates to any tests, docs,
> > or other code?
> > 
> > You mentioned above this would need an option to override.  I checked
> > that `df -a` works as expected, but `df -t squashfs` just says:
> > 
> >   src/df: no file systems processed
> > 
> > It looks like -t only works from the filtered fs list, not on the
> > excluded dummy systems, so it's a consistent behavior.  However, I do
> > agree it could be nice for users to have a way to override the default
> > behavior; are you thinking a new option would be added?  If so can you
> > outline your thoughts on how that should work?
> > 
> > Thanks,
> > Bryce

bug#37702: Suggestion for 'df' utility

[Bob Proulx]

Paul Eggert wrote:
> So I'd prefer having 'df' just do the "right" thing by default, and
> to have an option to override that. The "right" thing should be to
> ignore all these pseudofilesystems that hardly anybody cares about.

+1!  Which I thought I would say because often I am a status quo type
of person.  But this is clearly needed.  Hardly a day goes by that I
don't hear swearing from people about the current extremely noisy and
hard to use df output in the environment of dozens of pseudo file
systems.  And I don't think this will break legacy and scripted use.

Bob

bug#37702: Suggestion for 'df' utility

[Bryce Harrington]

On Mon, Jun 01, 2020 at 09:04:26AM -0700, Bryce Harrington wrote:
> On Sun, May 31, 2020 at 01:49:24PM +0100, Pádraig Brady wrote:
> > On 31/05/2020 10:36, Bernhard Voelker wrote:
> > > What about to start with this?
> > > 
> > >$ GIT_PAGER= git -C gnulib diff
> > >diff --git a/lib/mountlist.c b/lib/mountlist.c
> > >index 7abe0248e..5f6249dec 100644
> > >--- a/lib/mountlist.c
> > >+++ b/lib/mountlist.c
> > >@@ -164,6 +164,9 @@
> > > 
> > > #define ME_DUMMY_0(Fs_name, Fs_type)\
> > >   (strcmp (Fs_type, "autofs") == 0  \
> > >+   || strcmp (Fs_type, "tmpfs") == 0\
> > 
> > tmpfs is consumable, so wouldn't be appropriate to add I think
> > 
> > >+   || strcmp (Fs_type, "devtmpfs") == 0 \
> > >+   || strcmp (Fs_type, "squashfs") == 0 \

By the way, I'm told that the snap developers tag their squashfs mounts
with "x-gdu.hide", so if we want to be more precise in what types of
squashfs mount points to exclude filtering on that property might be
one way.

  $ mount | grep squashfs | head -n1
  /var/lib/snapd/snaps/chromium_1143.snap on /snap/chromium/1143 type squashfs 
(ro,nodev,relatime,x-gdu.hide)


Bryce

> > >|| strcmp (Fs_type, "proc") == 0 \
> > >|| strcmp (Fs_type, "subfs") == 0\
> > >/* for Linux 2.6/3.x */  \
> 
> This code appears to be part of gnulib, so I take it the patch would
> need to be added there?  Would it also need updates to any tests, docs,
> or other code?
> 
> You mentioned above this would need an option to override.  I checked
> that `df -a` works as expected, but `df -t squashfs` just says:
> 
>   src/df: no file systems processed
> 
> It looks like -t only works from the filtered fs list, not on the
> excluded dummy systems, so it's a consistent behavior.  However, I do
> agree it could be nice for users to have a way to override the default
> behavior; are you thinking a new option would be added?  If so can you
> outline your thoughts on how that should work?
> 
> Thanks,
> Bryce

bug#37702: Suggestion for 'df' utility

[Bryce Harrington]

On Sun, May 31, 2020 at 01:49:24PM +0100, Pádraig Brady wrote:
> On 31/05/2020 10:36, Bernhard Voelker wrote:
> > On 2020-05-31 01:07, Paul Eggert wrote:
> > > On 5/30/20 4:49 AM, Erik Auerswald wrote:
> > > > I concur that a command line option to override config file (or env var)
> > > > settings seems useful if a config file and/or env var approach is used.
> > 
> > Just to mention another alternative to control the behavior besides env 
> > variables or
> > config files: it would theoretically also be possible to add a 
> > './configure' option
> > to add more file system types to the list of pseudofilesystems at build 
> > time.
> > Obviously, such idea moves the responsibility to the packager - which is not
> > nice either.
> > 
> > > In other utilities we've been moving away from environment variables 
> > > and/or
> > > config files for the usual security and other-hassle reasons. So I'd 
> > > prefer
> > > having 'df' just do the "right" thing by default, and to have an option to
> > > override that. The "right" thing should be to ignore all these 
> > > pseudofilesystems
> > > that hardly anybody cares about.
> > 
> > +1

That sounds logical, and it'd certainly be least hassle for us as well.

> > What about to start with this?
> > 
> >$ GIT_PAGER= git -C gnulib diff
> >diff --git a/lib/mountlist.c b/lib/mountlist.c
> >index 7abe0248e..5f6249dec 100644
> >--- a/lib/mountlist.c
> >+++ b/lib/mountlist.c
> >@@ -164,6 +164,9 @@
> > 
> > #define ME_DUMMY_0(Fs_name, Fs_type)\
> >   (strcmp (Fs_type, "autofs") == 0  \
> >+   || strcmp (Fs_type, "tmpfs") == 0\
> 
> tmpfs is consumable, so wouldn't be appropriate to add I think
> 
> >+   || strcmp (Fs_type, "devtmpfs") == 0 \
> >+   || strcmp (Fs_type, "squashfs") == 0 \
> >|| strcmp (Fs_type, "proc") == 0 \
> >|| strcmp (Fs_type, "subfs") == 0\
> >/* for Linux 2.6/3.x */  \

This code appears to be part of gnulib, so I take it the patch would
need to be added there?  Would it also need updates to any tests, docs,
or other code?

You mentioned above this would need an option to override.  I checked
that `df -a` works as expected, but `df -t squashfs` just says:

  src/df: no file systems processed

It looks like -t only works from the filtered fs list, not on the
excluded dummy systems, so it's a consistent behavior.  However, I do
agree it could be nice for users to have a way to override the default
behavior; are you thinking a new option would be added?  If so can you
outline your thoughts on how that should work?

Thanks,
Bryce

bug#37702: Suggestion for 'df' utility

[Kamil Dudka]

On Sunday, May 31, 2020 2:49:24 PM CEST Pádraig Brady wrote:
> On 31/05/2020 10:36, Bernhard Voelker wrote:
> > What about to start with this?
> > 
> >$ GIT_PAGER= git -C gnulib diff
> >diff --git a/lib/mountlist.c b/lib/mountlist.c
> >index 7abe0248e..5f6249dec 100644
> >--- a/lib/mountlist.c
> >+++ b/lib/mountlist.c
> >@@ -164,6 +164,9 @@
> >
> > #define ME_DUMMY_0(Fs_name, Fs_type)\
> > 
> >   (strcmp (Fs_type, "autofs") == 0  \
> >
> >+   || strcmp (Fs_type, "tmpfs") == 0\
> 
> tmpfs is consumable, so wouldn't be appropriate to add I think

+1

No space left on a tmpfs file system mounted e.g. on /tmp is a common problem
that should be easily visible from the default output of df.

Kamil

bug#37702: Suggestion for 'df' utility

[Erik Auerswald]

Hi John,

On Sun, May 31, 2020 at 06:52:04PM +1000, John Pye wrote:
> The purpose of "df" is to show "disk free". Hence any filesystems that
> are read-only or which are FUSE-mounted one on of the local physical
> filesystems, or similar things (what others?) should be suppressed by
> default.

The FUSE case seems tricky, since, e.g., NTFS filesystems are mounted
via FUSE on, e.g., Ubuntu 18.04.  Thus FUSE should not be excluded by
default, I'd say.

HTH,
Erik

bug#37702: Suggestion for 'df' utility

[Pádraig Brady]

On 31/05/2020 10:36, Bernhard Voelker wrote:

On 2020-05-31 01:07, Paul Eggert wrote:

On 5/30/20 4:49 AM, Erik Auerswald wrote:

I concur that a command line option to override config file (or env var)
settings seems useful if a config file and/or env var approach is used.


Just to mention another alternative to control the behavior besides env 
variables or
config files: it would theoretically also be possible to add a './configure' 
option
to add more file system types to the list of pseudofilesystems at build time.
Obviously, such idea moves the responsibility to the packager - which is not
nice either.


In other utilities we've been moving away from environment variables and/or
config files for the usual security and other-hassle reasons. So I'd prefer
having 'df' just do the "right" thing by default, and to have an option to
override that. The "right" thing should be to ignore all these pseudofilesystems
that hardly anybody cares about.


+1

What about to start with this?

   $ GIT_PAGER= git -C gnulib diff
   diff --git a/lib/mountlist.c b/lib/mountlist.c
   index 7abe0248e..5f6249dec 100644
   --- a/lib/mountlist.c
   +++ b/lib/mountlist.c
   @@ -164,6 +164,9 @@

#define ME_DUMMY_0(Fs_name, Fs_type)\
  (strcmp (Fs_type, "autofs") == 0  \
   +   || strcmp (Fs_type, "tmpfs") == 0\


tmpfs is consumable, so wouldn't be appropriate to add I think


   +   || strcmp (Fs_type, "devtmpfs") == 0 \
   +   || strcmp (Fs_type, "squashfs") == 0 \
   || strcmp (Fs_type, "proc") == 0 \
   || strcmp (Fs_type, "subfs") == 0\
   /* for Linux 2.6/3.x */  \

bug#37702: Suggestion for 'df' utility

[Bernhard Voelker]

On 2020-05-31 01:07, Paul Eggert wrote:
> On 5/30/20 4:49 AM, Erik Auerswald wrote:
>> I concur that a command line option to override config file (or env var)
>> settings seems useful if a config file and/or env var approach is used.

Just to mention another alternative to control the behavior besides env 
variables or
config files: it would theoretically also be possible to add a './configure' 
option
to add more file system types to the list of pseudofilesystems at build time.
Obviously, such idea moves the responsibility to the packager - which is not
nice either.

> In other utilities we've been moving away from environment variables and/or
> config files for the usual security and other-hassle reasons. So I'd prefer
> having 'df' just do the "right" thing by default, and to have an option to
> override that. The "right" thing should be to ignore all these 
> pseudofilesystems
> that hardly anybody cares about.

+1

What about to start with this?

  $ GIT_PAGER= git -C gnulib diff
  diff --git a/lib/mountlist.c b/lib/mountlist.c
  index 7abe0248e..5f6249dec 100644
  --- a/lib/mountlist.c
  +++ b/lib/mountlist.c
  @@ -164,6 +164,9 @@

   #define ME_DUMMY_0(Fs_name, Fs_type)\
 (strcmp (Fs_type, "autofs") == 0  \
  +   || strcmp (Fs_type, "tmpfs") == 0\
  +   || strcmp (Fs_type, "devtmpfs") == 0 \
  +   || strcmp (Fs_type, "squashfs") == 0 \
  || strcmp (Fs_type, "proc") == 0 \
  || strcmp (Fs_type, "subfs") == 0\
  /* for Linux 2.6/3.x */  \

Example output (here on openSUSE:Tumbleweed):

  $ src/df -Th
  Filesystem Type  Size  Used Avail Use% Mounted on
  /dev/sda2  ext4   20G   14G  5.1G  73% /
  /dev/sda5  ext4  591G  278G  313G  48% /media/big_data
  /dev/loop0 ext2   31M   31M 0 100% /FULL_PARTITION_TMPDIR
  /dev/sda3  ext3  109G   90G   14G  87% /home

... thus omitting those mounts:

  $ diff -wu0 <(/usr/bin/df -Th) <(src/df -Th)
  --- /dev/fd/632020-05-31 11:30:29.991664999 +0200
  +++ /dev/fd/622020-05-31 11:30:29.987664947 +0200
  @@ -2,4 +1,0 @@
  -devtmpfs   devtmpfs  9.8G  8.0K  9.8G   1% /dev
  -tmpfs  tmpfs 9.8G  640K  9.8G   1% /dev/shm
  -tmpfs  tmpfs 9.8G   18M  9.8G   1% /run
  -tmpfs  tmpfs 9.8G 0  9.8G   0% /sys/fs/cgroup
  @@ -10 +5,0 @@
  -tmpfs  tmpfs 2.0G   44K  2.0G   1% /run/user/1000

Have a nice day,
Berny

bug#37702: Suggestion for 'df' utility

[John Pye]

Hi all

My vote would be on a "df -a" switch to revert to original behaviour of
visibility of all mounts.

The purpose of "df" is to show "disk free". Hence any filesystems that
are read-only or which are FUSE-mounted one on of the local physical
filesystems, or similar things (what others?) should be suppressed by
default.

Perhaps a message could be included in the new output to indicate that
some filesystems are suppressed, and "df -a" can be used to see all
mounted filesystems. That would allow people to 'autodiscover' the new
default functionality. That message could be kept in place for a few
years, or even indefinitely. The output message could be sent to stderr,
to allow people to continue to pipe the main "df" output into "wc -l" or
"awk" or whatever other unix-style tools they might like to use.

Perhaps it might be hard to decide what to do with network filesystems,
or locally-caching Amazon S3 filesystems, or whatever things like
that... I am sure there are some tricky corner cases that would need to
be considered.

Cheers
JP

On 31/5/20 9:07 am, Paul Eggert wrote:
> On 5/30/20 4:49 AM, Erik Auerswald wrote:
>> I concur that a command line option to override config file (or env var)
>> settings seems useful if a config file and/or env var approach is used.
> In other utilities we've been moving away from environment variables and/or
> config files for the usual security and other-hassle reasons. So I'd prefer
> having 'df' just do the "right" thing by default, and to have an option to
> override that. The "right" thing should be to ignore all these 
> pseudofilesystems
> that hardly anybody cares about.

bug#37702: Suggestion for 'df' utility

[Paul Eggert]

On 5/30/20 4:49 AM, Erik Auerswald wrote:
> I concur that a command line option to override config file (or env var)
> settings seems useful if a config file and/or env var approach is used.

In other utilities we've been moving away from environment variables and/or
config files for the usual security and other-hassle reasons. So I'd prefer
having 'df' just do the "right" thing by default, and to have an option to
override that. The "right" thing should be to ignore all these pseudofilesystems
that hardly anybody cares about.

bug#37702: Suggestion for 'df' utility

[Erik Auerswald]

Hi all,

On 30.05.20 05:18, Bryce Harrington wrote:

On Fri, Oct 11, 2019 at 12:56:20PM -0700, Paul Eggert wrote:

On 10/11/19 11:20 AM, Pádraig Brady wrote:


if you want to exclude nested file systems like that,
you could try:

    alias df='df -x squashfs'


On my Fedora 30 workstation that option doesn't make any difference.
Regardless of whether '-x squashfs' is used, I see this output from 'df':

Filesystem  1K-blocks  Used  Available Use% Mounted on
devtmpfs  4065704 04065704   0% /dev
tmpfs 4081560 366164044944   1% /dev/shm
tmpfs 4081560  16964079864   1% /run
tmpfs 4081560 04081560   0% /sys/fs/cgroup
/dev/sda559614116  16910684   39645412  30% /
tmpfs 4081560   1244081436   1% /tmp
/dev/sda2  1849433716 207781976 1547682948  12% /home
/dev/sda1 50950402444684572044   6% /boot
tmpfs  81631260 816252   1% /run/user/1000

and most of these lines are useless.


In the above example, it seems useful to exclude tmpfs as well:

alias df='df -x tmpfs -x squashfs'

That does remove the "useless" lines from df output on my Ubuntu 18.04
system, to be concrete.

What I do not like about this approach is the lack of an "unexclude"
option to add an excluded filesystem back in.  One could, e.g., use
\df to not use the alias, or use a different name for the alias (e.g.,
dfx), though.

I do remember a time when at least some distributions by default used
tmpfs for /tmp.  In that situation just this tmpfs filesystem should
probably not be excluded from the default df output.


For many years we've put up with the problem of too many filesystems in the
default plain 'df' output, and now's as good a time as any to fix that.
[...]
We can add a flag or two for the rare people who want to see these
normally-useless lines.

[...]
I'd like to help in fixing this issue.
[...]
I've taken a stab at a proof-of-concept implementation of #3, by adding
an environment variable DF_EXCLUDE_FSTYPES.
[...] > Further, even
with a config file users would probably want a cli switch and/or env var
to override the config file settings.


I concur that a command line option to override config file (or env var)
settings seems useful if a config file and/or env var approach is used.
Just as it seems useful to me to allow unsuppressing of output that has
been suppressed as useless by a possible new df default behavior.

HTH,
Erik

bug#37702: Suggestion for 'df' utility

[Bryce Harrington]

On Fri, Oct 11, 2019 at 12:56:20PM -0700, Paul Eggert wrote:
> On 10/11/19 11:20 AM, Pádraig Brady wrote:
> > 
> > if you want to exclude nested file systems like that,
> > you could try:
> > 
> >    alias df='df -x squashfs'
> 
> On my Fedora 30 workstation that option doesn't make any difference.
> Regardless of whether '-x squashfs' is used, I see this output from 'df':
> 
> Filesystem  1K-blocks  Used  Available Use% Mounted on
> devtmpfs  4065704 04065704   0% /dev
> tmpfs 4081560 366164044944   1% /dev/shm
> tmpfs 4081560  16964079864   1% /run
> tmpfs 4081560 04081560   0% /sys/fs/cgroup
> /dev/sda559614116  16910684   39645412  30% /
> tmpfs 4081560   1244081436   1% /tmp
> /dev/sda2  1849433716 207781976 1547682948  12% /home
> /dev/sda1 50950402444684572044   6% /boot
> tmpfs  81631260 816252   1% /run/user/1000
> 
> and most of these lines are useless.
> 
> For many years we've put up with the problem of too many filesystems in the
> default plain 'df' output, and now's as good a time as any to fix that. On
> my workstation there should be only four lines of information, one each for
> /, /home, /boot, and the shared tmpfs area.
> 
> Presumably readonly filesystems should also be omitted by default, since
> they're not something people ordinarily care about.
> 
> We can add a flag or two for the rare people who want to see these
> normally-useless lines.

Hi,

I'd like to help in fixing this issue.  You're right that the popularity
of non-consumable filesystems (tmpfs, squashfs, etc.) has made df harder
to use for its primary use case of checking amount of available disk
space.  (On my current desktop (Ubuntu 20.04) df outputs 39 lines.)  It
seems reasonable to expect the number of these kinds of file systems is
only destined to increase.

It sounds like there are four approaches that could be taken:

  0.  Leave df as is
  1.  Heuristics
  2.  Config files
  3.  Env variable

I've taken a stab at a proof-of-concept implementation of #3, by adding
an environment variable DF_EXCLUDE_FSTYPES.  The current WIP draft of this
POC is available in this branch:


https://github.com/bryceharrington/coreutils/commit/2759c7abb085079abb80c9499ce88eda6f21b4e0

However, before I submit a patch officially, I'd appreciate your advice
on what the right approach to take *should* be.


Several heuristic ideas (approach #1) have been discussed in this bug
report:

  a) exclude all read-only file systems
  b) exclude read-only file systems with no free space
  c) exclude file systems with usage <=1%
  d) exclude devtmpfs and tmpfs explicitly by name
  e) ditto, and also read-only squashfs mounts

Concerns raised so far in this bug report with these heuristics include:
undesired side effects, compatibility troubles caused by changing
defaults, and preferences of package maintainers of different Linux
distributions.  It's unclear to me if there is a consensus yet.


>From a system integrator point of view, a config file approach makes a
lot of sense.  Different distros could set their own policies for what
fstypes to display via /etc/*/df.conf or something, and sysadmins would
be able to override per-system.  If support for a userspace config file
was provided (e.g. ~/.config/coreutils/df.conf, ~/.shellrc or similar)
then it would allow users to customize behavior as well.

However, from what I can tell nothing in coreutils uses config files, so
while config file code is certainly not exotic, this would be charting
new territory for coreutils as a project.  It also seems a bit invasive
to add config file support for just a single parameter.  Further, even
with a config file users would probably want a cli switch and/or env var
to override the config file settings.  I'm curious about your thoughts
and advice on these points particularly.


The env var approach sidesteps some of the above concerns but has issues
of its own.  First, depending on which startup file they're placed in,
they may not be visible to all services, or to different tool
environments such as sudo, su, systemd-nspawn, lxc exec, chroot,
schroot, etc.  Second, a top-level env var will consume a bit of memory
- small, but present even in processes that will never call df.


Given the above, I still think the env var approach strikes the right
balance for coreutils.  However, I would very much appreciate your
advice and direction, and look forward to finding a satisfactory
resolution for this problem.

Thanks,
Bryce

bug#37702: Suggestion for 'df' utility

[L A Walsh]

On 2019/10/11 12:56, Paul Eggert wrote:
> On 10/11/19 11:20 AM, Pádraig Brady wrote:
>   
>> if you want to exclude nested file systems like that,
>> you could try:
>>alias df='df -x squashfs'
>> 
> On my Fedora 30 workstation that option doesn't make any difference. 
>   
--
I'd suggest adjusting the exclusion list to the file systems you want to
exclude.

I.e.
alias df='df -x tmpfs -x devtmpfs -x loop'

That works on my system as well as:
alias df='df -t xfs'

Certainly prompting the user for wanted defaults and adding them to
their .shellrc could be part of an initial run of *some* utilities
that wouldn't cause hard-coded side-effects.

bug#37702: Suggestion for 'df' utility

[Paul Eggert]

On 10/14/19 1:01 AM, Kamil Dudka wrote:

This is not an excuse to introduce new problems.
I'm not looking for an "excuse". df (through no fault of its own) has 
evolved into a bad program that needs fixing. Backward compatibility 
concerns are real and we should take them into account, but they should 
not be an "excuse" for refusing to fix a bad program.

bug#37702: Suggestion for 'df' utility

[Kamil Dudka]

On Monday, October 14, 2019 8:06:47 AM CEST Paul Eggert wrote:
> On 10/13/19 3:00 PM, Assaf Gordon wrote:
> > I'm not sure if it's easy to find a set of criteria
> > that would work well while having minimal unexpected side effects of
> > hiding
> > entries people in other systems do expect to see.
> 
> No matter what we do (even if we do nothing), there will be problems.

This is not an excuse to introduce new problems.

> But doing
> nothing is clearly a bad idea, as the output of plain df is quite bad
> right now in typical use. We can do better than that, even if we cannot be
> perfect and we cause problems by changing defaults.

I would prefer _not_ to change the default behavior (including the behavior
of commonly used options).  There are downstream consumers who care about 
compatibility in the first place.

Kamil

> > Out of curiosity,
> > can you share the output of the following commands on the same system?
> 
> Sure, here it is:
> 
> $ lsblk
> NAME  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
> loop0   7:00 140.7M  1 loop  /snap/gnome-3-26-1604/90
> loop1   7:10  44.2M  1 loop  /snap/gtk-common-themes/1353
> loop2   7:20 149.9M  1 loop  /snap/gnome-3-28-1804/71
> loop3   7:30   3.7M  1 loop  /snap/gnome-system-monitor/100
> loop4   7:40  14.8M  1 loop  /snap/gnome-characters/317
> loop5   7:50   956K  1 loop  /snap/gnome-logs/73
> loop6   7:60 149.9M  1 loop  /snap/gnome-3-28-1804/67
> loop7   7:70   3.7M  1 loop  /snap/gnome-system-monitor/95
> loop8   7:80 4M  1 loop  /snap/gnome-calculator/406
> loop9   7:90  54.5M  1 loop  /snap/core18/1192
> loop10  7:10   089M  1 loop  /snap/core/7713
> loop11  7:11   0  42.8M  1 loop  /snap/gtk-common-themes/1313
> loop12  7:12   0  14.8M  1 loop  /snap/gnome-characters/296
> loop13  7:13   0 140.7M  1 loop  /snap/gnome-3-26-1604/92
> loop14  7:14   0  89.1M  1 loop  /snap/core/7917
> loop15  7:15   0   956K  1 loop  /snap/gnome-logs/81
> loop16  7:16   0   4.2M  1 loop  /snap/gnome-calculator/501
> loop17  7:17   0  54.4M  1 loop  /snap/core18/1144
> sda 8:00 111.8G  0 disk
> ├─sda1  8:10  96.9G  0 part  /
> ├─sda2  8:20 1K  0 part
> └─sda5  8:5015G  0 part  [SWAP]
> sdb 8:16   0   2.7T  0 disk
> └─sdb1  8:17   0   2.7T  0 part
>└─md127   9:127  0   2.7T  0 raid1
>  └─md127p1 259:00   2.7T  0 md/home
> sdc 8:32   0   2.7T  0 disk
> └─sdc1  8:33   0   2.7T  0 part
>└─md127   9:127  0   2.7T  0 raid1
>  └─md127p1 259:00   2.7T  0 md/home
> sdd 8:48   1   7.5G  0 disk
> └─sdd1  8:49   1   7.5G  0 part  /media/eggert/B827-D456
> sr011:01  1024M  0 rom
> $ df -x tmpfs -x devtmpfs -x squashfs
> Filesystem  1K-blocks   Used  Available Use% Mounted on
> /dev/sda199431552   11740452   82597212  13% /
> /dev/md127p1   2884021472 1326329584 1411168908  49% /home
> /dev/sdd1 781286447051363107728  61% /media/eggert/B827-D456

bug#37702: Suggestion for 'df' utility

[Paul Eggert]

On 10/13/19 3:00 PM, Assaf Gordon wrote:


I'm not sure if it's easy to find a set of criteria
that would work well while having minimal unexpected side effects of hiding 
entries people in other systems do expect to see.


No matter what we do (even if we do nothing), there will be problems. But doing 
nothing is clearly a bad idea, as the output of plain df is quite bad right now 
in typical use. We can do better than that, even if we cannot be perfect and we 
cause problems by changing defaults.



Out of curiosity,
can you share the output of the following commands on the same system?


Sure, here it is:

$ lsblk
NAME  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop0   7:00 140.7M  1 loop  /snap/gnome-3-26-1604/90
loop1   7:10  44.2M  1 loop  /snap/gtk-common-themes/1353
loop2   7:20 149.9M  1 loop  /snap/gnome-3-28-1804/71
loop3   7:30   3.7M  1 loop  /snap/gnome-system-monitor/100
loop4   7:40  14.8M  1 loop  /snap/gnome-characters/317
loop5   7:50   956K  1 loop  /snap/gnome-logs/73
loop6   7:60 149.9M  1 loop  /snap/gnome-3-28-1804/67
loop7   7:70   3.7M  1 loop  /snap/gnome-system-monitor/95
loop8   7:80 4M  1 loop  /snap/gnome-calculator/406
loop9   7:90  54.5M  1 loop  /snap/core18/1192
loop10  7:10   089M  1 loop  /snap/core/7713
loop11  7:11   0  42.8M  1 loop  /snap/gtk-common-themes/1313
loop12  7:12   0  14.8M  1 loop  /snap/gnome-characters/296
loop13  7:13   0 140.7M  1 loop  /snap/gnome-3-26-1604/92
loop14  7:14   0  89.1M  1 loop  /snap/core/7917
loop15  7:15   0   956K  1 loop  /snap/gnome-logs/81
loop16  7:16   0   4.2M  1 loop  /snap/gnome-calculator/501
loop17  7:17   0  54.4M  1 loop  /snap/core18/1144
sda 8:00 111.8G  0 disk
├─sda1  8:10  96.9G  0 part  /
├─sda2  8:20 1K  0 part
└─sda5  8:5015G  0 part  [SWAP]
sdb 8:16   0   2.7T  0 disk
└─sdb1  8:17   0   2.7T  0 part
  └─md127   9:127  0   2.7T  0 raid1
└─md127p1 259:00   2.7T  0 md/home
sdc 8:32   0   2.7T  0 disk
└─sdc1  8:33   0   2.7T  0 part
  └─md127   9:127  0   2.7T  0 raid1
└─md127p1 259:00   2.7T  0 md/home
sdd 8:48   1   7.5G  0 disk
└─sdd1  8:49   1   7.5G  0 part  /media/eggert/B827-D456
sr011:01  1024M  0 rom
$ df -x tmpfs -x devtmpfs -x squashfs
Filesystem  1K-blocks   Used  Available Use% Mounted on
/dev/sda199431552   11740452   82597212  13% /
/dev/md127p1   2884021472 1326329584 1411168908  49% /home
/dev/sdd1 781286447051363107728  61% /media/eggert/B827-D456

bug#37702: Suggestion for 'df' utility

[Bernhard Voelker]

On 2019-10-14 00:13, Assaf Gordon wrote:
> Also in other systems where "/tmp" is a "tmpfs",
> users might want to see how much space is available.
> 
> If we hide it by default, they can of course use "df /tmp"
> or "df --all" - it's not about removing this option,
> it is just about making users' life harder or easier,
> and making unexpected changes.

I see the point ... and agree.

>> This seems to be a small change, and not satisfying the snap case.
> 
> Possibly hiding "squashfs" of readonly-mounts could get rid of those snaps?

Possibly.
But still the same question: how many users would like to see them
per default, and how many do not?
I would not like to argue for something like a ~/.dfrc, but it seems there
are a lot of different flavors of wishes out there.
Yet leaving the default seems to be a wise choice to me.

FWIW: df has the -x option, but if someone would use it in an alias like
"df -x sqashfs", then there's currently no chance to override it again:

  $ alias df='/usr/bin/df -x tmpfs'
  $ df -t tmpfs
  df: file system type 'tmpfs' both selected and excluded

Have a nice day,
Berny

bug#37702: Suggestion for 'df' utility

[Assaf Gordon]

Hello Bernhard,

On 2019-10-13 3:57 p.m., Bernhard Voelker wrote:

On 2019-10-13 23:28, Paul Eggert wrote:

In any sane system there would be only
four lines of non-header output (for tmpfs etc, /, /home, and
/media/eggert/B827-D456), but df is outputting 28 lines.


What is so special about tmpfs so that you would like to see it?


As an interesting use-case (though not common),
I recently configured a raspberry PI device,
and wanted to mount as many locations on tmpfs as possible,
e.g. "/tmp" "/var/tmp", "/var/log" etc.

In was very useful in those cases to be able to see separate
tmpfs file system listed, with information about how big they
are and how much space was used.

Also in other systems where "/tmp" is a "tmpfs",
users might want to see how much space is available.

If we hide it by default, they can of course use "df /tmp"
or "df --all" - it's not about removing this option,
it is just about making users' life harder or easier,
and making unexpected changes.


I recently also encountered a change in a default behavior of
a program which I've been using a very long time - and it is *very*
frustrating to have something that worked "just fine" for so long
being changed.



Here on my openSUSE:Tumbleweed system, I see the following:

   $ df -T
   Filesystem Type 1K-blocks  Used Available Use% Mounted on

[...]

   /dev/loop0 ext2 31729 31729 0 100% 
/FULL_PARTITION_TMPDIR

[...]


(The /FULL_PARTITION_TMPDIR is used by a special coreutils test.)



That's an interesting case, where I would think you'd want to see it,
because you explicitly mounted it.



I think I could well live with adding 'devtmpfs' and 'tmpfs' to the
pseudo file systems in gnulib's "mountlist.c".


I agree, but think this needs to be communicated very well,
and in advance - perhaps announce this change ahead of time to
the respective package maintainers of each distribution - just so
they'll know it's coming (and also have a way to revert it if they don't
like it).



This seems to be a small change, and not satisfying the snap case.


Possibly hiding "squashfs" of readonly-mounts could get rid of those snaps?

regards,
  -assaf

bug#37702: Suggestion for 'df' utility

[Assaf Gordon]

On 2019-10-13 3:28 p.m., Paul Eggert wrote:
[..]
I mean c'mon, here's the output of 'df' on the Ubuntu 18.04.3 LTS 
workstation I'm typing this particular message on. In any sane system 
there would be only four lines of non-header output (for tmpfs etc, /, 
/home, and /media/eggert/B827-D456), but df is outputting 28 lines. This 
is ridiculous.




It is certainly inconvenient if that's not what you are looking for
(and certainly most desktop users aren't).

But I'm not sure if it's easy to find a set of criteria
that would work well while having minimal unexpected side effects of 
hiding entries people in other systems do expect to see.


Out of curiosity,
can you share the output of the following commands on the same system?

lsblk

df -x tmpfs -x devtmpfs -x squashfs


Thanks,
 - assaf

bug#37702: Suggestion for 'df' utility

[Bernhard Voelker]

On 2019-10-13 23:28, Paul Eggert wrote:
> In any sane system there would be only 
> four lines of non-header output (for tmpfs etc, /, /home, and 
> /media/eggert/B827-D456), but df is outputting 28 lines.

What is so special about tmpfs so that you would like to see it?

Here on my openSUSE:Tumbleweed system, I see the following:

  $ df -T
  Filesystem Type 1K-blocks  Used Available Use% Mounted on
  devtmpfs   devtmpfs  10187924 8  10187916   1% /dev
  tmpfs  tmpfs 10199048 45788  10153260   1% /dev/shm
  tmpfs  tmpfs 10199048 18036  10181012   1% /run
  tmpfs  tmpfs 10199048 0  10199048   0% /sys/fs/cgroup
  /dev/sda2  ext4  20511312  12141864   7304488  63% /
  /dev/loop0 ext2 31729 31729 0 100% 
/FULL_PARTITION_TMPDIR
  /dev/sda5  ext4 619142920 390088908 229037628  64% /media/big_data
  /dev/sda3  ext3 103085876  90714416   7128248  93% /home
  tmpfs  tmpfs  203980820   2039788   1% /run/user/1000

(The /FULL_PARTITION_TMPDIR is used by a special coreutils test.)

I think I could well live with adding 'devtmpfs' and 'tmpfs' to the
pseudo file systems in gnulib's "mountlist.c".

This seems to be a small change, and not satisfying the snap case.
Yet, I agree with Assaf that changing the defaults has to be done
with caution.  Eliding r/o filesystems or where usage<1% doesn't
look like such.

Have a nice day,
Berny

bug#37702: Suggestion for 'df' utility

[Paul Eggert]

On 10/13/19 2:11 PM, Assaf Gordon wrote:


This thread originated by a request to "clean up" the output on newer
ubuntu machines which use "snap" packages as /dev/loopN .

Let's not turn that into a drastic change


It could certainly be multiple sets of patches. But let's face it, df's utility 
for ordinary interactive use has degraded significantly with time due to all the 
random filesystems people have been adding, and we shouldn't keep our heads in 
the sands about this. df's default needs to change, one way or another.


I mean c'mon, here's the output of 'df' on the Ubuntu 18.04.3 LTS workstation 
I'm typing this particular message on. In any sane system there would be only 
four lines of non-header output (for tmpfs etc, /, /home, and 
/media/eggert/B827-D456), but df is outputting 28 lines. This is ridiculous.


Filesystem  1K-blocks   Used  Available Use% Mounted on
udev  7644704  07644704   0% /dev
tmpfs 1533620   19241531696   1% /run
/dev/sda199431552   11740340   82597324  13% /
tmpfs 7668096  332127634884   1% /dev/shm
tmpfs5120  8   5112   1% /run/lock
tmpfs 7668096  07668096   0% /sys/fs/cgroup
/dev/loop0 144128 144128  0 100% /snap/gnome-3-26-1604/90
/dev/loop1  45312  45312  0 100% 
/snap/gtk-common-themes/1353
/dev/loop2 153600 153600  0 100% /snap/gnome-3-28-1804/71
/dev/loop4  15104  15104  0 100% /snap/gnome-characters/317
/dev/loop6 153600 153600  0 100% /snap/gnome-3-28-1804/67
/dev/loop5   1024   1024  0 100% /snap/gnome-logs/73
/dev/loop3   3840   3840  0 100% 
/snap/gnome-system-monitor/100
/dev/loop10 91264  91264  0 100% /snap/core/7713
/dev/loop7   3840   3840  0 100% 
/snap/gnome-system-monitor/95
/dev/loop8   4224   4224  0 100% /snap/gnome-calculator/406
/dev/loop9  55808  55808  0 100% /snap/core18/1192
/dev/loop11 43904  43904  0 100% 
/snap/gtk-common-themes/1313
/dev/loop12 15104  15104  0 100% /snap/gnome-characters/296
/dev/loop13144128 144128  0 100% /snap/gnome-3-26-1604/92
/dev/loop14 91264  91264  0 100% /snap/core/7917
/dev/loop15  1024   1024  0 100% /snap/gnome-logs/81
/dev/loop17 55808  55808  0 100% /snap/core18/1144
/dev/loop16  4352   4352  0 100% /snap/gnome-calculator/501
/dev/md127p1   2884021472 1326255744 1411242748  49% /home
tmpfs 1533616 161533600   1% /run/user/121
tmpfs 1533616 601533556   1% /run/user/1000
/dev/sdd1 781286447051363107728  61% /media/eggert/B827-D456

bug#37702: Suggestion for 'df' utility

[Assaf Gordon]

Hi all,

On 2019-10-13 2:27 p.m., Paul Eggert wrote:

On 10/13/19 2:41 AM, Pádraig Brady wrote:

I wonder could we key (also) on used==0||available==0.


Yes, looking at the sample output I gave earlier, I'd say we could by 
default drop filesystems where usage is 1% or less. That would solve the 
problem for my workstation. This is roughly akin to the "used==0" test 
you're suggesting.




I would humbly suggest caution with such unexpected user-facing changes
to the default output of 'df' - learning the lessons from changing the 
quotes in 'ls'.


Countless users have been using 'df' in their own ways, and have gotten
used to certain outputs.

This thread originated by a request to "clean up" the output on newer
ubuntu machines which use "snap" packages as /dev/loopN .

Let's not turn that into a drastic change that will affect many other
existing systems - the users on other systems did not ask for any changes.

---

Specifically for "default drop filesystems where usage is 1% or less" -
I can think of few cases off the top of my head where this would be
extremely confusing:

- I recently installed a 33TB raid file system. The usage on that system
is at %1 and will stay like so for at least several days.

- Amazon cloud services (AWS) offers an NFS4 service (they call it 
"EFS") that has reported size of 8 exabytes. There too usage could be at 
%1 for a long long time.


---


For cases where I want to list only the "real" storage, I typically use
an alias such as:

   alias dff='df -h -x tmpfs -x devtmpfs'

And it would be very easy and least disruptive to recommend
to ubuntu users to add "-x squashfs" or another file system to ignore.


Perhaps we can come up with a recommended list of "lesser" file systems
to ignore (or conditions such as read-only file
systems) and add it as a new option, but please let's not make it the
default.



My two cents,
 - assaf

bug#37702: Suggestion for 'df' utility

[Paul Eggert]

On 10/13/19 2:41 AM, Pádraig Brady wrote:

I wonder could we key (also) on used==0||available==0.


Yes, looking at the sample output I gave earlier, I'd say we could by default 
drop filesystems where usage is 1% or less. That would solve the problem for my 
workstation. This is roughly akin to the "used==0" test you're suggesting.


(I don't know if this would address the problem of small snap loop devices. I 
haven't seen sample df output from that.)


What you mean by "available==0"? Actual zero-size filesystems, or filesystems 
whose size is less than some epsilon? What would the epsilon be? Can you give an 
example?

bug#37702: Suggestion for 'df' utility

[Pádraig Brady]

On 11/10/2019 20:56, Paul Eggert wrote:

On 10/11/19 11:20 AM, Pádraig Brady wrote:


if you want to exclude nested file systems like that,
you could try:

    alias df='df -x squashfs'


On my Fedora 30 workstation that option doesn't make any difference.
Regardless of whether '-x squashfs' is used, I see this output from 'df':

Filesystem  1K-blocks  Used  Available Use% Mounted on
devtmpfs  4065704 04065704   0% /dev
tmpfs 4081560 366164044944   1% /dev/shm
tmpfs 4081560  16964079864   1% /run
tmpfs 4081560 04081560   0% /sys/fs/cgroup
/dev/sda559614116  16910684   39645412  30% /
tmpfs 4081560   1244081436   1% /tmp
/dev/sda2  1849433716 207781976 1547682948  12% /home
/dev/sda1 50950402444684572044   6% /boot
tmpfs  81631260 816252   1% /run/user/1000

and most of these lines are useless.

For many years we've put up with the problem of too many filesystems in
the default plain 'df' output, and now's as good a time as any to fix
that. On my workstation there should be only four lines of information,
one each for /, /home, /boot, and the shared tmpfs area.

Presumably readonly filesystems should also be omitted by default, since
they're not something people ordinarily care about.

We can add a flag or two for the rare people who want to see these
normally-useless lines.


What df tries to show now is a deduplicated list of file systems
that have consumable storage. Maybe read-only could be valid to drop
by default, but I've worked on systems where certain file-systems
were read-only most of the time, but they were consumable.
I wonder could we key (also) on used==0||available==0.

cheers,
Pádraig

bug#37702: Suggestion for 'df' utility

[Paul Eggert]

On 10/11/19 11:20 AM, Pádraig Brady wrote:


if you want to exclude nested file systems like that,
you could try:

   alias df='df -x squashfs'


On my Fedora 30 workstation that option doesn't make any difference. 
Regardless of whether '-x squashfs' is used, I see this output from 'df':


Filesystem  1K-blocks  Used  Available Use% Mounted on
devtmpfs  4065704 04065704   0% /dev
tmpfs 4081560 366164044944   1% /dev/shm
tmpfs 4081560  16964079864   1% /run
tmpfs 4081560 04081560   0% /sys/fs/cgroup
/dev/sda559614116  16910684   39645412  30% /
tmpfs 4081560   1244081436   1% /tmp
/dev/sda2  1849433716 207781976 1547682948  12% /home
/dev/sda1 50950402444684572044   6% /boot
tmpfs  81631260 816252   1% /run/user/1000

and most of these lines are useless.

For many years we've put up with the problem of too many filesystems in 
the default plain 'df' output, and now's as good a time as any to fix 
that. On my workstation there should be only four lines of information, 
one each for /, /home, /boot, and the shared tmpfs area.


Presumably readonly filesystems should also be omitted by default, since 
they're not something people ordinarily care about.


We can add a flag or two for the rare people who want to see these 
normally-useless lines.

bug#37702: Suggestion for 'df' utility

[Pádraig Brady]

On 11/10/2019 08:55, John Pye wrote:

Hi there

I got this email address from the 'df' man page. Hope it's still active
and the right place for feedback.

With recent changes to Ubuntu and the increasing use of 'snap' packaging
for all sorts of things, the 'df' utility output is now quite jumbled
and messy. My fairly normal system now has 17 /dev/loopNN mounts for
various different software packages. I would rather not see these things
by default. Or at least, I would like to have a convenient
single-character flag that allows me to suppress those mounts from my
'df' output.

One useful approach would be a flag that suppresses read-only
filesystems. Another approach would be to suppress read-only filesystems
that have no free space. Perhaps there are other good approaches too.

I don't know if this issue is also relevant for 'docker' or not.


Well if you want to exclude nested file systems like that,
you could try:

  alias df='df -x squashfs'

cheers,
Pádraig

bug#37702: Suggestion for 'df' utility

[John Pye]

Hi there

I got this email address from the 'df' man page. Hope it's still active
and the right place for feedback.

With recent changes to Ubuntu and the increasing use of 'snap' packaging
for all sorts of things, the 'df' utility output is now quite jumbled
and messy. My fairly normal system now has 17 /dev/loopNN mounts for
various different software packages. I would rather not see these things
by default. Or at least, I would like to have a convenient
single-character flag that allows me to suppress those mounts from my
'df' output.

One useful approach would be a flag that suppresses read-only
filesystems. Another approach would be to suppress read-only filesystems
that have no free space. Perhaps there are other good approaches too.

I don't know if this issue is also relevant for 'docker' or not.

Cheers
JP

https://askubuntu.com/questions/834093/small-snap-loop-devices-visible-in-gnome-disk-utility-or-what-is-the-function-of