Re: [RFC][PATCH 00/12] Enhanced file stat system call

[David Howells]

Christoph Hellwig  wrote:

> from a quick look the statx bits looks fine in general.  I think Ted
> last time had a problem with the IOC flag allocation, so you might
> want to ping him.

Yeah - he commented on that.

> But fsinfo with the multiplexer and the void pointer is just horrible.
> What were you thinking there?

I think the fsinfo data struct probably wants splitting up.  Now this could be
done in a number of ways, including:

 (1) By adding multiple syscalls (statfsx, fsinfo, netfsinfo, ...) but each
 one needs a bit in the kernel to handle the basics (path/fd lookup,
 security check, buffer allocation and freeing, ...) which could all be in
 common - hence the mux method.

 (2) Adding an argument to the fsinfo syscall since it has at least one
 syscall argument spare.

 (3) Offloading some of the bits to standardised xattr calls.  The large
 string fields (domain name, volume name, ...) would seem to be obvious
 candidates for this.

Given that the core VFS gets to manage the contents of the buffer, it
shouldn't be as controversial as pioctl().

David
--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC][PATCH 00/12] Enhanced file stat system call

[Andreas Dilger]

On Nov 25, 2015, at 10:51 AM, J. Bruce Fields  wrote:
> 
> On Fri, Nov 20, 2015 at 04:28:35PM +, David Howells wrote:
>> Martin Steigerwald  wrote:
>> 
>>> Any plans to add limitations of filesystem to the call like maximum file
>>> size?  I know its mostly relevant for just for FAT32, but on any account
>>> rather than trying to write 4 GiB and then file, it would be good to at some
>>> time get a dialog at the beginning of the copy.
>> 
>> Adding filesystem limits can be done.  I got a shopping list of things people
>> wanted a while back and I've worked off of that list.  I can add other things
>> - that's on of the reasons I left room for expansion.
> 
> I ran across systemd/src/basic/path-util.c:fd_is_mount_point() the other
> day, and the contortions it goes through made me wonder if we should
> also add mnt_id and/or an is_mountpoint boolean--it's annoying to have
> to do name_to_handle_at() (not supported on all filesystems) just to get
> mnt_id.
> 
> (Looking at it now I see it falls back on reading mount id from
> /proc/self/fdinfo/.  Maybe that's good enough.  May depend on
> whether there's a potential user that doesn't want to assume access to
> /proc?)

IMHO, it should be possible to get information about a file or directory
from the file itself (i.e. statx() or fsinfo() on the path/fd), rather than
having to grub around in a /proc file that the application magically has to
know about, and parse text files there for every file being handled.

Cheers, Andreas







signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [RFC][PATCH 00/12] Enhanced file stat system call

[Andreas Dilger]

On Nov 20, 2015, at 9:50 AM, Casey Schaufler  wrote:
> On 11/20/2015 6:54 AM, David Howells wrote:
>> Implement new system calls to provide enhanced file stats and enhanced
>> filesystem stats.  The patches can be found here:
>> 
>>  
>> http://git.kernel.org/cgit/linux/kernel/git/dhowells/linux-fs.git/log/?h=xstat
>> 
>> 
>> ===
>> DESCRIPTION
>> ===
>> 
>> The third patch provides this new system call:
>> 
>>  long ret = statx(int dfd,
>>   const char *filename,
>>   unsigned atflag,
>>   unsigned mask,
>>   struct statx *buffer);
>> 
>> This is an enhanced file stat function that provides a number of useful
>> features, in summary:
>> 
>>  (1) More information: creation time, data version number,
>>  flags/attributes.  A subset of these is available through a number of
>>  filesystems (such as CIFS, NFS, AFS, Ext4 and BTRFS).
>> 
>>  (2) Lightweight stat (AT_NO_ATTR_SYNC): Ask for just those details of
>>  interest, and allow a network fs to approximate anything not of
>>  interest, without going to the server.
>> 
>>  (3) Heavyweight stat (AT_FORCE_ATTR_SYNC): Force a network fs to flush
>>  buffers and go to the server, even if it thinks its cached attributes
>>  are up to date.
>> 
>>  (4) Allow the filesystem to indicate what it can/cannot provide: A
>>  filesystem can now say it doesn't support a standard stat feature if
>>  that isn't available.
>> 
>>  (5) Make the fields a consistent size on all arches, and make them large.
>> 
>>  (6) Can be extended by using more request flags and using up the padding
>>  space in the statx struct.
> 
> How about relevant xattrs? SELinux context, ACL, that sort of thing.
> The fact that these are optional should be taken care of by (4).

Given that there are a wide variety of xattrs that different apps might be
interested in, this would probably be better served by an enhancement to
getxattr() or listxattr() to be able to retrieve a whole list of xattrs
at once, possibly with some wildcard support (e.g. "security.*") instead
of returning all or a specific subset of xattrs with statx() (which is
geared toward fixed-size attributes).

Cheers, Andreas


>> Note that no lstat() equivalent is required as that can be implemented
>> through statx() with atflag == 0.  There is also no fstat() equivalent as
>> that can be implemented through statx() with filename == NULL and the
>> relevant fd passed as dfd.
>> 
>> 
>> The seventh patch provides another new system call:
>> 
>>  long ret = fsinfo(int dfd,
>>const char *filename,
>>unsigned atflag,
>>unsigned request,
>>void *buffer);
>> 
>> This is an enhanced filesystem stat and information retrieval function that
>> provides more information, in summary:
>> 
>>  (1) All the information provided by statfs() and more.  The fields are
>>  made large.
>> 
>>  (2) Provides information about timestamp range and resolution to
>>  complement statx().
>> 
>>  (3) Provides information about IOC flags supported in statx()'s return.
>> 
>>  (4) Provides volume binary IDs and UUIDs.
>> 
>>  (5) Provides the filesystem name according to the kernel as a string
>>  (eg. "ext4" or "nfs3") in addition to the magic number.
>> 
>>  (6) Provides information obtained from network filesystems, such as volume
>>  and domain names.
>> 
>>  (7) Has lots of spare space that can be used for future extenstions and a
>>  bit mask indicating what was provided.
>> 
>> Note that I've added a 'request' identifier.  This is to select the set of
>> data to be returned.  The idea is that 'buffer' points to a fixed-size
>> struct selected by request.  Currently only 0 is available and this refers
>> to 'struct fsinfo'.  However, I could split up the buffer into say 3:
>> 
>>  (0) statfs-type information
>> 
>>  (1) Timestamp and IOC flags info.
>> 
>>  (2) Network fs strings.
>> 
>> However, some of this might be better retrieved through getxattr().
>> 
>> 
>> ===
>> TESTING
>> ===
>> 
>> Test programs are added into samples/statx/ by the appropriate patches.
>> 
>> David
>> ---
>> David Howells (12):
>>   Ext4: Fix extended timestamp encoding and decoding
>>   statx: Provide IOC flags for Windows fs attributes
>>   statx: Add a system call to make enhanced file info available
>>   statx: AFS: Return enhanced file attributes
>>   statx: Ext4: Return enhanced file attributes
>>   statx: NFS: Return enhanced file attributes
>>   statx: CIFS: Return enhanced attributes
>>   fsinfo: Add a system call to make enhanced filesystem info available
>>   fsinfo: Ext4: Return information through the filesystem info syscall
>>   fsinfo: AFS: Return information through the filesystem info syscall
>>   fsinfo: NFS: Return information 

Re: [RFC][PATCH 00/12] Enhanced file stat system call

[Christoph Hellwig]

On Fri, Nov 20, 2015 at 05:19:31PM +0100, Martin Steigerwald wrote:
> I know its mostly relevant for just for FAT32, but on any account rather than 
> trying to write 4 GiB and then file, it would be good to at some time get a 
> dialog at the beginning of the copy.

pathconf/fpathconf is supposed to handle that.  It's not super pretty
but part of Posix.  Linus hates it, but it might be time to give it
another try.
--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC][PATCH 00/12] Enhanced file stat system call

[Christoph Hellwig]

On Fri, Nov 20, 2015 at 08:50:22AM -0800, Casey Schaufler wrote:
> How about relevant xattrs? SELinux context, ACL, that sort of thing.
> The fact that these are optional should be taken care of by (4).

Those are not simple, fixed size stat data and would make the system
call a giant mess.
--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC][PATCH 00/12] Enhanced file stat system call

[Martin Steigerwald]

Am Dienstag, 24. November 2015, 00:13:08 CET schrieb Christoph Hellwig:
> On Fri, Nov 20, 2015 at 05:19:31PM +0100, Martin Steigerwald wrote:
> > I know its mostly relevant for just for FAT32, but on any account rather
> > than trying to write 4 GiB and then file, it would be good to at some
> > time get a dialog at the beginning of the copy.
> 
> pathconf/fpathconf is supposed to handle that.  It's not super pretty
> but part of Posix.  Linus hates it, but it might be time to give it
> another try.

It might be interesting for BTRFS as well, to be able to ask what amount of 
free space there currently is *at* a given path. Cause with BTRFS and 
Subvolumes this may differ between different paths. Even tough its not 
implemented yet, it may be possible in the future to have one subvolume with 
RAID 1 profile and one with RAID 0 profile.

That said an application wanting to make sure it can write a certain amount of 
data can use fallocate. And thats thats the only reliable way to ensure it, I 
know of. Which can become tedious for several files, but there is no principal 
problem with preallocating all files if their sizes are known. Even rsync or 
desktop environments could work like that. First fallocate everything, then, 
only if that succeeds, start actually copying data. Disadvantage: On aborted 
copies you have all files with their correct sizes and no easy indicates on 
where the copy stopped.

Thanks,
-- 
Martin
--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC][PATCH 00/12] Enhanced file stat system call

[Christoph Hellwig]

On Tue, Nov 24, 2015 at 09:48:22AM +0100, Martin Steigerwald wrote:
> It might be interesting for BTRFS as well, to be able to ask what amount of 
> free space there currently is *at* a given path. Cause with BTRFS and 
> Subvolumes this may differ between different paths.

We can handle this trivial with the current statfs interface.  Take a
look at xfs_fs_statfs and xfs_qm_statvfs.
--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC][PATCH 00/12] Enhanced file stat system call

[Casey Schaufler]

On 11/24/2015 12:15 AM, Christoph Hellwig wrote:

On Fri, Nov 20, 2015 at 08:50:22AM -0800, Casey Schaufler wrote:

How about relevant xattrs? SELinux context, ACL, that sort of thing.
The fact that these are optional should be taken care of by (4).

Those are not simple, fixed size stat data and would make the system
call a giant mess.


I didn't say it would be easy. I do think that adding a system call
that only deals with simple, fixed size data is going to fall short
of solving "the problem".

Actually, a Smack label is fixed size (256 bytes). I suspect there
is a maximum for SELinux contexts as well. ACLs I'll grant you are
infinite.

--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC][PATCH 00/12] Enhanced file stat system call

[Martin Steigerwald]

Am Freitag, 20. November 2015, 14:54:22 CET schrieb David Howells:
> The seventh patch provides another new system call:
> 
> long ret = fsinfo(int dfd,
>   const char *filename,
>   unsigned atflag,
>   unsigned request,
>   void *buffer);
> 
> This is an enhanced filesystem stat and information retrieval function that
> provides more information, in summary:
> 
>  (1) All the information provided by statfs() and more.  The fields are
>  made large.
> 
>  (2) Provides information about timestamp range and resolution to
>  complement statx().
> 
>  (3) Provides information about IOC flags supported in statx()'s return.
> 
>  (4) Provides volume binary IDs and UUIDs.
> 
>  (5) Provides the filesystem name according to the kernel as a string
>  (eg. "ext4" or "nfs3") in addition to the magic number.
> 
>  (6) Provides information obtained from network filesystems, such as volume
>  and domain names.
> 
>  (7) Has lots of spare space that can be used for future extenstions and a
>  bit mask indicating what was provided.

Any plans to add limitations of filesystem to the call like maximum file size? 
I know its mostly relevant for just for FAT32, but on any account rather than 
trying to write 4 GiB and then file, it would be good to at some time get a 
dialog at the beginning of the copy.

Well, but okay, maybe its use case is too limited as FAT32 is not an in any 
kind modern filesystem anymore and limits of modern filesystems are much 
higher.

But other limits like maximum amount of extended attributes, maximum amount of 
acls or symlinks on one directory may be nice to query. Symlinks for BTRFS 
without extended symlink support and acls maybe for XFS, I remember there at 
least has been a limit at some time that was quite low.

Thanks,
-- 
Martin
--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC][PATCH 00/12] Enhanced file stat system call

[Martin Steigerwald]

Cc linux-afs dropped due to bounce:

: host vger.kernel.org[209.132.180.67] said: 554
5.0.0 Hi [194.150.191.11], unresolvable address:
; nosuchuser; linux-...@vger.kernel.org (in
reply to RCPT TO command)

Am Freitag, 20. November 2015, 16:28:35 CET schrieb David Howells:
> Martin Steigerwald  wrote:
> > Any plans to add limitations of filesystem to the call like maximum file
> > size?  I know its mostly relevant for just for FAT32, but on any account
> > rather than trying to write 4 GiB and then file, it would be good to at
> > some time get a dialog at the beginning of the copy.
> 
> Adding filesystem limits can be done.  I got a shopping list of things
> people wanted a while back and I've worked off of that list.  I can add
> other things - that's on of the reasons I left room for expansion.

Cool, thank you for bringing this forward.

I am teaching in my Linux trainings since years that one day one will be able 
to see file creation time with stat shell command, actually mentioning ideas 
to make some xstat/statx system call. :)

Thanks,
-- 
Martin
--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC][PATCH 00/12] Enhanced file stat system call

[David Howells]

Martin Steigerwald  wrote:

> Any plans to add limitations of filesystem to the call like maximum file
> size?  I know its mostly relevant for just for FAT32, but on any account
> rather than trying to write 4 GiB and then file, it would be good to at some
> time get a dialog at the beginning of the copy.

Adding filesystem limits can be done.  I got a shopping list of things people
wanted a while back and I've worked off of that list.  I can add other things
- that's on of the reasons I left room for expansion.

David
--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC][PATCH 00/12] Enhanced file stat system call

[Casey Schaufler]

On 11/20/2015 6:54 AM, David Howells wrote:

Implement new system calls to provide enhanced file stats and enhanced
filesystem stats.  The patches can be found here:


http://git.kernel.org/cgit/linux/kernel/git/dhowells/linux-fs.git/log/?h=xstat


===
DESCRIPTION
===

The third patch provides this new system call:

long ret = statx(int dfd,
 const char *filename,
 unsigned atflag,
 unsigned mask,
 struct statx *buffer);

This is an enhanced file stat function that provides a number of useful
features, in summary:

  (1) More information: creation time, data version number,
  flags/attributes.  A subset of these is available through a number of
  filesystems (such as CIFS, NFS, AFS, Ext4 and BTRFS).

  (2) Lightweight stat (AT_NO_ATTR_SYNC): Ask for just those details of
  interest, and allow a network fs to approximate anything not of
  interest, without going to the server.

  (3) Heavyweight stat (AT_FORCE_ATTR_SYNC): Force a network fs to flush
  buffers and go to the server, even if it thinks its cached attributes
  are up to date.

  (4) Allow the filesystem to indicate what it can/cannot provide: A
  filesystem can now say it doesn't support a standard stat feature if
  that isn't available.

  (5) Make the fields a consistent size on all arches, and make them large.

  (6) Can be extended by using more request flags and using up the padding
  space in the statx struct.


How about relevant xattrs? SELinux context, ACL, that sort of thing.
The fact that these are optional should be taken care of by (4).



Note that no lstat() equivalent is required as that can be implemented
through statx() with atflag == 0.  There is also no fstat() equivalent as
that can be implemented through statx() with filename == NULL and the
relevant fd passed as dfd.


The seventh patch provides another new system call:

long ret = fsinfo(int dfd,
  const char *filename,
  unsigned atflag,
  unsigned request,
  void *buffer);

This is an enhanced filesystem stat and information retrieval function that
provides more information, in summary:

  (1) All the information provided by statfs() and more.  The fields are
  made large.

  (2) Provides information about timestamp range and resolution to
  complement statx().

  (3) Provides information about IOC flags supported in statx()'s return.

  (4) Provides volume binary IDs and UUIDs.

  (5) Provides the filesystem name according to the kernel as a string
  (eg. "ext4" or "nfs3") in addition to the magic number.

  (6) Provides information obtained from network filesystems, such as volume
  and domain names.

  (7) Has lots of spare space that can be used for future extenstions and a
  bit mask indicating what was provided.

Note that I've added a 'request' identifier.  This is to select the set of
data to be returned.  The idea is that 'buffer' points to a fixed-size
struct selected by request.  Currently only 0 is available and this refers
to 'struct fsinfo'.  However, I could split up the buffer into say 3:

  (0) statfs-type information

  (1) Timestamp and IOC flags info.

  (2) Network fs strings.

However, some of this might be better retrieved through getxattr().


===
TESTING
===

Test programs are added into samples/statx/ by the appropriate patches.

David
---
David Howells (12):
   Ext4: Fix extended timestamp encoding and decoding
   statx: Provide IOC flags for Windows fs attributes
   statx: Add a system call to make enhanced file info available
   statx: AFS: Return enhanced file attributes
   statx: Ext4: Return enhanced file attributes
   statx: NFS: Return enhanced file attributes
   statx: CIFS: Return enhanced attributes
   fsinfo: Add a system call to make enhanced filesystem info available
   fsinfo: Ext4: Return information through the filesystem info syscall
   fsinfo: AFS: Return information through the filesystem info syscall
   fsinfo: NFS: Return information through the filesystem info syscall
   fsinfo: CIFS: Return information through the filesystem info syscall


  arch/x86/entry/syscalls/syscall_32.tbl |2
  arch/x86/entry/syscalls/syscall_64.tbl |2
  fs/afs/inode.c |   23 ++
  fs/afs/super.c |   39 
  fs/cifs/cifsfs.c   |   25 +++
  fs/cifs/cifsfs.h   |4
  fs/cifs/cifsglob.h |8 +
  fs/cifs/dir.c  |2
  fs/cifs/inode.c|  124 ++---
  fs/cifs/netmisc.c  |4
  fs/exportfs/expfs.c|4
  fs/ext4/ext4.h |   24 ++-
  fs/ext4/file.c |