Re: [RFC][PATCH 00/12] Enhanced file stat system call
Christoph Hellwigwrote: > from a quick look the statx bits looks fine in general. I think Ted > last time had a problem with the IOC flag allocation, so you might > want to ping him. Yeah - he commented on that. > But fsinfo with the multiplexer and the void pointer is just horrible. > What were you thinking there? I think the fsinfo data struct probably wants splitting up. Now this could be done in a number of ways, including: (1) By adding multiple syscalls (statfsx, fsinfo, netfsinfo, ...) but each one needs a bit in the kernel to handle the basics (path/fd lookup, security check, buffer allocation and freeing, ...) which could all be in common - hence the mux method. (2) Adding an argument to the fsinfo syscall since it has at least one syscall argument spare. (3) Offloading some of the bits to standardised xattr calls. The large string fields (domain name, volume name, ...) would seem to be obvious candidates for this. Given that the core VFS gets to manage the contents of the buffer, it shouldn't be as controversial as pioctl(). David -- To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC][PATCH 00/12] Enhanced file stat system call
On Nov 25, 2015, at 10:51 AM, J. Bruce Fieldswrote: > > On Fri, Nov 20, 2015 at 04:28:35PM +, David Howells wrote: >> Martin Steigerwald wrote: >> >>> Any plans to add limitations of filesystem to the call like maximum file >>> size? I know its mostly relevant for just for FAT32, but on any account >>> rather than trying to write 4 GiB and then file, it would be good to at some >>> time get a dialog at the beginning of the copy. >> >> Adding filesystem limits can be done. I got a shopping list of things people >> wanted a while back and I've worked off of that list. I can add other things >> - that's on of the reasons I left room for expansion. > > I ran across systemd/src/basic/path-util.c:fd_is_mount_point() the other > day, and the contortions it goes through made me wonder if we should > also add mnt_id and/or an is_mountpoint boolean--it's annoying to have > to do name_to_handle_at() (not supported on all filesystems) just to get > mnt_id. > > (Looking at it now I see it falls back on reading mount id from > /proc/self/fdinfo/. Maybe that's good enough. May depend on > whether there's a potential user that doesn't want to assume access to > /proc?) IMHO, it should be possible to get information about a file or directory from the file itself (i.e. statx() or fsinfo() on the path/fd), rather than having to grub around in a /proc file that the application magically has to know about, and parse text files there for every file being handled. Cheers, Andreas signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [RFC][PATCH 00/12] Enhanced file stat system call
On Nov 20, 2015, at 9:50 AM, Casey Schauflerwrote: > On 11/20/2015 6:54 AM, David Howells wrote: >> Implement new system calls to provide enhanced file stats and enhanced >> filesystem stats. The patches can be found here: >> >> >> http://git.kernel.org/cgit/linux/kernel/git/dhowells/linux-fs.git/log/?h=xstat >> >> >> === >> DESCRIPTION >> === >> >> The third patch provides this new system call: >> >> long ret = statx(int dfd, >> const char *filename, >> unsigned atflag, >> unsigned mask, >> struct statx *buffer); >> >> This is an enhanced file stat function that provides a number of useful >> features, in summary: >> >> (1) More information: creation time, data version number, >> flags/attributes. A subset of these is available through a number of >> filesystems (such as CIFS, NFS, AFS, Ext4 and BTRFS). >> >> (2) Lightweight stat (AT_NO_ATTR_SYNC): Ask for just those details of >> interest, and allow a network fs to approximate anything not of >> interest, without going to the server. >> >> (3) Heavyweight stat (AT_FORCE_ATTR_SYNC): Force a network fs to flush >> buffers and go to the server, even if it thinks its cached attributes >> are up to date. >> >> (4) Allow the filesystem to indicate what it can/cannot provide: A >> filesystem can now say it doesn't support a standard stat feature if >> that isn't available. >> >> (5) Make the fields a consistent size on all arches, and make them large. >> >> (6) Can be extended by using more request flags and using up the padding >> space in the statx struct. > > How about relevant xattrs? SELinux context, ACL, that sort of thing. > The fact that these are optional should be taken care of by (4). Given that there are a wide variety of xattrs that different apps might be interested in, this would probably be better served by an enhancement to getxattr() or listxattr() to be able to retrieve a whole list of xattrs at once, possibly with some wildcard support (e.g. "security.*") instead of returning all or a specific subset of xattrs with statx() (which is geared toward fixed-size attributes). Cheers, Andreas >> Note that no lstat() equivalent is required as that can be implemented >> through statx() with atflag == 0. There is also no fstat() equivalent as >> that can be implemented through statx() with filename == NULL and the >> relevant fd passed as dfd. >> >> >> The seventh patch provides another new system call: >> >> long ret = fsinfo(int dfd, >>const char *filename, >>unsigned atflag, >>unsigned request, >>void *buffer); >> >> This is an enhanced filesystem stat and information retrieval function that >> provides more information, in summary: >> >> (1) All the information provided by statfs() and more. The fields are >> made large. >> >> (2) Provides information about timestamp range and resolution to >> complement statx(). >> >> (3) Provides information about IOC flags supported in statx()'s return. >> >> (4) Provides volume binary IDs and UUIDs. >> >> (5) Provides the filesystem name according to the kernel as a string >> (eg. "ext4" or "nfs3") in addition to the magic number. >> >> (6) Provides information obtained from network filesystems, such as volume >> and domain names. >> >> (7) Has lots of spare space that can be used for future extenstions and a >> bit mask indicating what was provided. >> >> Note that I've added a 'request' identifier. This is to select the set of >> data to be returned. The idea is that 'buffer' points to a fixed-size >> struct selected by request. Currently only 0 is available and this refers >> to 'struct fsinfo'. However, I could split up the buffer into say 3: >> >> (0) statfs-type information >> >> (1) Timestamp and IOC flags info. >> >> (2) Network fs strings. >> >> However, some of this might be better retrieved through getxattr(). >> >> >> === >> TESTING >> === >> >> Test programs are added into samples/statx/ by the appropriate patches. >> >> David >> --- >> David Howells (12): >> Ext4: Fix extended timestamp encoding and decoding >> statx: Provide IOC flags for Windows fs attributes >> statx: Add a system call to make enhanced file info available >> statx: AFS: Return enhanced file attributes >> statx: Ext4: Return enhanced file attributes >> statx: NFS: Return enhanced file attributes >> statx: CIFS: Return enhanced attributes >> fsinfo: Add a system call to make enhanced filesystem info available >> fsinfo: Ext4: Return information through the filesystem info syscall >> fsinfo: AFS: Return information through the filesystem info syscall >> fsinfo: NFS: Return information
Re: [RFC][PATCH 00/12] Enhanced file stat system call
On Fri, Nov 20, 2015 at 05:19:31PM +0100, Martin Steigerwald wrote: > I know its mostly relevant for just for FAT32, but on any account rather than > trying to write 4 GiB and then file, it would be good to at some time get a > dialog at the beginning of the copy. pathconf/fpathconf is supposed to handle that. It's not super pretty but part of Posix. Linus hates it, but it might be time to give it another try. -- To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC][PATCH 00/12] Enhanced file stat system call
On Fri, Nov 20, 2015 at 08:50:22AM -0800, Casey Schaufler wrote: > How about relevant xattrs? SELinux context, ACL, that sort of thing. > The fact that these are optional should be taken care of by (4). Those are not simple, fixed size stat data and would make the system call a giant mess. -- To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC][PATCH 00/12] Enhanced file stat system call
Am Dienstag, 24. November 2015, 00:13:08 CET schrieb Christoph Hellwig: > On Fri, Nov 20, 2015 at 05:19:31PM +0100, Martin Steigerwald wrote: > > I know its mostly relevant for just for FAT32, but on any account rather > > than trying to write 4 GiB and then file, it would be good to at some > > time get a dialog at the beginning of the copy. > > pathconf/fpathconf is supposed to handle that. It's not super pretty > but part of Posix. Linus hates it, but it might be time to give it > another try. It might be interesting for BTRFS as well, to be able to ask what amount of free space there currently is *at* a given path. Cause with BTRFS and Subvolumes this may differ between different paths. Even tough its not implemented yet, it may be possible in the future to have one subvolume with RAID 1 profile and one with RAID 0 profile. That said an application wanting to make sure it can write a certain amount of data can use fallocate. And thats thats the only reliable way to ensure it, I know of. Which can become tedious for several files, but there is no principal problem with preallocating all files if their sizes are known. Even rsync or desktop environments could work like that. First fallocate everything, then, only if that succeeds, start actually copying data. Disadvantage: On aborted copies you have all files with their correct sizes and no easy indicates on where the copy stopped. Thanks, -- Martin -- To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC][PATCH 00/12] Enhanced file stat system call
On Tue, Nov 24, 2015 at 09:48:22AM +0100, Martin Steigerwald wrote: > It might be interesting for BTRFS as well, to be able to ask what amount of > free space there currently is *at* a given path. Cause with BTRFS and > Subvolumes this may differ between different paths. We can handle this trivial with the current statfs interface. Take a look at xfs_fs_statfs and xfs_qm_statvfs. -- To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC][PATCH 00/12] Enhanced file stat system call
On 11/24/2015 12:15 AM, Christoph Hellwig wrote: On Fri, Nov 20, 2015 at 08:50:22AM -0800, Casey Schaufler wrote: How about relevant xattrs? SELinux context, ACL, that sort of thing. The fact that these are optional should be taken care of by (4). Those are not simple, fixed size stat data and would make the system call a giant mess. I didn't say it would be easy. I do think that adding a system call that only deals with simple, fixed size data is going to fall short of solving "the problem". Actually, a Smack label is fixed size (256 bytes). I suspect there is a maximum for SELinux contexts as well. ACLs I'll grant you are infinite. -- To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC][PATCH 00/12] Enhanced file stat system call
Am Freitag, 20. November 2015, 14:54:22 CET schrieb David Howells: > The seventh patch provides another new system call: > > long ret = fsinfo(int dfd, > const char *filename, > unsigned atflag, > unsigned request, > void *buffer); > > This is an enhanced filesystem stat and information retrieval function that > provides more information, in summary: > > (1) All the information provided by statfs() and more. The fields are > made large. > > (2) Provides information about timestamp range and resolution to > complement statx(). > > (3) Provides information about IOC flags supported in statx()'s return. > > (4) Provides volume binary IDs and UUIDs. > > (5) Provides the filesystem name according to the kernel as a string > (eg. "ext4" or "nfs3") in addition to the magic number. > > (6) Provides information obtained from network filesystems, such as volume > and domain names. > > (7) Has lots of spare space that can be used for future extenstions and a > bit mask indicating what was provided. Any plans to add limitations of filesystem to the call like maximum file size? I know its mostly relevant for just for FAT32, but on any account rather than trying to write 4 GiB and then file, it would be good to at some time get a dialog at the beginning of the copy. Well, but okay, maybe its use case is too limited as FAT32 is not an in any kind modern filesystem anymore and limits of modern filesystems are much higher. But other limits like maximum amount of extended attributes, maximum amount of acls or symlinks on one directory may be nice to query. Symlinks for BTRFS without extended symlink support and acls maybe for XFS, I remember there at least has been a limit at some time that was quite low. Thanks, -- Martin -- To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC][PATCH 00/12] Enhanced file stat system call
Cc linux-afs dropped due to bounce:: host vger.kernel.org[209.132.180.67] said: 554 5.0.0 Hi [194.150.191.11], unresolvable address: ; nosuchuser; linux-...@vger.kernel.org (in reply to RCPT TO command) Am Freitag, 20. November 2015, 16:28:35 CET schrieb David Howells: > Martin Steigerwald wrote: > > Any plans to add limitations of filesystem to the call like maximum file > > size? I know its mostly relevant for just for FAT32, but on any account > > rather than trying to write 4 GiB and then file, it would be good to at > > some time get a dialog at the beginning of the copy. > > Adding filesystem limits can be done. I got a shopping list of things > people wanted a while back and I've worked off of that list. I can add > other things - that's on of the reasons I left room for expansion. Cool, thank you for bringing this forward. I am teaching in my Linux trainings since years that one day one will be able to see file creation time with stat shell command, actually mentioning ideas to make some xstat/statx system call. :) Thanks, -- Martin -- To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC][PATCH 00/12] Enhanced file stat system call
Martin Steigerwaldwrote: > Any plans to add limitations of filesystem to the call like maximum file > size? I know its mostly relevant for just for FAT32, but on any account > rather than trying to write 4 GiB and then file, it would be good to at some > time get a dialog at the beginning of the copy. Adding filesystem limits can be done. I got a shopping list of things people wanted a while back and I've worked off of that list. I can add other things - that's on of the reasons I left room for expansion. David -- To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC][PATCH 00/12] Enhanced file stat system call
On 11/20/2015 6:54 AM, David Howells wrote: Implement new system calls to provide enhanced file stats and enhanced filesystem stats. The patches can be found here: http://git.kernel.org/cgit/linux/kernel/git/dhowells/linux-fs.git/log/?h=xstat === DESCRIPTION === The third patch provides this new system call: long ret = statx(int dfd, const char *filename, unsigned atflag, unsigned mask, struct statx *buffer); This is an enhanced file stat function that provides a number of useful features, in summary: (1) More information: creation time, data version number, flags/attributes. A subset of these is available through a number of filesystems (such as CIFS, NFS, AFS, Ext4 and BTRFS). (2) Lightweight stat (AT_NO_ATTR_SYNC): Ask for just those details of interest, and allow a network fs to approximate anything not of interest, without going to the server. (3) Heavyweight stat (AT_FORCE_ATTR_SYNC): Force a network fs to flush buffers and go to the server, even if it thinks its cached attributes are up to date. (4) Allow the filesystem to indicate what it can/cannot provide: A filesystem can now say it doesn't support a standard stat feature if that isn't available. (5) Make the fields a consistent size on all arches, and make them large. (6) Can be extended by using more request flags and using up the padding space in the statx struct. How about relevant xattrs? SELinux context, ACL, that sort of thing. The fact that these are optional should be taken care of by (4). Note that no lstat() equivalent is required as that can be implemented through statx() with atflag == 0. There is also no fstat() equivalent as that can be implemented through statx() with filename == NULL and the relevant fd passed as dfd. The seventh patch provides another new system call: long ret = fsinfo(int dfd, const char *filename, unsigned atflag, unsigned request, void *buffer); This is an enhanced filesystem stat and information retrieval function that provides more information, in summary: (1) All the information provided by statfs() and more. The fields are made large. (2) Provides information about timestamp range and resolution to complement statx(). (3) Provides information about IOC flags supported in statx()'s return. (4) Provides volume binary IDs and UUIDs. (5) Provides the filesystem name according to the kernel as a string (eg. "ext4" or "nfs3") in addition to the magic number. (6) Provides information obtained from network filesystems, such as volume and domain names. (7) Has lots of spare space that can be used for future extenstions and a bit mask indicating what was provided. Note that I've added a 'request' identifier. This is to select the set of data to be returned. The idea is that 'buffer' points to a fixed-size struct selected by request. Currently only 0 is available and this refers to 'struct fsinfo'. However, I could split up the buffer into say 3: (0) statfs-type information (1) Timestamp and IOC flags info. (2) Network fs strings. However, some of this might be better retrieved through getxattr(). === TESTING === Test programs are added into samples/statx/ by the appropriate patches. David --- David Howells (12): Ext4: Fix extended timestamp encoding and decoding statx: Provide IOC flags for Windows fs attributes statx: Add a system call to make enhanced file info available statx: AFS: Return enhanced file attributes statx: Ext4: Return enhanced file attributes statx: NFS: Return enhanced file attributes statx: CIFS: Return enhanced attributes fsinfo: Add a system call to make enhanced filesystem info available fsinfo: Ext4: Return information through the filesystem info syscall fsinfo: AFS: Return information through the filesystem info syscall fsinfo: NFS: Return information through the filesystem info syscall fsinfo: CIFS: Return information through the filesystem info syscall arch/x86/entry/syscalls/syscall_32.tbl |2 arch/x86/entry/syscalls/syscall_64.tbl |2 fs/afs/inode.c | 23 ++ fs/afs/super.c | 39 fs/cifs/cifsfs.c | 25 +++ fs/cifs/cifsfs.h |4 fs/cifs/cifsglob.h |8 + fs/cifs/dir.c |2 fs/cifs/inode.c| 124 ++--- fs/cifs/netmisc.c |4 fs/exportfs/expfs.c|4 fs/ext4/ext4.h | 24 ++- fs/ext4/file.c |