Re: svn commit: r254627 - in head: bin/chflags bin/ls lib/libc/gen lib/libc/sys sys/cddl/contrib/opensolaris/uts/common/fs/zfs sys/fs/msdosfs sys/fs/smbfs sys/sys sys/ufs/ufs
[My mail connection wasn't working back in June when I wrote this. This is the first of many replies to try to prevent breakage of mv. I have now checked what happens for simple tests on ref11. Details in later replies.] On Sat, 28 Jun 2014, Bruce Evans wrote: On Fri, 27 Jun 2014, Kenneth D. Merry wrote: On Fri, Jun 27, 2014 at 12:48:29 -0700, Xin LI wrote: Hi, Craig have hit an interesting issue today, where he tried to 'mv' a file from ZFS dataset to a NFS mount, 'mv' bails out because chflags failed. I think it's probably sensible to have mv ignoring UF_ARCHIVE, and set the flag on the target unconditionally? i.e.: Index: mv.c === --- mv.c (revision 267940) +++ mv.c (working copy) @@ -337,8 +337,8 @@ * on a file that we copied, i.e., that we didn't create.) */ errno = 0; - if (fchflags(to_fd, sbp-st_flags)) - if (errno != EOPNOTSUPP || sbp-st_flags != 0) + if (fchflags(to_fd, sbp-st_flags | UF_ARCHIVE)) + if (errno != EOPNOTSUPP || (sbp-st_flags ~UF_ARCHIVE) != 0) warn(%s: set flags (was: 0%07o), to, sbp-st_flags); tval[0].tv_sec = sbp-st_atime; Yes, that sounds like a good way to do it. No, this is very broken. Ignoring the error is bad enough. POSIX requires duplicating all of the attributes and certain error handling when they cannot be duplicated. File flags aren't a POSIX attribute, but not duplicating or handling errors differently for them them breaks the spirit of the POSIX spec. Forcing the archive flag to be set on the copy is worse. It is broken especially broken if the source and target both support the archive flag, since it then fails to preserve the flag when it is clear on the source. The old code was bad too. I think it usually gives the POSIX behaviour, but it only applies to the unusual case where only a few regular files are moved, and its checking if the preservation worked can be done better by stat()ing the result and comparing with the original. The usual case (by number of files moved, if not by mv instances), is for moving whole directory heirarchies. The above code is not used in that case. cp -pR is used. cp -pR is more buggy than the above in general, but for the chflags() its error handling is less fancy and thus stricter than the above, so tends to produce thousands or warnings instead of only 1. More and different details in another reply. I sent the following mail to ken about this (mostly for cp -p instead of mv) in April, but received no reply: old Copying files on freefall now causes annoying warnings. This is because old zfs supports UF_ARCHIVE but nfs doesn't: old old % Script started on Sat Apr 5 05:10:55 2014 old % pts/29:bde@freefall:~/zmsun cp -p $l/msun/Makefile . old % cp: chflags: ./Makefile: Operation not supported old % pts/29:bde@freefall:~/zmsun echo $? old % 1 old % pts/29:bde@freefall:~/zmsun ls -lo $l/msun/Makefile Makefile old % -rw-r--r-- 1 root wheel uarch 8610 Mar 2 11:00 /usr/src/lib/msun/Makefile old % -rw-r--r-- 1 bde devel - 8610 Mar 2 11:00 Makefile old % pts/29:bde@freefall:~/zmsun exit old % old % Script done on Sat Apr 5 05:11:28 2014 old old cp works, but this is hard to determine since the exit status is 1. Oops, old that means that cp doesn't work. It also cannot copy the more important old uid and gid, but it doesn't warn about this or change the exit status to old 1 for this. Not warning is a historical hack to keep cp usable. Not old indicating the error in any other way is not good, but is also historical. old This is only done when chown() returns EPERM. For chflags() on nfs, we're old getting EOPNOTSUPPORT for the whole syscall. So cp -pR is completely broken for use by mv for the uid and gid, but works almost as correctly as possibly for file flags. POSIX has relaxed requirements for cp relative to mv, since cp without -p is not required to preserve any attributes, and cp with -p can't be expected to preserve all the attributes in many cases, unlike the usual case for mv where it is not across a file system. old The support is useless in practice, at least on freefall, because zfs old always sets UF_ARCHIVE and nothing ever clears it. zfs sets it even old for directories and symlinks. old old Also, sys/stat.h still has SF_ARCHIVED: old old % #define UF_ARCHIVE 0x0800 /* file needs to be archived */ old % #define SF_ARCHIVED 0x0001 /* file is archived */ old old It's not clear what SF_ARCHIVED means, especially since no file system old supports it. The only references to it are: old old % ./fs/tmpfs/tmpfs_subr.c: if ((flags ~(SF_APPEND | SF_ARCHIVED | SF_IMMUTABLE | SF_NOUNLINK | old % ./ufs/ufs/ufs_vnops.c: if ((vap-va_flags ~(SF_APPEND | SF_ARCHIVED | SF_IMMUTABLE | old old So applications can set this flag for ffs and tmpfs, but since the fs never old changes it, it is almost useless. Perhaps we left it for compatibility old in ffs. tmpfs doesn't have
Re: svn commit: r254627 - in head: bin/chflags bin/ls lib/libc/gen lib/libc/sys sys/cddl/contrib/opensolaris/uts/common/fs/zfs sys/fs/msdosfs sys/fs/smbfs sys/sys sys/ufs/ufs
On Fri, 27 Jun 2014, Kenneth D. Merry wrote: On Fri, Jun 27, 2014 at 12:48:29 -0700, Xin LI wrote: Hi, Craig have hit an interesting issue today, where he tried to 'mv' a file from ZFS dataset to a NFS mount, 'mv' bails out because chflags failed. I think it's probably sensible to have mv ignoring UF_ARCHIVE, and set the flag on the target unconditionally? i.e.: Index: mv.c === --- mv.c (revision 267940) +++ mv.c (working copy) @@ -337,8 +337,8 @@ * on a file that we copied, i.e., that we didn't create.) */ errno = 0; - if (fchflags(to_fd, sbp-st_flags)) - if (errno != EOPNOTSUPP || sbp-st_flags != 0) + if (fchflags(to_fd, sbp-st_flags | UF_ARCHIVE)) + if (errno != EOPNOTSUPP || (sbp-st_flags ~UF_ARCHIVE) != 0) warn(%s: set flags (was: 0%07o), to, sbp-st_flags); tval[0].tv_sec = sbp-st_atime; Yes, that sounds like a good way to do it. No, this is very broken. Ignoring the error is bad enough. POSIX requires duplicating all of the attributes and certain error handling when they cannot be duplicated. File flags aren't a POSIX attribute, but not duplicating or handling errors differently for them them breaks the spirit of the POSIX spec. Forcing the archive flag to be set on the copy is worse. It is broken especially broken if the source and target both support the archive flag, since it then fails to preserve the flag when it is clear on the source. The old code was bad too. I think it usually gives the POSIX behaviour, but it only applies to the unusual case where only a few regular files are moved, and its checking if the preservation worked can be done better by stat()ing the result and comparing with the original. The usual case (by number of files moved, if not by mv instances), is for moving whole directory heirarchies. The above code is not used in that case. cp -pR is used. cp -pR is more buggy than the above in general, but for the chflags() its error handling is less fancy and thus stricter than the above, so tends to produce thousands or warnings instead of only 1. More and different details in another reply. I sent the following mail to ken about this (mostly for cp -p instead of mv) in April, but received no reply: old Copying files on freefall now causes annoying warnings. This is because old zfs supports UF_ARCHIVE but nfs doesn't: old old % Script started on Sat Apr 5 05:10:55 2014 old % pts/29:bde@freefall:~/zmsun cp -p $l/msun/Makefile . old % cp: chflags: ./Makefile: Operation not supported old % pts/29:bde@freefall:~/zmsun echo $? old % 1 old % pts/29:bde@freefall:~/zmsun ls -lo $l/msun/Makefile Makefile old % -rw-r--r-- 1 root wheel uarch 8610 Mar 2 11:00 /usr/src/lib/msun/Makefile old % -rw-r--r-- 1 bde devel - 8610 Mar 2 11:00 Makefile old % pts/29:bde@freefall:~/zmsun exit old % old % Script done on Sat Apr 5 05:11:28 2014 old old cp works, but this is hard to determine since the exit status is 1. Oops, old that means that cp doesn't work. It also cannot copy the more important old uid and gid, but it doesn't warn about this or change the exit status to old 1 for this. Not warning is a historical hack to keep cp usable. Not old indicating the error in any other way is not good, but is also historical. old This is only done when chown() returns EPERM. For chflags() on nfs, we're old getting EOPNOTSUPPORT for the whole syscall. So cp -pR is completely broken for use by mv for the uid and gid, but works almost as correctly as possibly for file flags. POSIX has relaxed requirements for cp relative to mv, since cp without -p is not required to preserve any attributes, and cp with -p can't be expected to preserve all the attributes in many cases, unlike the usual case for mv where it is not across a file system. old The support is useless in practice, at least on freefall, because zfs old always sets UF_ARCHIVE and nothing ever clears it. zfs sets it even old for directories and symlinks. old old Also, sys/stat.h still has SF_ARCHIVED: old old % #define UF_ARCHIVE 0x0800 /* file needs to be archived */ old % #define SF_ARCHIVED 0x0001 /* file is archived */ old old It's not clear what SF_ARCHIVED means, especially since no file system old supports it. The only references to it are: old old % ./fs/tmpfs/tmpfs_subr.c: if ((flags ~(SF_APPEND | SF_ARCHIVED | SF_IMMUTABLE | SF_NOUNLINK | old % ./ufs/ufs/ufs_vnops.c:if ((vap-va_flags ~(SF_APPEND | SF_ARCHIVED | SF_IMMUTABLE | old old So applications can set this flag for ffs and tmpfs, but since the fs never old changes it, it is almost useless. Perhaps we left it for compatibility old in ffs. tmpfs doesn't have anything to be compatible with. old old UF_ARCHIVE and UF_NODUMP are fairly bogus for tmpfs too. I think all they old do is prevent the above error when copying files from fs's that support old them. old old Bruce Attributes like file times cannot
Re: svn commit: r254627 - in head: bin/chflags bin/ls lib/libc/gen lib/libc/sys sys/cddl/contrib/opensolaris/uts/common/fs/zfs sys/fs/msdosfs sys/fs/smbfs sys/sys sys/ufs/ufs
On Fri, Jun 27, 2014 at 12:48:29 -0700, Xin LI wrote: Hi, Craig have hit an interesting issue today, where he tried to 'mv' a file from ZFS dataset to a NFS mount, 'mv' bails out because chflags failed. I think it's probably sensible to have mv ignoring UF_ARCHIVE, and set the flag on the target unconditionally? i.e.: Index: mv.c === --- mv.c (revision 267940) +++ mv.c (working copy) @@ -337,8 +337,8 @@ * on a file that we copied, i.e., that we didn't create.) */ errno = 0; - if (fchflags(to_fd, sbp-st_flags)) - if (errno != EOPNOTSUPP || sbp-st_flags != 0) + if (fchflags(to_fd, sbp-st_flags | UF_ARCHIVE)) + if (errno != EOPNOTSUPP || (sbp-st_flags ~UF_ARCHIVE) != 0) warn(%s: set flags (was: 0%07o), to, sbp-st_flags); tval[0].tv_sec = sbp-st_atime; Yes, that sounds like a good way to do it. Ken -- Kenneth Merry k...@freebsd.org ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r254627 - in head: bin/chflags bin/ls lib/libc/gen lib/libc/sys sys/cddl/contrib/opensolaris/uts/common/fs/zfs sys/fs/msdosfs sys/fs/smbfs sys/sys sys/ufs/ufs
On Fri, Jun 27, 2014 at 12:48:29PM -0700 I heard the voice of Xin LI, and lo! it spake thus: Craig have hit an interesting issue today, where he tried to 'mv' a file from ZFS dataset to a NFS mount, 'mv' bails out because chflags failed. I've been getting these for a while on my -CURRENT box, where I mv things from ZFS to NFS quite often, but it doesn't bail out; it keep on going and just yells at me about every file. I tried smacking it around to fix it, but I wound up spilling my coffee instead 8-} -- Matthew Fuller (MF4839) | fulle...@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r254627 - in head: bin/chflags bin/ls lib/libc/gen lib/libc/sys sys/cddl/contrib/opensolaris/uts/common/fs/zfs sys/fs/msdosfs sys/fs/smbfs sys/sys sys/ufs/ufs
On 2013-08-21 18:04, Kenneth D. Merry wrote: Author: ken Date: Wed Aug 21 23:04:48 2013 New Revision: 254627 URL: http://svnweb.freebsd.org/changeset/base/254627 Log: Expand the use of stat(2) flags to allow storing some Windows/DOS and CIFS file attributes as BSD stat(2) flags. This work is intended to be compatible with ZFS, the Solaris CIFS server's interaction with ZFS, somewhat compatible with MacOS X, and of course compatible with Windows. The Windows attributes that are implemented were chosen based on the attributes that ZFS already supports. The summary of the flags is as follows: [...] UF_ARCHIVE: Command line name: uarch, uarchive ZFS_NAME: XAT_ARCHIVE, ZFS_ARCHIVE Windows name: FILE_ATTRIBUTE_ARCHIVE The UF_ARCHIVED flag means that the file has changed and needs to be archived. The meaning is same as the Windows FILE_ATTRIBUTE_ARCHIVE attribute, and the ZFS XAT_ARCHIVE and ZFS_ARCHIVE attribute. msdosfs and ZFS have special handling for this flag. i.e. they will set it when the file changes. Is it intended that this flag is automatically added to all new and existing ZFS files? # touch test # ls -alo test -rw-r--r--- 1 root wheel uarch 0 Aug 28 15:46 test This breaks 'cp -p' to tmpfs as tmpfs does not allow this flag. # mkdir /tmp/tmpfs # mount -t tmpfs tmpfs /tmp/tmpfs # cp -f test /tmp/tmpfs cp: test: Operation not supported [...] Modified: head/sys/ufs/ufs/ufs_vnops.c == --- head/sys/ufs/ufs/ufs_vnops.cWed Aug 21 22:57:29 2013 (r254626) +++ head/sys/ufs/ufs/ufs_vnops.cWed Aug 21 23:04:48 2013 (r254627) @@ -528,9 +528,11 @@ ufs_setattr(ap) return (EINVAL); } if (vap-va_flags != VNOVAL) { - if ((vap-va_flags ~(UF_NODUMP | UF_IMMUTABLE | UF_APPEND | - UF_OPAQUE | UF_NOUNLINK | SF_ARCHIVED | SF_IMMUTABLE | - SF_APPEND | SF_NOUNLINK | SF_SNAPSHOT)) != 0) + if ((vap-va_flags ~(SF_APPEND | SF_ARCHIVED | SF_IMMUTABLE | + SF_NOUNLINK | SF_SNAPSHOT | UF_APPEND | UF_ARCHIVE | + UF_HIDDEN | UF_IMMUTABLE | UF_NODUMP | UF_NOUNLINK | + UF_OFFLINE | UF_OPAQUE | UF_READONLY | UF_REPARSE | + UF_SPARSE | UF_SYSTEM)) != 0) return (EOPNOTSUPP); if (vp-v_mount-mnt_flag MNT_RDONLY) return (EROFS); Seems a similar change is needed in tmpfs_subr.c:tmpfs_chflags() (antoine pointed this out) ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r254627 - in head: bin/chflags bin/ls lib/libc/gen lib/libc/sys sys/cddl/contrib/opensolaris/uts/common/fs/zfs sys/fs/msdosfs sys/fs/smbfs sys/sys sys/ufs/ufs
On Wed, Aug 28, 2013 at 15:54:18 -0500, Bryan Drewery wrote: On 2013-08-21 18:04, Kenneth D. Merry wrote: Author: ken Date: Wed Aug 21 23:04:48 2013 New Revision: 254627 URL: http://svnweb.freebsd.org/changeset/base/254627 Log: Expand the use of stat(2) flags to allow storing some Windows/DOS and CIFS file attributes as BSD stat(2) flags. This work is intended to be compatible with ZFS, the Solaris CIFS server's interaction with ZFS, somewhat compatible with MacOS X, and of course compatible with Windows. The Windows attributes that are implemented were chosen based on the attributes that ZFS already supports. The summary of the flags is as follows: [...] UF_ARCHIVE:Command line name: uarch, uarchive ZFS_NAME: XAT_ARCHIVE, ZFS_ARCHIVE Windows name: FILE_ATTRIBUTE_ARCHIVE The UF_ARCHIVED flag means that the file has changed and needs to be archived. The meaning is same as the Windows FILE_ATTRIBUTE_ARCHIVE attribute, and the ZFS XAT_ARCHIVE and ZFS_ARCHIVE attribute. msdosfs and ZFS have special handling for this flag. i.e. they will set it when the file changes. Is it intended that this flag is automatically added to all new and existing ZFS files? Yes, that is intentional. ZFS already has the flag internally, this just exposes it to FreeBSD. With ZFS at least, it is set any time a file changes. If an application clears it on files it has archived, it will know when the file has changed and it needs to archive the file again. This is the inverse (more or less) of the SF_ARCHIVED flag. With UFS, the flag is just passed through and stored. One application for this is to support CIFS servers that need to store DOS/CIFS/Windows attributes on a FreeBSD server. # touch test # ls -alo test -rw-r--r--- 1 root wheel uarch 0 Aug 28 15:46 test This breaks 'cp -p' to tmpfs as tmpfs does not allow this flag. # mkdir /tmp/tmpfs # mount -t tmpfs tmpfs /tmp/tmpfs # cp -f test /tmp/tmpfs cp: test: Operation not supported Right. For some filesystems, like UFS and probably tmpfs, the right answer may be to just pass through most all of the file flags, and reject specific flags that it doesn't support. For some filesystems, like smbfs, the right answer seems to be to just ignore flags that are set that aren't supported. Other filesystems, like msdosfs, support a small number of flags and reject any that aren't supported. In other words, this isn't a new problem, and it would have cropped up if you tried to copy a file with the SF_IMMUTABLE flag set from UFS to msdosfs with cp -p. [...] Modified: head/sys/ufs/ufs/ufs_vnops.c == --- head/sys/ufs/ufs/ufs_vnops.c Wed Aug 21 22:57:29 2013 (r254626) +++ head/sys/ufs/ufs/ufs_vnops.c Wed Aug 21 23:04:48 2013 (r254627) @@ -528,9 +528,11 @@ ufs_setattr(ap) return (EINVAL); } if (vap-va_flags != VNOVAL) { -if ((vap-va_flags ~(UF_NODUMP | UF_IMMUTABLE | UF_APPEND | -UF_OPAQUE | UF_NOUNLINK | SF_ARCHIVED | SF_IMMUTABLE | -SF_APPEND | SF_NOUNLINK | SF_SNAPSHOT)) != 0) +if ((vap-va_flags ~(SF_APPEND | SF_ARCHIVED | SF_IMMUTABLE | +SF_NOUNLINK | SF_SNAPSHOT | UF_APPEND | UF_ARCHIVE | +UF_HIDDEN | UF_IMMUTABLE | UF_NODUMP | UF_NOUNLINK | +UF_OFFLINE | UF_OPAQUE | UF_READONLY | UF_REPARSE | +UF_SPARSE | UF_SYSTEM)) != 0) return (EOPNOTSUPP); if (vp-v_mount-mnt_flag MNT_RDONLY) return (EROFS); Seems a similar change is needed in tmpfs_subr.c:tmpfs_chflags() (antoine pointed this out) Sure, I can fix tmpfs. Ken -- Kenneth Merry k...@freebsd.org ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
svn commit: r254627 - in head: bin/chflags bin/ls lib/libc/gen lib/libc/sys sys/cddl/contrib/opensolaris/uts/common/fs/zfs sys/fs/msdosfs sys/fs/smbfs sys/sys sys/ufs/ufs
Author: ken Date: Wed Aug 21 23:04:48 2013 New Revision: 254627 URL: http://svnweb.freebsd.org/changeset/base/254627 Log: Expand the use of stat(2) flags to allow storing some Windows/DOS and CIFS file attributes as BSD stat(2) flags. This work is intended to be compatible with ZFS, the Solaris CIFS server's interaction with ZFS, somewhat compatible with MacOS X, and of course compatible with Windows. The Windows attributes that are implemented were chosen based on the attributes that ZFS already supports. The summary of the flags is as follows: UF_SYSTEM:Command line name: system or usystem ZFS name: XAT_SYSTEM, ZFS_SYSTEM Windows: FILE_ATTRIBUTE_SYSTEM This flag means that the file is used by the operating system. FreeBSD does not enforce any special handling when this flag is set. UF_SPARSE:Command line name: sparse or usparse ZFS name: XAT_SPARSE, ZFS_SPARSE Windows: FILE_ATTRIBUTE_SPARSE_FILE This flag means that the file is sparse. Although ZFS may modify this in some situations, there is not generally any special handling for this flag. UF_OFFLINE: Command line name: offline or uoffline ZFS name: XAT_OFFLINE, ZFS_OFFLINE Windows: FILE_ATTRIBUTE_OFFLINE This flag means that the file has been moved to offline storage. FreeBSD does not have any special handling for this flag. UF_REPARSE: Command line name: reparse or ureparse ZFS name: XAT_REPARSE, ZFS_REPARSE Windows: FILE_ATTRIBUTE_REPARSE_POINT This flag means that the file is a Windows reparse point. ZFS has special handling code for reparse points, but we don't currently have the other supporting infrastructure for them. UF_HIDDEN:Command line name: hidden or uhidden ZFS name: XAT_HIDDEN, ZFS_HIDDEN Windows: FILE_ATTRIBUTE_HIDDEN This flag means that the file may be excluded from a directory listing if the application honors it. FreeBSD has no special handling for this flag. The name and bit definition for UF_HIDDEN are identical to the definition in MacOS X. UF_READONLY: Command line name: urdonly, rdonly, readonly ZFS name: XAT_READONLY, ZFS_READONLY Windows: FILE_ATTRIBUTE_READONLY This flag means that the file may not written or appended, but its attributes may be changed. ZFS currently enforces this flag, but Illumos developers have discussed disabling enforcement. The behavior of this flag is different than MacOS X. MacOS X uses UF_IMMUTABLE to represent the DOS readonly permission, but that flag has a stronger meaning than the semantics of DOS readonly permissions. UF_ARCHIVE: Command line name: uarch, uarchive ZFS_NAME: XAT_ARCHIVE, ZFS_ARCHIVE Windows name: FILE_ATTRIBUTE_ARCHIVE The UF_ARCHIVED flag means that the file has changed and needs to be archived. The meaning is same as the Windows FILE_ATTRIBUTE_ARCHIVE attribute, and the ZFS XAT_ARCHIVE and ZFS_ARCHIVE attribute. msdosfs and ZFS have special handling for this flag. i.e. they will set it when the file changes. sys/param.h: Bump __FreeBSD_version to 147 for the addition of new stat(2) flags. chflags.1:Document the new command line flag names (e.g. system, hidden) available to the user. ls.1: Reference chflags(1) for a list of file flags and their meanings. strtofflags.c:Implement the mapping between the new command line flag names and new stat(2) flags. chflags.2:Document all of the new stat(2) flags, and explain the intended behavior in a little more detail. Explain how they map to Windows file attributes. Different filesystems behave differently with respect to flags, so warn the application developer to take care when using them. zfs_vnops.c: Add support for getting and setting the UF_ARCHIVE, UF_READONLY, UF_SYSTEM, UF_HIDDEN, UF_REPARSE, UF_OFFLINE, and UF_SPARSE flags.