Re: Status of pipefail for standardization
- Original Message - > From: "Joerg Schilling" > To: k...@munnari.oz.au, "chet ramey" > Cc: austin-group-l@opengroup.org > Sent: Friday, September 14, 2018 1:05:17 PM > Subject: Re: Status of pipefail for standardization > > There seems to be a problem with the people who currently work on the ksh93 > source in the public git: > > It seems that these people are not talking with David Korn. I tried to reach dgk several times in last couple of years, but never managed to get a response from him. It would be great if he becomes active in the discussions again, but it's unlikely to happen. > > These people did already introduce changes that cannot be seen as an > improvement as they e.g. removed code and thus features that they simply > don't > understand. It was done to bring this code to more maintainable state. The way ksh is written makes it faster than other shells, but at the same time there are strange bugs hidden in it. It's a maintainer's nightmare. For example look at this bug report[1]. > > If you compile this code from git, you get a binary that is 30% slower than > the > binary you get when you compile the version that has been integrated into > OpenSolaris. In 2008, ksh93 was the fastest known shell and nearly twice as > fast as bash, the git version now is slower than "bosh" that used to be the > second fastest shell. For us keeping ksh relevant is more important than being the fastest shell. This requires simplying build process, improving code quality so that mere mortals can contribute to it, improving documentation etc. There is no point in being the fastest shell when your code is so horrible that no new contributor wants to touch it. In any case, if people disagree with our ideas, they can continue to use last stable version that came out from AT (ksh9u+ i.e.), or maintain their own forks. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1451057 > > Jörg > > -- > EMail:jo...@schily.net(home) Jörg Schilling D-13353 > Berlin > joerg.schill...@fokus.fraunhofer.de (work) Blog: > http://schily.blogspot.com/ > URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' > -- -- Siteshwar Vashisht
Re: Status of pipefail for standardization
- Original Message - > From: "Chet Ramey" > To: "Robert Elz" > Cc: "chet ramey" , austin-group-l@opengroup.org > Sent: Tuesday, September 11, 2018 5:43:07 PM > Subject: Re: Status of pipefail for standardization > > On 9/11/18 10:56 AM, Robert Elz wrote: > > Date:Tue, 11 Sep 2018 09:44:01 -0400 > > From:Chet Ramey > > Message-ID: > > > > | There is prior art here. The group has simply chosen not to standardize > > | `type -t'. > > > > Oh, this is too good - you'd almost think it was a setup (it wasn't...) > > There's nothing false or even controversial in that statement. > > > OK, so we have the problem, and the solution - which as best I can > > tell is implemented only by bash right? So not standardising it would be > > the correct (non-)action. > > Bash and ksh93 (at least the latest version I have from git). > > $ ./src/cmd/ksh93/ksh -c 'type -t printf' > builtin This flag was added with last beta release of ksh (ksh93v-) that came out from AT This is related entry from changelog[1] : 14-07-11 +Added -t flag and -P flag to whence and type for bash compaiblity. Since ksh93u+ was the last release that was embraced by distributions, I would suggest to use that as a reference. Our current work on github repo is based on ksh93v-. Whether it was a good decision or bad is a discussion for another day. > > > However, for people like me, who have a need for the functionality, and > > desire to implement something, having to hunt around through every > > other shell, trying random stuff, or hoping to find the right place in > > their > > doc (where "not found" might just mean "didn't look in the right place") > > it would be really good to have a place where things like this can be > > located, so there can be more implementations, and eventual widespread > > use and standardisation. > > [expressive shrug] > > That's a different issue. I'm simply pointing out that the specific issue > you raised has already been implemented, and, since one shell's > implementation of something is enough (cf. the pipefail discussion), we > could just go ahead. > > > I think the NetBSD sh is about to become the second shell to add type -t > > (and perhaps type -a, at least in the type -at form) but probably not (all) > > the > > other options bash supports. > > The third. > > > > > Pity that (even then) I won't be able to rely upon it, as almost no-one > > else > > has copied it (I'd guess, at least half because of lack of knowledge.) > > I wouldn't be so quick to conclude that. It's been around for 25 years, > after all (and before that, it was `type -type'). > > > I was using just "type name" and trying to find "builtin" in the various > > forms > > I have seen used, which worked for me, only to be told that > > ist eine von der Shell mitgelieferte Funktion > > wasn't recognised by my script as meaning builtin. I wonder why? > > I know that forcing the locale solves that issue, but the script is still > > guessing, as there's no defined output format. > > We could choose one as part of standardizing `type -t'. It's a set of > single words, after all. > > > > > kre > > > > ps: I think we need to slightly adjust the definition of "type -t" (or > > perhaps > > add a new option) as ... > > > > jinx$ type -at foo printf echo bar > > function > > builtin > > file > > builtin > > file > > > > jinx$ type -at foo printf echo bar > > function > > builtin > > file > > builtin > > file > > > > Those look as if they're identical, but they're not. Both exited with > > an error status (there is no "bar") but in the first it is "foo" that is > > the function, in the second it is "printf" (and there is, also, no "foo") > > Without other guidance, I think I'd change the output format when > > there is more than one name arg to include the relevant name, otherwise > > allowing multiple names is useless (so another option would be to > > only permit one name with -a - or at least with -at). > > I'll take a look. > > > It might also be useful to distinguish the special builtins from the > > others... > > Special builtins are only relevant in posix mode, so even if they were > distinguished somehow, you wouldn't get that in bash's default mode. > > > jinx$ type -t break echo > > builtin > > builtin > > > > makes them look just the same.Are changes to bash in this area > > possible? > > I suppose posix mode could display `spec-builtin', but that doesn't seem > compelling enough to break backwards compatibility. > > Chet > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer >``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ > > [1] https://github.com/att/ast/blob/e770c77e9816e156c6df4a455e71b5f9fff79310/src/cmd/ksh93/RELEASE#L48 -- -- Siteshwar Vashisht
Re: Bug 1133 find -xdev (was: Minutes of the 13th September 2018 Teleconference)
Joerg Schilling wrote, on 14 Sep 2018: > > From the Solaris man page: > > -mount Always true. Restricts the search to the file system > containing the directory specified. Does not list > mount points to other file systems. > >-xdevSame as the -mount primary. > > FTW_MOUNTIf set, nftw() reports only files in the same file system > as path. If clear, nftw() reports all files encountered > during the walk. [...] > > Consistency between find and pax should be a consideration, so it would > > be good to find out what the various pax implementations do here. > > I tried Solaris, HP-UX, macOS and Debian stretch (the "pax" package) > > and they all wrote the mount point to the archive. > > I believe we rather need a new flag for nftw() that includes a mount point > and > a new primary for find(1) that includes the mount point. I believe that the > current POSIX text is a bug since I expect that it's intend was to be > compatible to the SVr4 find(1) behavior that is aligned with and based on > nftw(). Perhaps an acceptable solution would be: Add -mount to find with behaviour matching nftw() FTW_MOUNT. Add FTW_XDEV to nftw() with behaviour matching find -xdev and pax -X. Make it clear that find -xdev acts on the mount points and -mount does not. (No need to add anything to pax since you can use find ... -mount | pax if you want the mount points excluded.) -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: Bug 1133 find -xdev (was: Minutes of the 13th September 2018 Teleconference)
Geoff Clare wrote: > Bruce Evans wrote, on 14 Sep 2018: > > The historic System V way is better. This is the way that is aligned with the SVSV docmentation and with the nftw() documentation and nftw() POSIX standard requirements: >From the Solaris man page: -mount Always true. Restricts the search to the file system containing the directory specified. Does not list mount points to other file systems. -xdevSame as the -mount primary. FTW_MOUNTIf set, nftw() reports only files in the same file system as path. If clear, nftw() reports all files encountered during the walk. > > Putting mount points in the list is not even suitable for backups, since > > the file corresponding to a mount point has attributes from a different > > file system. There is a difference on whether you like to do a backup (in this case, you mount the filesystem as a snapshot and see the real attributes) or if you do a typical tar BTW: star was the first tar implementation that introduced a -M option in 1985 that includes the mount point but does not descend to the content. It is however more complex since it is possible to mount files on file (see e.g. the file /etc/mnttab that typcally is mounted from a VFS. So this does not apply to directories only. > I suppose this is a matter of personal preference. To me it is more > important that restoring from backup creates the mount points so that > the mounts don't fail when the system is started after the restore. > The ownership and permissions on the directory "under" the mount point > don't matter most of the time. It's existence very much does matter. This is why star in 1985 decided to include the mount point. > > > Tree-walking functions have the same problem. For nftw(), FTW_MOUNT > > is clearly specified as "shall only report files [not pathnames] in > > the same file system as [the] path". However, for ftw.h, FTW_MOUNT > > is unclearly specified as "The walk does not _cross_ a mount point". As said, nftw() is aligned with the SVR4 find(1) behavior. > Generally the descriptions of flags on header pages are a short > indication of roughly what the flag does and the description on the > function page gives the precise details. So this should be considered > a minor editorial problem on the ftw.h page. We should probably > change it to say "ignores" instead of "does not cross". > > > nftw() should also have more options to keep or not keep mount points. > > Again, if there was a genuine need for this, one or more implementations > would have added it long ago. I did not realize that there is a problem yet, I now see it as a big problem since "star -cM ..." will include the mount point but "star -c -find -xdev ..." will not include it. > > Not many POSIX utilties have special options for mount points. In the > > old 2001 version of POSIX, the only ones that do seem to be find and pax. > > pax -X is unclearly specified as "... shall not descend into directories > > that have a different ... st_dev". I think this intends to specify the > > same behaviour as nftw() with FTW_MOUNT (with excessive implementation > > details for st_dev), but actually specifies the same behaviour as for find. > > Consistency between find and pax should be a consideration, so it would > be good to find out what the various pax implementations do here. > I tried Solaris, HP-UX, macOS and Debian stretch (the "pax" package) > and they all wrote the mount point to the archive. I believe we rather need a new flag for nftw() that includes a mount point and a new primary for find(1) that includes the mount point. I believe that the current POSIX text is a bug since I expect that it's intend was to be compatible to the SVr4 find(1) behavior that is aligned with and based on nftw(). Jörg -- EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'
Re: Status of pipefail for standardization
Robert Elz wrote: > Date:Tue, 11 Sep 2018 11:43:07 -0400 > From:Chet Ramey > Message-ID: > > > | Bash and ksh93 (at least the latest version I have from git). > | > | $ ./src/cmd/ksh93/ksh -c 'type -t printf' > | builtin > > The version of ksh93 I have doesn't ... There seems to be a problem with the people who currently work on the ksh93 source in the public git: It seems that these people are not talking with David Korn. These people did already introduce changes that cannot be seen as an improvement as they e.g. removed code and thus features that they simply don't understand. If you compile this code from git, you get a binary that is 30% slower than the binary you get when you compile the version that has been integrated into OpenSolaris. In 2008, ksh93 was the fastest known shell and nearly twice as fast as bash, the git version now is slower than "bosh" that used to be the second fastest shell. Jörg -- EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'
Bug 1133 find -xdev (was: Minutes of the 13th September 2018 Teleconference)
Bruce Evans wrote, on 14 Sep 2018: > > On Thu, 13 Sep 2018, Andrew Josey wrote: > > >Bug 0001133: find clarification on -xdev behavior for mounted filesystem > >within primary OPEN > >http://austingroupbugs.net/view.php?id=1133 > > > >We discussed this item. We believe that the current wording in the standard > >requires mount points to be included in the output from find > >some_directory -xdev but to not list files found by traversing > >those mount points. This allows one to make a backup of a directory > >containing mount points and have those mount points included in the > >backup without including any files found under that mount point. > > > >Unfortunately, this is not the way System V find behaved and there > >is not discussion in the rationale explaining why this change to > >historic behavior was made. We would guess that the documented > >behavior of the System V find matches what is in the standard even > >though that was not the way System V find behaved. We believe that > >this was a bug in the System V find and not an incorrect requirement > >in the standard. > > The historic System V way is better. > > Putting mount points in the list is not even suitable for backups, since > the file corresponding to a mount point has attributes from a different > file system. I suppose this is a matter of personal preference. To me it is more important that restoring from backup creates the mount points so that the mounts don't fail when the system is started after the restore. The ownership and permissions on the directory "under" the mount point don't matter most of the time. It's existence very much does matter. > [...] So there should > be options to keep or not keep the mount points as pathnames. If there was a genuine need for this, one or more find implementations would have added it long ago. > Tree-walking functions have the same problem. For nftw(), FTW_MOUNT > is clearly specified as "shall only report files [not pathnames] in > the same file system as [the] path". However, for ftw.h, FTW_MOUNT > is unclearly specified as "The walk does not _cross_ a mount point". Generally the descriptions of flags on header pages are a short indication of roughly what the flag does and the description on the function page gives the precise details. So this should be considered a minor editorial problem on the ftw.h page. We should probably change it to say "ignores" instead of "does not cross". > nftw() should also have more options to keep or not keep mount points. Again, if there was a genuine need for this, one or more implementations would have added it long ago. > Not many POSIX utilties have special options for mount points. In the > old 2001 version of POSIX, the only ones that do seem to be find and pax. > pax -X is unclearly specified as "... shall not descend into directories > that have a different ... st_dev". I think this intends to specify the > same behaviour as nftw() with FTW_MOUNT (with excessive implementation > details for st_dev), but actually specifies the same behaviour as for find. Consistency between find and pax should be a consideration, so it would be good to find out what the various pax implementations do here. I tried Solaris, HP-UX, macOS and Debian stretch (the "pax" package) and they all wrote the mount point to the archive. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
