Re: Status of pipefail for standardization

2018-12-16 Thread Simon Ser
Hi all,

On Sunday, December 16, 2018 11:01 PM, Robert Elz  wrote:
> | The chance of this happening in the future, when it has not happened
> | so far, is vanishingly small.
>
> I agree that it would be very rare, which is exactly why now, it doesn't
> really matter which way it gets defined. But I'm sue you've run into
> users who just love to push the envelope and do the wierdest things,
> which no-one had ever attempted before - having good, clear, definitions
> helps avoid problems in those cases.
>
> | We'll see how it goes. It took a #define and three lines of code.
>
> Great, thanks, and yes, that's just about the same as it took me .. define
> a new flag, set it appropriately (from pipefail) when creating the job, and
> then use that rather than the pipefail opt value when collecting the status.
>
> Boringly simple, and, as above, really unlikely to bother any current users,
> and makes for a much more predictable outcome when someone does decide
> to do something bizarre.
>
> kre

I'd like to bump this thread: are there any news about this?

I think leaving this unspecified is reasonable, at least as a first step. If
not, has anybody a list of shells we need to update to comply with the new
desired behaviour? Maybe someone already talked about this on the bash mailing
list?

Thanks,

--
Simon Ser
https://emersion.fr



Re: Status of pipefail for standardization

2018-09-17 Thread Joerg Schilling
Siteshwar Vashisht  wrote:

> > Last time I got him (I did no try to contact him later) he answered my
> > questions, but it may be that he will not actively help in the development.
>
> This discussion is diverting from original topic of pipefail standardization,
> but still I would try to clarify few things.

If we have a nearly final conclusion on how to deal with this, I could try to 
contact him again.

> > This is the wrong method. The right method is to read the source again and
> > again until you understand it. This is what I did with the Bourne Shell and
> > this helps me to understand the ksh93 source. Note that in his life before
> > AT, he designed supersonic wings. So he is used to think different than
> > most
> > people.
>
> You have a fair point about reading sources.

Well, I know that my "bsh" from 1984 is easier to maintain than the Bourne 
Shell because it is written in a simpler way. Now that I fully understand the 
Bourne Shell, I prefer it ove my old shell.


> The biggest change we have made is, we made it easier to build. This has only
> helped our cause because now people are able to build ksh easily on platforms
> like FreeBSD, OpenBSD etc. It was not impossible to build it earlier on these
> platforms but the build process was so arcane that half of the potential
> contributors will give up trying to build it.

In 2006, when I first personally met David, I had a discussion about his 
buildsystem that really is from Glenn Fowler (who BTW answers mail typically 
with less than a week delay). It turns out, that the ast build system is 
implemented in a completely different way than the Schily Makefilesystem, but 
it has the same basic concept: It uses leaf makefiles that look very similar 
because these makefiles just conatain a list of source files and compiler flags.

The knowledge on how to basically build commands is in the makefile system.
The main difference is that ast is based on knowledge that is coded in the 
"nmake" program while the schily makefilesystem codes the same knowledge in the
"make" language as part of object specific make rules.

Both methods grant portability to more platforms than the GNU method.

I believe that if we like to cntinue this discussion, it should be done in 
private as the build system is not related to POSIX.

> > If ksh93 is not maintained, it will loose importance, but this should not
> > result in quick changes. A good concept would e.g. be to introduce mandatory
> > code reviews for changes before they could make it into the pulic git.
> > 
> > I would like to see a restart from the v- state and have code reviews for
>
> You are right about the fact we have less experience with ksh source code and
> we have introduced bugs while trying to improve it's code quality. But the
> right method is not to start from scratch with `ksh93v-`. Right method is to
> improve our test coverage and find defects before they get shipped into a

Did you run the hand crafted unit tests that are included in the ksh93 source?

What I believe is missing for ksh93 is stochastic testing with unexpected 
random 
input. I friend helps me to do this with the Bourne Shell and we discovered and 
fixed hundreds of bugs this way, some of them have been in the source since the 
beginning in 1976.

> release. And before you say it, I am not promising the next release will 100%
> bug free or will not have regression. All legacy softwares that I know, 
> specially
> shells, have done a historically bad job at testing. I see this as an
> opportunity to change that. It's amazing to hear that dgk has a background in
> aviation, but aviation industry is known for it's insane focus on quality and
> testing. I see that focus missing in ksh. Why should a shell crash when 
> compiler
> optimization level is changed ?

During my nearly 40 years of programming experiences, I did observe more than a 
dozen cases where the reason for a problem was in the optimizer of the compiler.
If I did see the piece of code that causes the problem, I could give a comment.

> I agree that we need more code reviews from someone like you who has more
> experience with ksh source code. I could not find you on GitHub, but can 
> request
> for reviews over e-mail. If you have seen any questionable changes in current
> upstream, please open (reopen) related issue or send us an e-mail. If your
> counterargument is that all changes are questionable then I have got nothing
> to say to you.

I did not yet create an account on github, should I create one?

> We would like to keep ksh as close as possible to POSIX too. We surely agree 
> on
> this.

Fine

> We are in no hurry to make a release, even if it takes next 4 years of our 
> life.
> And it may not be even perfect. Whether it will be worth using or not, should 
> be
> left to be judged by ksh community. That said, it's more appropriate to have
> these discussions on ast-developers mailing list or GitHub.

This seems to be a good base. Being in a hurry 

Re: Status of pipefail for standardization

2018-09-15 Thread Robert Elz
Date:Sat, 15 Sep 2018 19:54:20 -0400
From:Chet Ramey 
Message-ID:  <6236131b-b65d-7d08-c48a-fbca06fc2...@case.edu>

  | The chance of this happening in the future, when it has not happened
  | so far, is vanishingly small.

I agree that it would be very rare, which is exactly why now, it doesn't
really matter which way it gets defined.   But I'm sue you've run into
users who just love to push the envelope and do the wierdest things,
which no-one had ever attempted before - having good, clear, definitions
helps avoid problems in those cases.

  | We'll see how it goes. It took a #define and three lines of code.

Great, thanks, and yes, that's just about the same as it took me .. define
a new flag, set it appropriately (from pipefail) when creating the job, and
then use that rather than the pipefail opt value when collecting the status.

Boringly simple, and, as above, really unlikely to bother any current users,
and makes for a much more predictable outcome when someone does decide
to do something bizarre.

kre



Re: Status of pipefail for standardization

2018-09-15 Thread Chet Ramey
On 9/11/18 4:46 PM, Robert Elz wrote:

> If it is unspecified, then a script cannot start an async pipeline,
> then want to run another while the first is executing, with a
> different pipefail setting (different to that for the first) as as soon as
> the pipefail option is changed while the async pipelline is still
> running we get unspecified results from its status.That is a poor
> outcome, and really, needless.  If the second pipeline were also
> async, then we end up with a real mess...

The chance of this happening in the future, when it has not happened
so far, is vanishingly small.

> So, why not try changing it (without announcing any change) in
> the alpha (or is it beta yet?) version of bash-5 - all of us here
> will promise not to tell the "real" users, or complain about it, and
> just see if there is anyone who notices.   If you get complaints,
> you can always change it back, if all your users are happy (ie:
> they never even notice) then we will all be happy too!

We'll see how it goes. It took a #define and three lines of code.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Status of pipefail for standardization

2018-09-15 Thread Siteshwar Vashisht



- Original Message -
> From: "Joerg Schilling" 
> To: svashi...@redhat.com
> Cc: k...@munnari.oz.au, "chet ramey" , 
> austin-group-l@opengroup.org
> Sent: Saturday, September 15, 2018 1:23:42 PM
> Subject: Re: Status of pipefail for standardization
> 
> Siteshwar Vashisht  wrote:
> 
> > > It seems that these people are not talking with David Korn.
> >
> > I tried to reach dgk several times in last couple of years, but never
> > managed to get a response from him. It would be great if he becomes active
> > in
> > the discussions again, but it's unlikely to happen.
> 
> I know it is hard to reach him but note that he even left Google because of
> the
> problems he still has.
> 
> Last time I got him (I did no try to contact him later) he answered my
> questions, but it may be that he will not actively help in the development.

This discussion is diverting from original topic of pipefail standardization,
but still I would try to clarify few things.

> 
> > > These people did already introduce changes that cannot be seen as an
> > > improvement as they e.g. removed code and thus features that they simply
> > > don't
> > > understand.
> >
> > It was done to bring this code to more maintainable state. The way ksh is
> > written makes it faster than other shells, but at the same time there are
> > strange bugs hidden in it. It's a maintainer's nightmare. For example look
> > at
> > this bug report[1].
> 
> This is the wrong method. The right method is to read the source again and
> again until you understand it. This is what I did with the Bourne Shell and
> this helps me to understand the ksh93 source. Note that in his life before
> AT, he designed supersonic wings. So he is used to think different than
> most
> people.

You have a fair point about reading sources.

> 
> You should not expect to be able to do in depth changes with less than 4
> years
> of experience in understanding this source.

The biggest change we have made is, we made it easier to build. This has only
helped our cause because now people are able to build ksh easily on platforms
like FreeBSD, OpenBSD etc. It was not impossible to build it earlier on these
platforms but the build process was so arcane that half of the potential
contributors will give up trying to build it.

> 
> 
> > For us keeping ksh relevant is more important than being the fastest shell.
> > This requires simplying build process, improving code quality so that mere
> > mortals can contribute to it, improving documentation etc. There is no
> > point
> > in being the fastest shell when your code is so horrible that no new
> > contributor wants to touch it.
> 
> The korn shell source is not horrible, it just uses concepts from a person
> that
> is much better than the average.
> 
> You cannot expect that people with too few programming experinces are able to
> change it in a way that could be seen as an improvement.
> 
> I created the first nearly portable Bourne Shell fork in 2006 and that added
> myhistory editor from 1984 to it at that time. I started again with limited
> effort in 2008 but I was not able to make in depth changes such as the
> conversion to the malloc() based "stak" code code from Geoff Collyer before
> 2012. I am really firm with the code since I changed it to use vfork() in
> 2014.
> That that time, I also changed the parser and interpreter to use a pipelne
> concept where all commands in the pipeline are children of the main shell.
> 
> > In any case, if people disagree with our ideas, they can continue to use
> > last
> > stable version that came out from AT (ksh9u+ i.e.), or maintain their own
> > forks.
> 
> If ksh93 is not maintained, it will loose importance, but this should not
> result in quick changes. A good concept would e.g. be to introduce mandatory
> code reviews for changes before they could make it into the pulic git.
> 
> I would like to see a restart from the v- state and have code reviews for

You are right about the fact we have less experience with ksh source code and
we have introduced bugs while trying to improve it's code quality. But the
right method is not to start from scratch with `ksh93v-`. Right method is to
improve our test coverage and find defects before they get shipped into a
release. And before you say it, I am not promising the next release will 100%
bug free or will not have regression. All legacy softwares that I know, 
specially
shells, have done a historically bad job at testing. I see this as an
opportunity to change that. It's amazing to hear that dgk has a background in
aviation, but aviation industry is known for it's insane focus on qua

Re: Status of pipefail for standardization

2018-09-15 Thread Joerg Schilling
Siteshwar Vashisht  wrote:

> > It seems that these people are not talking with David Korn.
>
> I tried to reach dgk several times in last couple of years, but never
> managed to get a response from him. It would be great if he becomes active in
> the discussions again, but it's unlikely to happen.

I know it is hard to reach him but note that he even left Google because of the 
problems he still has.

Last time I got him (I did no try to contact him later) he answered my 
questions, but it may be that he will not actively help in the development.

> > These people did already introduce changes that cannot be seen as an
> > improvement as they e.g. removed code and thus features that they simply
> > don't
> > understand.
>
> It was done to bring this code to more maintainable state. The way ksh is
> written makes it faster than other shells, but at the same time there are
> strange bugs hidden in it. It's a maintainer's nightmare. For example look at
> this bug report[1].

This is the wrong method. The right method is to read the source again and 
again until you understand it. This is what I did with the Bourne Shell and 
this helps me to understand the ksh93 source. Note that in his life before 
AT, he designed supersonic wings. So he is used to think different than most 
people.

You should not expect to be able to do in depth changes with less than 4 years 
of experience in understanding this source.


> For us keeping ksh relevant is more important than being the fastest shell.
> This requires simplying build process, improving code quality so that mere
> mortals can contribute to it, improving documentation etc. There is no point
> in being the fastest shell when your code is so horrible that no new
> contributor wants to touch it.

The korn shell source is not horrible, it just uses concepts from a person that 
is much better than the average. 

You cannot expect that people with too few programming experinces are able to 
change it in a way that could be seen as an improvement.

I created the first nearly portable Bourne Shell fork in 2006 and that added
myhistory editor from 1984 to it at that time. I started again with limited 
effort in 2008 but I was not able to make in depth changes such as the 
conversion to the malloc() based "stak" code code from Geoff Collyer before 
2012. I am really firm with the code since I changed it to use vfork() in 2014.
That that time, I also changed the parser and interpreter to use a pipelne 
concept where all commands in the pipeline are children of the main shell.

> In any case, if people disagree with our ideas, they can continue to use last
> stable version that came out from AT (ksh9u+ i.e.), or maintain their own
> forks.

If ksh93 is not maintained, it will loose importance, but this should not 
result in quick changes. A good concept would e.g. be to introduce mandatory 
code reviews for changes before they could make it into the pulic git.

I would like to see a restart from the v- state and have code reviews for 
changes to that state and since I understand the ks93 source, I am available 
for qualified code reviews.

BTW: We currently have the problem that ksh93 has many deviations from the 
POSIX standard and if we like ksh93 to be used as a master implementation, we 
need to keep the quality of the source.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'



Re: Status of pipefail for standardization

2018-09-15 Thread Joerg Schilling
Siteshwar Vashisht  wrote:

> > Bash and ksh93 (at least the latest version I have from git).
> > 
> > $ ./src/cmd/ksh93/ksh -c 'type -t printf'
> > builtin
>
> This flag was added with last beta release of ksh (ksh93v-) that came out from
> AT This is related entry from changelog[1] :
>
> 14-07-11 +Added -t flag and -P flag to whence and type for bash compaiblity.
>
> Since ksh93u+ was the last release that was embraced by distributions, I would
> suggest to use that as a reference. Our current work on github repo is based
> on ksh93v-. Whether it was a good decision or bad is a discussion for another
> day.

Thank you for the hint. This release still seems to be initiated by David Korn 
as IIRC David left AT later. I guess this is the state that staring from late 
2014 was no longer retrievable from AT as their servers suffer from a 
missconfiguration and die after less than a day after a restart.

The v- release fixes a lot of bugs and some of them even cause core dumps. 
Despite the fact that it is slow when you compile it using the supplied "nmake" 
based environment, it could be seen as a good starting point.

BTW: -a also works in this release with "type" that still is an alias to 
"whence v".

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'



Re: Status of pipefail for standardization

2018-09-14 Thread Siteshwar Vashisht
- Original Message -
> From: "Joerg Schilling" 
> To: k...@munnari.oz.au, "chet ramey" 
> Cc: austin-group-l@opengroup.org
> Sent: Friday, September 14, 2018 1:05:17 PM
> Subject: Re: Status of pipefail for standardization
> 
> There seems to be a problem with the people who currently work on the ksh93
> source in the public git:
> 
> It seems that these people are not talking with David Korn.

I tried to reach dgk several times in last couple of years, but never
managed to get a response from him. It would be great if he becomes active in
the discussions again, but it's unlikely to happen.

> 
> These people did already introduce changes that cannot be seen as an
> improvement as they e.g. removed code and thus features that they simply
> don't
> understand.

It was done to bring this code to more maintainable state. The way ksh is
written makes it faster than other shells, but at the same time there are
strange bugs hidden in it. It's a maintainer's nightmare. For example look at
this bug report[1].

> 
> If you compile this code from git, you get a binary that is 30% slower than
> the
> binary you get when you compile the version that has been integrated into
> OpenSolaris. In 2008, ksh93 was the fastest known shell and nearly twice as
> fast as bash, the git version now is slower than "bosh" that used to be the
> second fastest shell.

For us keeping ksh relevant is more important than being the fastest shell.
This requires simplying build process, improving code quality so that mere
mortals can contribute to it, improving documentation etc. There is no point
in being the fastest shell when your code is so horrible that no new
contributor wants to touch it.

In any case, if people disagree with our ideas, they can continue to use last
stable version that came out from AT (ksh9u+ i.e.), or maintain their own
forks.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1451057

> 
> Jörg
> 
> --
>  EMail:jo...@schily.net(home) Jörg Schilling D-13353
>  Berlin
> joerg.schill...@fokus.fraunhofer.de (work) Blog:
> http://schily.blogspot.com/
>  URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'
> 

-- 
--
Siteshwar Vashisht



Re: Status of pipefail for standardization

2018-09-14 Thread Siteshwar Vashisht



- Original Message -
> From: "Chet Ramey" 
> To: "Robert Elz" 
> Cc: "chet ramey" , austin-group-l@opengroup.org
> Sent: Tuesday, September 11, 2018 5:43:07 PM
> Subject: Re: Status of pipefail for standardization
> 
> On 9/11/18 10:56 AM, Robert Elz wrote:
> > Date:Tue, 11 Sep 2018 09:44:01 -0400
> > From:Chet Ramey 
> > Message-ID:  
> > 
> >   | There is prior art here. The group has simply chosen not to standardize
> >   | `type -t'.
> > 
> > Oh, this is too good - you'd almost think it was a setup (it wasn't...)
> 
> There's nothing false or even controversial in that statement.
> 
> > OK, so we have the problem, and the solution - which as best I can
> > tell is implemented only by bash right?  So not standardising it would be
> > the correct (non-)action.
> 
> Bash and ksh93 (at least the latest version I have from git).
> 
> $ ./src/cmd/ksh93/ksh -c 'type -t printf'
> builtin

This flag was added with last beta release of ksh (ksh93v-) that came out from
AT This is related entry from changelog[1] :

14-07-11 +Added -t flag and -P flag to whence and type for bash compaiblity.

Since ksh93u+ was the last release that was embraced by distributions, I would
suggest to use that as a reference. Our current work on github repo is based
on ksh93v-. Whether it was a good decision or bad is a discussion for another
day.

> 
> > However, for people like me, who have a need for the functionality, and
> > desire to implement something, having to hunt around through every
> > other shell, trying random stuff, or hoping to find the right place in
> > their
> > doc (where "not found" might just mean "didn't look in the right place")
> > it would be really good to have a place where things like this can be
> > located, so there can be more implementations, and eventual widespread
> > use and standardisation.
> 
> [expressive shrug]
> 
> That's a different issue. I'm simply pointing out that the specific issue
> you raised has already been implemented, and, since one shell's
> implementation of something is enough (cf. the pipefail discussion), we
> could just go ahead.
> 
> > I think the NetBSD sh is about to become the second shell to add type -t
> > (and perhaps type -a, at least in the type -at form) but probably not (all)
> > the
> > other options bash supports.
> 
> The third.
> 
> > 
> > Pity that (even then) I won't be able to rely upon it, as almost no-one
> > else
> > has copied it (I'd guess, at least half because of lack of knowledge.)
> 
> I wouldn't be so quick to conclude that. It's been around for 25 years,
> after all (and before that, it was `type -type').
> 
> > I was using just "type name" and trying to find "builtin" in the various
> > forms
> > I have seen used, which worked for me, only to be told that
> > ist eine von der Shell mitgelieferte Funktion
> > wasn't recognised by my script as meaning builtin.   I wonder why?
> > I know that forcing the locale solves that issue, but the script is still
> > guessing, as there's no defined output format.
> 
> We could choose one as part of standardizing `type -t'. It's a set of
> single words, after all.
> 
> > 
> > kre
> > 
> > ps: I think we need to slightly adjust the definition of "type -t" (or
> > perhaps
> > add a new option) as ...
> > 
> > jinx$ type -at foo printf echo bar
> > function
> > builtin
> > file
> > builtin
> > file
> > 
> > jinx$ type -at foo printf echo bar
> > function
> > builtin
> > file
> > builtin
> > file
> > 
> > Those look as if they're identical, but they're not.   Both exited with
> > an error status (there is no "bar") but in the first it is "foo" that is
> > the function, in the second it is "printf" (and there is, also, no "foo")
> > Without other guidance, I think I'd change the output format when
> > there is more than one name arg to include the relevant name, otherwise
> > allowing multiple names is useless (so another option would be to
> > only permit one name with -a - or at least with -at).
> 
> I'll take a look.
> 
> > It might also be useful to distinguish the special builtins from the
> > others...
> 
> Special builtins are only relevant in posix mode, so even if they were
> distinguished somehow, you wouldn't get that in bash's default mode.
> 
> > jinx$ type -t break echo
> > builtin
> > builtin
> > 
> > makes them look just the same.Are changes to bash in this area
> > possible?
> 
> I suppose posix mode could display `spec-builtin', but that doesn't seem
> compelling enough to break backwards compatibility.
> 
> Chet
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
> 
> 

[1] 
https://github.com/att/ast/blob/e770c77e9816e156c6df4a455e71b5f9fff79310/src/cmd/ksh93/RELEASE#L48
-- 
--
Siteshwar Vashisht



Re: Status of pipefail for standardization

2018-09-14 Thread Joerg Schilling
Robert Elz  wrote:

> Date:Tue, 11 Sep 2018 11:43:07 -0400
> From:Chet Ramey 
> Message-ID:  
>
>
>   | Bash and ksh93 (at least the latest version I have from git).
>   |
>   | $ ./src/cmd/ksh93/ksh -c 'type -t printf'
>   | builtin
>
> The version of ksh93 I have doesn't ...

There seems to be a problem with the people who currently work on the ksh93 
source in the public git:

It seems that these people are not talking with David Korn.

These people did already introduce changes that cannot be seen as an 
improvement as they e.g. removed code and thus features that they simply don't
understand.

If you compile this code from git, you get a binary that is 30% slower than the 
binary you get when you compile the version that has been integrated into 
OpenSolaris. In 2008, ksh93 was the fastest known shell and nearly twice as 
fast as bash, the git version now is slower than "bosh" that used to be the
second fastest shell.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'



Re: Status of pipefail for standardization

2018-09-11 Thread Robert Elz
Date:Tue, 11 Sep 2018 14:52:26 -0400
From:Chet Ramey 
Message-ID:  <58f1d638-c77c-279d-fdd1-81a104a10...@case.edu>

  | I'm ok with making it unspecified (and recommended it in note 4120).

Yes, though in this case the alternative is actually a more rational
result, which makes this less obvious ... to comment on your comment
in the (or a) note, the only users who would ever expect the bash behaviour
are those who have been explicitly told it is that way, there is nothing
in the manual that suggests it (the closest would be the sentence
"The shell waits for all commands in the pipeline to terminate before
returning a value." but that is actually talking about something different.

I'd almost guarantee that there are no scripts around that are replying upon 
the ability to set pipefail after starting a pipeline, and have it affect
the outcome (or the converse).   That is, you could change the behaviour
tomorrow, and almost no-one would notice.   And those who do are
people like me who go hunting for corner cases, and can easily adapt
(and are certainly not relying upon it for anything).

The reason I initially implemented the NetBSD sh that way (aside from
that we discussed it at the time, and you told me what the bash method
was) was because iit was (fractionally) easier, in that it avoids the need
to remember the pipefail setting with the pipe, just look at it when
collecting status.

When I decided tp change, it took literally 2 minutes to make the alteration
and half of that was the recompile, which touched quite a few files, as
a new #define was needed in a header file.Testing that it worked
took longer (but not all that much).

It is perhaps also pertinent that the "when the pipeline is started" method
is easily implementable for all three rational methods of implementing
pipes.   There's the (these days) common one, where the parent
shell creates each process in the pipeline as a child of itself.
There's the case where the parent forks, and the child then creates
the pipeline by forking as needed to create the rest of the pipe,
with all the commands in the pipe (except the last) being children
of the last, and third, where the parent creates a process (or does
it inside itself) for the rightmost process, then if there are preceding
commands remaining in the pipe sequence, forks to create a
process for a pipeline which is all but the last command in the
pipe sequence, all of which is then recursively applied (and so
each process in the pipe sequence is a child of the one to its right)
(And yes this method does require both a shell, and if the command
to be run is not builtin, another process to execute whatever is needed,
so is more costly, but is trivially easy to implement)

The alternative "when the status is collected" can only be implemented
in any rational way for the first of those - which means that if you
do it that way (and allow users to know it is done that way) you are
forever tied to implementing pipes that way, and cannot switch to
one of the others (not that you'd probably want to, realistically).


  | My principal objection, if it rises to that level, is that we're
  | doing things differently than usual in the face of conflicting
  | implementations.

Understood.   But there are times when it is easier to change an
implementation or two and have a tighter standard (one more useful
to the users) than leaving stuff unspecified when it doesn't really
need to be.

If it is unspecified, then a script cannot start an async pipeline,
then want to run another while the first is executing, with a
different pipefail setting (different to that for the first) as as soon as
the pipefail option is changed while the async pipelline is still
running we get unspecified results from its status.That is a poor
outcome, and really, needless.  If the second pipeline were also
async, then we end up with a real mess...

So, why not try changing it (without announcing any change) in
the alpha (or is it beta yet?) version of bash-5 - all of us here
will promise not to tell the "real" users, or complain about it, and
just see if there is anyone who notices.   If you get complaints,
you can always change it back, if all your users are happy (ie:
they never even notice) then we will all be happy too!


[Here onto type -t, rather than pipefail]

  | But if we're just standardizing -t, we can decide what to do with these,
  | or leave them unspecified.

Yes, the standard could have a cut down but still usefiul version of
the command (the option) and leave the rest (like having multiple word
args) as an unspecified extension that shells can implement if they
need to.  A script can always just run type -t xx; type -t yy; type -t zz
if it needs the output from 3 words, and if it wants to do that in one
command substitution, it can insert whatever syntax it likes between the
3 outputs (in that case) to allow it to parse the result.

Now I have to think about how to handle reserved 

Re: Status of pipefail for standardization

2018-09-11 Thread Chet Ramey
On 9/11/18 12:24 PM, Robert Elz wrote:
> Date:Tue, 11 Sep 2018 11:43:07 -0400
> From:Chet Ramey 
> Message-ID:  
> 
> 
>   | Bash and ksh93 (at least the latest version I have from git).
>   |
>   | $ ./src/cmd/ksh93/ksh -c 'type -t printf'
>   | builtin
> 
> The version of ksh93 I have doesn't ...
> 
> magnolia$ ksh93
> $ type -t printf
> ksh93: whence: -t: unknown option
> Usage: whence [-afpqv] name  ...
> 
> so it must be relatively recent (in ksh terms anyway).
> 
> (recent) ksh93 seems to pride itself in being absurdly difficult 
> to build, so I prefer not to do so too frequently (and that explains
> why I only have it on one system, which is not one I use much)

Yeah, I wouldn't have done it if Macports didn't have a port that
installed `mesos' and `ninja'.

> 
>   | That's a different issue.
> 
> Sure.
> 
>   | I'm simply pointing out that the specific issue
>   | you raised has already been implemented
> 
> So my example had a hidden benefit...
> 
>   | and, since one shell's implementation of something is
>   | enough (cf. the pipefail discussion), we could just go ahead.
> 
> pipefail has lots of implementations.   The only issue is the
> (relatively minor) issue of when the value of the option is
> examined.   That's so minor (overall) that it wasn't mentioned
> at all in the earlier proposal.   Even now, making it unspecified
> would not be horrible, ie: "Script writers, don't change pipefail
> while a pipeline is running or anything might happen" ...   

I'm ok with making it unspecified (and recommended it in note 4120).
My principal objection, if it rises to that level, is that we're
doing things differently than usual in the face of conflicting
implementations.

> 
>   | The third.
> 
> That's OK, it isn't a race - and someone else reading this might
> find the time to do it before I do,

I think that, in this case, three is better than two. It's not a race;
it's the total number of implementations that weights more heavily for
standardization, as you've pointed out.

And I appreciate your bringing up the corner cases.

> 
>   | I wouldn't be so quick to conclude that. It's been around for 25 years,
>   | after all (and before that, it was `type -type').
> 
> I have been a bash user all that time (that is, from before you took over
> maintaining it) and I never knew that existed.

OK. All that time, it was there, and people were using it. ;-)

But I get your point.

> 
>   | We could choose one as part of standardizing `type -t'. It's a set of
>   | single words, after all.
> 
> Of course, for use with -t we want well defined output - that is the point
> (and the words bash lists as possible look fine, at least with the possible
> special builtin caveat..)   I think I'd change the manual to make it clear
> that the order the words are listed there is the order in which the
> search is made, which I believe (from testing) is what bash implements,
> but it would be good to make it clear.

I think that it's the interaction with -a that needs to be cleaned up as
well, not just the multiple arguments. I don't know if it would be better
to prefix the output with the argument name in both cases (or only if there
are multiple instances in the -a case), or just warn against doing it in
the documentation. The same goes for adding an error if the command isn't
found, which -t explicitly inhibits.

But if we're just standardizing -t, we can decide what to do with these,
or leave them unspecified.

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Status of pipefail for standardization

2018-09-11 Thread Chet Ramey
On 9/11/18 10:56 AM, Robert Elz wrote:
> Date:Tue, 11 Sep 2018 09:44:01 -0400
> From:Chet Ramey 
> Message-ID:  
> 
>   | There is prior art here. The group has simply chosen not to standardize
>   | `type -t'.
> 
> Oh, this is too good - you'd almost think it was a setup (it wasn't...)

There's nothing false or even controversial in that statement.

> OK, so we have the problem, and the solution - which as best I can
> tell is implemented only by bash right?  So not standardising it would be
> the correct (non-)action.

Bash and ksh93 (at least the latest version I have from git).

$ ./src/cmd/ksh93/ksh -c 'type -t printf'
builtin

> However, for people like me, who have a need for the functionality, and
> desire to implement something, having to hunt around through every
> other shell, trying random stuff, or hoping to find the right place in their
> doc (where "not found" might just mean "didn't look in the right place")
> it would be really good to have a place where things like this can be
> located, so there can be more implementations, and eventual widespread
> use and standardisation.

[expressive shrug]

That's a different issue. I'm simply pointing out that the specific issue
you raised has already been implemented, and, since one shell's
implementation of something is enough (cf. the pipefail discussion), we
could just go ahead.

> I think the NetBSD sh is about to become the second shell to add type -t
> (and perhaps type -a, at least in the type -at form) but probably not (all) 
> the
> other options bash supports.

The third.

> 
> Pity that (even then) I won't be able to rely upon it, as almost no-one else
> has copied it (I'd guess, at least half because of lack of knowledge.)

I wouldn't be so quick to conclude that. It's been around for 25 years,
after all (and before that, it was `type -type').

> I was using just "type name" and trying to find "builtin" in the various forms
> I have seen used, which worked for me, only to be told that
>   ist eine von der Shell mitgelieferte Funktion
> wasn't recognised by my script as meaning builtin.   I wonder why?
> I know that forcing the locale solves that issue, but the script is still
> guessing, as there's no defined output format.

We could choose one as part of standardizing `type -t'. It's a set of
single words, after all.

> 
> kre
> 
> ps: I think we need to slightly adjust the definition of "type -t" (or perhaps
> add a new option) as ...
> 
> jinx$ type -at foo printf echo bar
> function
> builtin
> file
> builtin
> file
> 
> jinx$ type -at foo printf echo bar
> function
> builtin
> file
> builtin
> file
> 
> Those look as if they're identical, but they're not.   Both exited with
> an error status (there is no "bar") but in the first it is "foo" that is
> the function, in the second it is "printf" (and there is, also, no "foo")
> Without other guidance, I think I'd change the output format when
> there is more than one name arg to include the relevant name, otherwise
> allowing multiple names is useless (so another option would be to
> only permit one name with -a - or at least with -at).

I'll take a look.

> It might also be useful to distinguish the special builtins from the
> others...

Special builtins are only relevant in posix mode, so even if they were
distinguished somehow, you wouldn't get that in bash's default mode.

> jinx$ type -t break echo
> builtin
> builtin
> 
> makes them look just the same.Are changes to bash in this area possible?

I suppose posix mode could display `spec-builtin', but that doesn't seem
compelling enough to break backwards compatibility.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Status of pipefail for standardization

2018-09-11 Thread Robert Elz
Date:Tue, 11 Sep 2018 15:18:11 +0100
From:Geoff Clare 
Message-ID:  <20180911141811.GA30977@lt2.masqnet>

  | So it looks like we do have a way to do what you were hoping.

That's great - even if there is just that one issue currently (plus 3 aardvark
bugs) using that method (are the aardvark bugs still relevant?)

It would be nicer if there was an easier interface to use it however.

kre



Re: Status of pipefail for standardization

2018-09-11 Thread Robert Elz
Date:Tue, 11 Sep 2018 09:44:01 -0400
From:Chet Ramey 
Message-ID:  

  | There is prior art here. The group has simply chosen not to standardize
  | `type -t'.

Oh, this is too good - you'd almost think it was a setup (it wasn't...)

OK, so we have the problem, and the solution - which as best I can
tell is implemented only by bash right?  So not standardising it would be
the correct (non-)action.

However, for people like me, who have a need for the functionality, and
desire to implement something, having to hunt around through every
other shell, trying random stuff, or hoping to find the right place in their
doc (where "not found" might just mean "didn't look in the right place")
it would be really good to have a place where things like this can be
located, so there can be more implementations, and eventual widespread
use and standardisation.

I think the NetBSD sh is about to become the second shell to add type -t
(and perhaps type -a, at least in the type -at form) but probably not (all) the
other options bash supports.

Pity that (even then) I won't be able to rely upon it, as almost no-one else
has copied it (I'd guess, at least half because of lack of knowledge.)
I was using just "type name" and trying to find "builtin" in the various forms
I have seen used, which worked for me, only to be told that
ist eine von der Shell mitgelieferte Funktion
wasn't recognised by my script as meaning builtin.   I wonder why?
I know that forcing the locale solves that issue, but the script is still
guessing, as there's no defined output format.

kre

ps: I think we need to slightly adjust the definition of "type -t" (or perhaps
add a new option) as ...

jinx$ type -at foo printf echo bar
function
builtin
file
builtin
file

jinx$ type -at foo printf echo bar
function
builtin
file
builtin
file

Those look as if they're identical, but they're not.   Both exited with
an error status (there is no "bar") but in the first it is "foo" that is
the function, in the second it is "printf" (and there is, also, no "foo")
Without other guidance, I think I'd change the output format when
there is more than one name arg to include the relevant name, otherwise
allowing multiple names is useless (so another option would be to
only permit one name with -a - or at least with -at).

It might also be useful to distinguish the special builtins from the
others...

jinx$ type -t break echo
builtin
builtin

makes them look just the same.Are changes to bash in this area possible?




Re: Status of pipefail for standardization

2018-09-11 Thread Geoff Clare
Robert Elz  wrote, on 11 Sep 2018:
>
>   | Usually if a feature is proposed that isn't ready for standardisation
>   | we would close the Mantis bug with a note saying (but less bluntly) "try
>   | again when it's ready".  However, I suppose we might make an exception if
>   | there is good reason to do so.
> 
> I was kind of hoping it might be possible for them to have some special 
> status,
> not just "languishing" - not something to be used often, just when there is
> agreement that there is a problem to solve, and what a solution for that
> problem might be - but not yet sufficient implementation to standardise.

This jogged my memory of a similar case a few years ago - bug 786 - and what
we did there was set the resolution to "Future Enhancement" but leave the
status as Under Review.  There is also bugnote 2086 explaining why it has
been left that way.

So it looks like we do have a way to do what you were hoping.

-- 
Geoff Clare 
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England



Re: Status of pipefail for standardization

2018-09-11 Thread Chet Ramey
On 9/11/18 7:38 AM, Robert Elz wrote:

> Eg - and for this one I have no proposed solution for at the minute - as best 
> I can tell right now, there's no standard way to tell if a command is built 
> into the
> shell or not.   In fact the standard goes to some extreme with the intent of
> making it impossible to tell.   That's unfortunate for some uses, though 
> having
> the two cases function identically, as much as possible, is ideal.   It would
> be nice to design a method to provide this information.   That should not be
> hard.   (While it could be a modification of the "type" command, as it is now,
> that is useless for the purpose.)Clearly any such thing would not be
> implemented enough to standardise - but it could be written as if it were,
> and then implemented by most shells - with the expectation that if they don't
> they will become non-conforming if enough other shells do implement it.

There is prior art here. The group has simply chosen not to standardize
`type -t'.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Status of pipefail for standardization

2018-09-11 Thread Robert Elz
Date:Tue, 11 Sep 2018 11:02:24 +0100
From:Geoff Clare 
Message-ID:  <20180911100224.GA27521@lt2.masqnet>

  | Okay, I have added an alternative (bugnote 4115) based on this suggestion,
  | although it uses "pipe sequence status" as a locally-defined term rather
  | than using the symbol from the grammar.

Looks fine to me :-)

I will leave it to those of you who make the decisions which format to
prefer, they both look as if they cover all that needs to be covered to
me, and I can see no reason not to move forward with one of them.

  | Yes, it would be considered if it is existing practice.

OK, good, and caveat understood.

  | Probably the most important consideration would be whether any of the
  | candidates are already in use by some shell for another purpose.

Agreed - but as I said before, as best I can tell, no-one ever invents a
new special parameter.   I think the ones we have are all from Steve Bourne
(well, those that did not pre-exist from the Thompson shell).

  | Usually if a feature is proposed that isn't ready for standardisation
  | we would close the Mantis bug with a note saying (but less bluntly) "try
  | again when it's ready".  However, I suppose we might make an exception if
  | there is good reason to do so.

I was kind of hoping it might be possible for them to have some special status,
not just "languishing" - not something to be used often, just when there is
agreement that there is a problem to solve, and what a solution for that
problem might be - but not yet sufficient implementation to standardise.

Currently mantis seems to allow searching for "unresolved" and "approved" 
"pending"
and "proposed" interpretation.   Could (when there is something which needs it)
there be a new category, perhaps "Future Plans" or something, to enable such
issues to easily be located, so implementors could have a guide on what would
be worth working on (or demonstrating the folly of, if it turns out that way).

Eg - and for this one I have no proposed solution for at the minute - as best 
I can tell right now, there's no standard way to tell if a command is built 
into the
shell or not.   In fact the standard goes to some extreme with the intent of
making it impossible to tell.   That's unfortunate for some uses, though having
the two cases function identically, as much as possible, is ideal.   It would
be nice to design a method to provide this information.   That should not be
hard.   (While it could be a modification of the "type" command, as it is now,
that is useless for the purpose.)Clearly any such thing would not be
implemented enough to standardise - but it could be written as if it were,
and then implemented by most shells - with the expectation that if they don't
they will become non-conforming if enough other shells do implement it.

kre



Re: Status of pipefail for standardization

2018-09-11 Thread Geoff Clare
Robert Elz  wrote, on 11 Sep 2018:
>
>   | I looked at the text on netbsd.gw.com/cgi-bin/man-cgi?sh++NetBSD-current
>   | and I do not think it could easily be adapted for use in the standard.
>   | The main problem is that it considers the '!' not to be part of the
>   | "pipeline" that has a "pipeline status".
> 
> Well, perhaps The name could be changed to something other
> than "pipeline" status,   To borrow wording from the grammar (though
> it is a little unweildy) it could be "pipe_sequence status" instead of
> pipeline status.
> 
> The point was more that extracing the status in two steps (which I suspect
> is how it is always  implemented) allows each of those speps to be
> explained once, rather than the table, which requires each to be
> explained twice, in different combinations.

Okay, I have added an alternative (bugnote 4115) based on this suggestion,
although it uses "pipe sequence status" as a locally-defined term rather
than using the symbol from the grammar.

>   | > I am not sure what the attitude is here to creating new special 
> parameters,
> 
>   | We would need it to be implemented in at least one widely-used shell
>   | before we could consider standardising it.
> 
> Of course.  More than one would be better (even necessary I think.)
> 
> My question, before doing a "less than 'widely' used" implementation was
> as to whether adding a new special parameter is something that would
> ever be considered acceptable at all.

Yes, it would be considered if it is existing practice.

>   | Using the character '|' is problematic because it is an operator.
> 
> Yes, I can see that might be an issue - as I said I have not implemented
> this, as I have not seen the demand for it.
> 
> But I guess I considered that $| could be treated in a way similar to $(
> and treated a little specially by the tokeniser - but upon reflection while
> easy to do, that does look to be a little crude (just as is the "io_token").
> 
> The actual name of a new special parameter is the least relevant question
> here though, it could be $/ which has almost as much appeal as $| given its
> relationship to $? (same keyboard key...) and that / is just | written 
> sloppily ... and if we finally drop ^ as any kind of operator at all, $^ would
> keep that historic reference alive,  or $+ as a list of (more than one 
> usually)
> statuses that are "added" ,,,  there are lots of other less desirable
> possibilities ...  $% $, $. $= )

I like the suggestion of $/ (for both of the reasons you give).  The
relationship between ^ and | is long gone (it was not in POSIX.2-1992),
so I see no reason to prefer $^ over $%, etc.

Probably the most important consideration would be whether any of the
candidates are already in use by some shell for another purpose.

> ps: as a group operational question, is it, or would it be, reasonable to have
> an "issue" that is opened, discussed and revised as needed, but then simply
> kept unresolved (for perhaps quite a long time) for situations where something
> that is new seems desirable, but is not ready to be standardised ... just so 
> there is somewhere where implementors can find specifications for potential
> new features, so that when a need arises they can see how a solution has 
> already been proposed, rather than needing to go invent something new, and
> possibly different, from what others are doing ?

Usually if a feature is proposed that isn't ready for standardisation
we would close the Mantis bug with a note saying (but less bluntly) "try
again when it's ready".  However, I suppose we might make an exception if
there is good reason to do so.

-- 
Geoff Clare 
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England



Re: Status of pipefail for standardization

2018-09-10 Thread Robert Elz
Date:Mon, 10 Sep 2018 16:49:49 +0100
From:Geoff Clare 
Message-ID:  <20180910154949.GA26031@lt2.masqnet>

  | I looked at the text on netbsd.gw.com/cgi-bin/man-cgi?sh++NetBSD-current
  | and I do not think it could easily be adapted for use in the standard.

OK

  | The main problem is that it considers the '!' not to be part of the
  | "pipeline" that has a "pipeline status".

Well, perhaps The name could be changed to something other
than "pipeline" status,   To borrow wording from the grammar (though
it is a little unweildy) it could be "pipe_sequence status" instead of
pipeline status.

The point was more that extracing the status in two steps (which I suspect
is how it is always  implemented) allows each of those speps to be
explained once, rather than the table, which requires each to be
explained twice, in different combinations.

  | > I am not sure what the attitude is here to creating new special 
parameters,

  | We would need it to be implemented in at least one widely-used shell
  | before we could consider standardising it.

Of course.  More than one would be better (even necessary I think.)

My question, before doing a "less than 'widely' used" implementation was
as to whether adding a new special parameter is something that would
ever be considered acceptable at all.

And as an additional point, that arrays are not needed to expose the
status of the commands in a pipe_sequence, it can be made available
in an easily accessible manner that does not require addition of a
whole new  element syntax, plus all the minor changes everywhere
else that would be needed to cope with it.

  | Using the character '|' is problematic because it is an operator.

Yes, I can see that might be an issue - as I said I have not implemented
this, as I have not seen the demand for it.

But I guess I considered that $| could be treated in a way similar to $(
and treated a little specially by the tokeniser - but upon reflection while
easy to do, that does look to be a little crude (just as is the "io_token").

The actual name of a new special parameter is the least relevant question
here though, it could be $/ which has almost as much appeal as $| given its
relationship to $? (same keyboard key...) and that / is just | written 
sloppily ... and if we finally drop ^ as any kind of operator at all, $^ would
keep that historic reference alive,  or $+ as a list of (more than one usually)
statuses that are "added" ,,,  there are lots of other less desirable
possibilities ...  $% $, $. $= )

kre

ps: as a group operational question, is it, or would it be, reasonable to have
an "issue" that is opened, discussed and revised as needed, but then simply
kept unresolved (for perhaps quite a long time) for situations where something
that is new seems desirable, but is not ready to be standardised ... just so 
there is somewhere where implementors can find specifications for potential
new features, so that when a need arises they can see how a solution has 
already been proposed, rather than needing to go invent something new, and
possibly different, from what others are doing ?



Re: Status of pipefail for standardization

2018-09-10 Thread Geoff Clare
Robert Elz  wrote, on 07 Sep 2018:
>
> [...] in the NetBSD man page description of pipefail we handle the
> exit status issue (with/without pipefail, with/without !) a little 
> differently.
> Our text defines a pipestatus for a pipeline, which is the thing that
> pipefail controls, and then derives the exit status of a pipeline from
> the pipestatus of that pipeline based upon the presence, or not, of
> the ! word.   The effect is the same as the big table, but it is rather
> more concise.

I looked at the text on netbsd.gw.com/cgi-bin/man-cgi?sh++NetBSD-current
and I do not think it could easily be adapted for use in the standard.
The main problem is that it considers the '!' not to be part of the
"pipeline" that has a "pipeline status".  In POSIX the '!' is part of
the pipeline, so if we were to use the term "pipeline status" as it is
used in the NetBSD text, the term itself would be misleading.

[...]
> I am not sure what the attitude is here to creating new special parameters,
> the ones we have all seem to have originated in the original Bourne shell,
> and no-one has ever added anything new (I actually had $+ for a short
> while - never released anywhere - which was a comlpement to $- and
> contained the single letter options that were not set ... more for symmetry
> than to solve any real problem ... so it went away again).
> 
> But if anything cries out for a new special parameter it would be this - to
> make the exit status of the processes that form a pipeline available.

We would need it to be implemented in at least one widely-used shell
before we could consider standardising it.

> If I had to pick something, I'd use $| and define it to be the a space
> separated list of exit status values ($?) for the commands that form a
> pipeline.   It would be set (and have a lifetime) exactly the same as
> applies to $? - and in the case of a degenerate pipeline (a pipeline
> with no pipe operators, ie: a simple command), $| and $? would be
> identical (as they would be before any command had run.)

Using the character '|' is problematic because it is an operator.
The only way to access the parameter would be to put it in double
quotes.  So to access the status of, say, the second command in the
pipeline you would have to do:

statuses="$|"
set -- $statuses
status2=$2

instead of just:

set -- $|
status2=$2

(the latter would be equivalent to: set -- $ | status2=$2)

-- 
Geoff Clare 
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England



Re: Status of pipefail for standardization

2018-09-07 Thread Robert Elz
Date:Fri, 7 Sep 2018 09:35:39 +0100
From:Geoff Clare 
Message-ID:  <20180907083539.GA5774@lt2.masqnet>

  | Since kre reported in bugnote 4102 that he added pipefail to the
  | NetBSD sh and it does not support arrays, it seems we have our
  | answer on that one.

I am not sure that what we do necessarily provides proof for anything,
but yes, pipefail is useful without being able to see the status of the
individual processes (which is what the "support arrays" side issue
is really all about.).It might be even more useful with that extra,
but sometimes simply knowing "did the entire pipe work, or did
anything fail?" is all that is needed - and my current impression is
that in many cases, that is exactly what is needed.

Before going on to the "what if" part of this message, I would also say
that in the NetBSD man page description of pipefail we handle the
exit status issue (with/without pipefail, with/without !) a little differently.
Our text defines a pipestatus for a pipeline, which is the thing that
pipefail controls, and then derives the exit status of a pipeline from
the pipestatus of that pipeline based upon the presence, or not, of
the ! word.   The effect is the same as the big table, but it is rather
more concise.

For when more than success/fail is needed, I still don't believe that
arrays are needed, the status information for a list of commands is
just a list of integers, and can easily be handled as a list using
traditional shell strings.  (This is not a case where random unknown
data needs to be handled, a space seeparated ordered list of
numbers would be just fine.)

And while in note 3905 Stephane provided a link to a web page showing
how the status information can be extracted now (with no additional
mechanism at all - not even pipefail) I kind of suspect that's rather more
complex than most people would want, and too susceptible to minor
errors (accidentally leave a \ out from ahead of one of the | and it fails
in not so obvious ways).

So, what would be needed, not to have pipefail, but to allow the status
to be extracted (pipefail enabled or not) would be some new mechanism
to provide the info (of which an array, or just a variable, with a magic
name is certtainly a possibility.)

I am not sure what the attitude is here to creating new special parameters,
the ones we have all seem to have originated in the original Bourne shell,
and no-one has ever added anything new (I actually had $+ for a short
while - never released anywhere - which was a comlpement to $- and
contained the single letter options that were not set ... more for symmetry
than to solve any real problem ... so it went away again).

But if anything cries out for a new special parameter it would be this - to
make the exit status of the processes that form a pipeline available.

If I had to pick something, I'd use $| and define it to be the a space
separated list of exit status values ($?) for the commands that form a
pipeline.   It would be set (and have a lifetime) exactly the same as
applies to $? - and in the case of a degenerate pipeline (a pipeline
with no pipe operators, ie: a simple command), $| and $? would be
identical (as they would be before any command had run.)

I have not implemented that, as there has been no demand, so I
cannot state that it will work (it is obviously implementable, just as it is
obviously possible to implement the array method, for shells willing
to add all the mechanism necessary to support arrays as a general
concept)  - and so it is way too early to even consider standardising
anything in this area - this is just so I can get a general "sense of
the room" as to whether this would be an acceptable solution, or
simply considered out of the question.

kre



Re: Status of pipefail for standardization

2018-09-07 Thread Geoff Clare
Drew DeVault  wrote, on 05 Sep 2018:
>
> On 2018-09-05  8:18 AM, Drew DeVault wrote:
> > Copying David on the thread. David, do you have some time to look over
> > this proposal for the POSIX shell?
> 
> My email to David bounced and I don't have a better address for him.
> Does anyone else have a way of getting in touch?

As discussed in yesterday's call, it seems unlikely that we will get
any input from David.

So the options are to try and get input from someone else who works
on ksh93 (e.g. Glenn Fowler?) or just to proceed with what we have
already.

I think the main question we were hoping for input on is whether
it is worth standardising pipefail without support for arrays.
Since kre reported in bugnote 4102 that he added pipefail to the
NetBSD sh and it does not support arrays, it seems we have our
answer on that one.

-- 
Geoff Clare 
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England



Re: Status of pipefail for standardization

2018-09-05 Thread Drew DeVault
On 2018-09-05  8:18 AM, Drew DeVault wrote:
> Copying David on the thread. David, do you have some time to look over
> this proposal for the POSIX shell?

My email to David bounced and I don't have a better address for him.
Does anyone else have a way of getting in touch?



Re: Status of pipefail for standardization

2018-09-05 Thread Drew DeVault
> We had previously asked for input from Chet Ramey and David Korn and
> got a reply from Chet but not from David.

Copying David on the thread. David, do you have some time to look over
this proposal for the POSIX shell?

http://austingroupbugs.net/view.php?id=789

I also saw some more discussion on the bug yesterday afternoon, I'll
look that over and see where I can help. Thanks Geoff!



Re: Status of pipefail for standardization

2018-09-05 Thread Geoff Clare
Drew DeVault  wrote, on 04 Sep 2018:
>
> Hi there! The following ticket has been around for a while discussing
> the possibility of including set -o pipefail in the standard shell. I'm
> writing in to see if this is still under consideration - it's been some
> time and the issue seems well-defined.
> 
> http://austingroupbugs.net/view.php?id=789

The last minutes that say something about this bug are from 23 January 2014
in http://www.opengroup.org/austin/docs/austin_641.txt
They say:

Bug 789: Add set -o pipefail  OPEN
http://austingroupbugs.net/view.php?id=789

We agreed to leave this item open pending input from David Korn.

We had previously asked for input from Chet Ramey and David Korn and
got a reply from Chet but not from David.

-- 
Geoff Clare 
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England