RE: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Schwarz, Konrad]

> -Original Message-
> From: Martijn Dekker [mailto:mart...@inlv.org]
> Subject: Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant
> 
> Op 30-09-17 om 17:35 schreef Alan Coopersmith:
> >> Where/how would I report Solaris bugs?
> >
> > Customers with support contracts can report bugs via Oracle Support.
> 
> I was afraid it might be something like that.

Perhaps your tests could be included in the POSIX conformance test suite?

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Martijn Dekker]

Op 30-09-17 om 17:35 schreef Alan Coopersmith:
> It is true that ksh88 is no longer being developed, but the Solaris
> /usr/xpg4/bin/sh is a fork of ksh88, modified for standards compliance,
> that is still being maintained, and bugs fixed as necessary.

That's good to know!

>> Where/how would I report Solaris bugs?
> 
> Customers with support contracts can report bugs via Oracle Support.

I was afraid it might be something like that.

I'm no corporate customer, I'm just a slightly obsessed guy with a weird
love for that quirky shell language, trying to prove the idea that a
good solid shell library could (a) really enhance the language, as well
as get rid of some of its main annoyances, and (b) make cross-platform
scripting downright feasible. https://github.com/modernish/modernish

Shell bugs are a major obstacle to that goal. As part of that, I've been
on a bit of a crusade to get all the POSIX-related bugs fixed in all the
shells. My library experiments with things in shell that I don't believe
have really been tried before, so I keep running into all sorts of bugs
in various shells. Some of those bugs went unreported for decades.

All of which is to say, I'm not about to take out a support contract
just to report bugs, but I think it might still be in your interest to
get my bug reports for /usr/xpg4/bin/sh.

> The people responsible for running the standards conformance test suites
> on Solaris are also on this mailing list, and can file bugs if they see
> issues we need to fix.

I hope one of them can help by verifying and filing the bugs that I
posted to this list.

Is there anyone in particular I could cc in hopes of getting it noticed?

Thanks,

- Martijn

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Joerg Schilling]

Alan Coopersmith  wrote:

> On 10/ 1/17 04:10 AM, Joerg Schilling wrote:
> > Is Oracle planning to run a SUSv7-tc2 certification?
>
> I don't know which version we plan to certify against next, and probably 
> couldn't pre-announce it here if I did.

Well, good to see that you are still "there".


> > My impression is that as a result of various modification in the standard, 
> > more
> > than trivial bug-fixes need to be applied on ksh88 to make it SUSv7-tc2
> > compiant.
>
> Have the test suites been updated to catch all of these cases?

I don't know, but I will probably run the SUSv7-tc2 tests soon on Sun's ksh88 
from 2010 and on my current bosh.

Before I can do that, I need to get a dedicated machine, as the tests like to 
get root privileges and I am afraid of the state of my development machine.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Joerg Schilling]

Geoff Clare  wrote:

> > What I have in mind for "${a-"*"}" is [...]
>
> This topic was discussed at great length when we resolved bug 221
> back in 2010.  I don't think there is anything to be gained from
> starting a new discussion of "${a-"*"}".

Since I did not yet see any explanation that would make any of the 
implementations that behave different than "bosh" appear more correct than
the implementation I use in bosh, I am fine with that resolution.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Chet Ramey]

On 10/6/17 8:08 AM, Joerg Schilling wrote:
> Chet Ramey  wrote:
> 
>> ksh had functions dating from the early 1980s in its original incarnation
>> as a form interpreter. Korn mentions functions as one of the first things
>> he added to the Bourne shell. I don't know whether ksh-83 had the SVR2
>> function declaration syntax in addition to `function name', but ksh-86
>> certainly did.
> 
> Given that functions appeared in the Bourne Shell in 1984, a ksh from 1983 
> did 
> most likely not support a compatible syntax.

Maybe. Bourne added functions to the research version of the shell years
before they appeared in the SVR2 sh (1984), and Korn certainly had access
to that. But we'll never know unless someone digs up ksh-83 sources.

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Joerg Schilling]

"stephane.chaze...@gmail.com"  wrote:

> Note that I'm not under the impression that ksh93 aimed for
> POSIX compliant. For instance, it did explicitely break POSIX
> compliance (and backward compatibility) when the <> redirection
> operator changed from being short for 0<> to  short for 1<> (I
> suppose to be consistent with the new <>; operator (and that one
> more generally wants to use it on stdout than stdin))

This is a really interesting information.

But this is in conflict with the documentation in the man page
The file "RELEASE" however mentions the change.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Joerg Schilling]

Chet Ramey  wrote:

> ksh had functions dating from the early 1980s in its original incarnation
> as a form interpreter. Korn mentions functions as one of the first things
> he added to the Bourne shell. I don't know whether ksh-83 had the SVR2
> function declaration syntax in addition to `function name', but ksh-86
> certainly did.

Given that functions appeared in the Bourne Shell in 1984, a ksh from 1983 did 
most likely not support a compatible syntax.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Chet Ramey]

On 10/5/17 1:04 PM, Joerg Schilling wrote:
> "Schwarz, Konrad"  wrote:
> 
>> Perhaps I haven't understood the full import, but ksh92 has an additional
>> function syntax (function name { body; }), where "function" is a keyword,
>> that provides local variables declared with typeset.
>>
>> "The New Kornshell" states
>>
>>
>> Because of deficiencies in the POSIX shell standard, there are two
>> types of functions.  POSIX functions [...] do not provide any scoping [...]
>> This makes it hard to write functions that don't have side effects.
> 
> Both types of functions did already exist in ksh88

Both types of function declarations existed in ksh88 (and ksh86), but
functions  declared using the `Posix' syntax did not behave any differently
from functions declared using the orignal ksh `function name' syntax. They
allowed local variables, and scoped traps and options locally.

> 
>> So it seems ksh92's implementation of POSIX functions is for standards 
>> conformance,
>> and the new syntax was introduced to avoid breaking that conformance.
> 
> Since in 1988, there was no POSIX standard for the behavior of commands, this 
> must exist for compatibility with the Bourne Shell.

The `name() { command; }' function declaration syntax was introduced for
Bourne shell compatibility. The behavior change with ksh92-ksh93 was for
Posix conformance.

> 
> The Bourne Shell introduced the function method, that in the 1990s was added 
> to 
> the standard, with it's 1984 release.
> 
> I am sorry, but I cannot verify whether ksh did implement functions before.

ksh had functions dating from the early 1980s in its original incarnation
as a form interpreter. Korn mentions functions as one of the first things
he added to the Bourne shell. I don't know whether ksh-83 had the SVR2
function declaration syntax in addition to `function name', but ksh-86
certainly did.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Joerg Schilling]

"Schwarz, Konrad"  wrote:

> Perhaps I haven't understood the full import, but ksh92 has an additional
> function syntax (function name { body; }), where "function" is a keyword,
> that provides local variables declared with typeset.
>
> "The New Kornshell" states
>
>
> Because of deficiencies in the POSIX shell standard, there are two
> types of functions.  POSIX functions [...] do not provide any scoping [...]
> This makes it hard to write functions that don't have side effects.

Both types of functions did already exist in ksh88

> So it seems ksh92's implementation of POSIX functions is for standards 
> conformance,
> and the new syntax was introduced to avoid breaking that conformance.

Since in 1988, there was no POSIX standard for the behavior of commands, this 
must exist for compatibility with the Bourne Shell.

The Bourne Shell introduced the function method, that in the 1990s was added to 
the standard, with it's 1984 release.

I am sorry, but I cannot verify whether ksh did implement functions before.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[stephane.chaze...@gmail.com]

2017-10-05 10:40:14 -0400, Chet Ramey:
[...]
> It's not new syntax. The function keyword was introduced in ksh83 before
> Korn added the name() { command; } syntax, and always had the
> separate scoping semantics. But you're right about the Posix functions:
> he modified ksh93 to remove the non-Posix functionality (local variables,
> options, and traps) from functions declared using the standard syntax.
[...]

Note that "typeset" introducing local scope does not make ksh88
non-compliant as "typeset" is not specified by POSIX, but the
fact that there is local scope for options and traps defined
with the POSIX "set"/"trap" API makes it non-compliant.

IOW, ksh93 did not have to change to behaviour of "typeset" in
POSIX-defined functions to make it POSIX compliant (that would
even be the contrary as POSIX currently says that "typeset" may
be used to restrict scope in future versions of the standard),
but it still makes sense that it did to be consistent with the
function/trap handling and to have a type of function that runs
in the caller's scope.

Note that I'm not under the impression that ksh93 aimed for
POSIX compliant. For instance, it did explicitely break POSIX
compliance (and backward compatibility) when the <> redirection
operator changed from being short for 0<> to  short for 1<> (I
suppose to be consistent with the new <>; operator (and that one
more generally wants to use it on stdout than stdin))

-- 
Stephane

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Chet Ramey]

On 10/5/17 8:38 AM, Schwarz, Konrad wrote:
>> -Original Message-
>> From: Chet Ramey [mailto:chet.ra...@case.edu]
>> On 10/4/17 8:32 AM, Joerg Schilling wrote:
>>
 The portion about "local variables" in the quoted text means that the
 presence of the `typeset' command doesn't make a variable local:
 there are no local variables in ksh93 posix functions at all.
>>>
>>> The problem with ksh93 is that it does not allow to create variables
>>> with local scope inside POSIX/Bourne functions while ksh88 allows to
>>> make a variable "local" by using the "local builtin" that really is a
>>> builtin alias to
>> This is exactly what I said, though Korn doesn't consider it a "problem"
>> at all, since Posix has no local variables.
> 
> Perhaps I haven't understood the full import, but ksh92 has an additional
> function syntax (function name { body; }), where "function" is a keyword,
> that provides local variables declared with typeset.

This is, again, what I said. These are not "Posix" functions, so they
have different semantics.

> 
> "The New Kornshell" states
> 
> 
> Because of deficiencies in the POSIX shell standard, there are two
> types of functions.  POSIX functions [...] do not provide any scoping [...]
> This makes it hard to write functions that don't have side effects.
> 
> 
> So it seems ksh92's implementation of POSIX functions is for standards 
> conformance,
> and the new syntax was introduced to avoid breaking that conformance.

It's not new syntax. The function keyword was introduced in ksh83 before
Korn added the name() { command; } syntax, and always had the
separate scoping semantics. But you're right about the Posix functions:
he modified ksh93 to remove the non-Posix functionality (local variables,
options, and traps) from functions declared using the standard syntax.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Geoff Clare]

Joerg Schilling  wrote, on 04 Oct 2017:
>
> Jilles Tjoelker  wrote:
> 
> > However, things like "${a-"*"}" are unspecified (in practice, the
> > asterisk may either be literal or generate pathnames, depending on the
> > shell). With
> >
> > cat < > ${a-"*"}
> > EOF
> 
> echo "${a-"*"}"
> 
> is equivalent to "echo *" for the Bourne Shell, ksh88, ksh93 and bosh.
> 
> ${a-"*"} is different, as this quotes "*". Even with this, 
> Bourne Shell, ksh88 and bosh behave identical, only ksh93 differs.
> 
> What I have in mind for "${a-"*"}" is [...]

This topic was discussed at great length when we resolved bug 221
back in 2010.  I don't think there is anything to be gained from
starting a new discussion of "${a-"*"}".

-- 
Geoff Clare 
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Chet Ramey]

On 10/4/17 8:52 AM, Joerg Schilling wrote:
> Chet Ramey  wrote:
> 
>>> But this is what you get with ksh93:
>>>
>>> ksh93 -c 'a=0; a() { local a; a=12; } ; a; echo $a'
>>> ksh93[1]: local: not found [No such file or directory]
>>> 12
>>
>> It doesn't work even if you use `typeset', which is accepted.
> 
> But this would prevent ksh93 from being used as /bin/sh on various Linux 
> distros.

So? Why is that a problem here? We're talking about Posix conformance.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Joerg Schilling]

Chet Ramey  wrote:

> > But this is what you get with ksh93:
> > 
> > ksh93 -c 'a=0; a() { local a; a=12; } ; a; echo $a'
> > ksh93[1]: local: not found [No such file or directory]
> > 12
>
> It doesn't work even if you use `typeset', which is accepted.

But this would prevent ksh93 from being used as /bin/sh on various Linux 
distros.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Chet Ramey]

On 10/4/17 8:32 AM, Joerg Schilling wrote:

>> The portion about "local variables" in the quoted text means that
>> the presence of the `typeset' command doesn't make a variable local:
>> there are no local variables in ksh93 posix functions at all.
> 
> The problem with ksh93 is that it does not allow to create variables with 
> local scope inside POSIX/Bourne functions while ksh88 allows to make a 
> variable 
> "local" by using the "local builtin" that really is a builtin alias to 
> "typeset":
> 
>   ksh -c 'a=0; a() { local a; a=12; } ; a; echo $a'
> 
> prints 0 with ksh88.

This is exactly what I said, though Korn doesn't consider it a "problem"
at all, since Posix has no local variables.

> 
> But this is what you get with ksh93:
> 
> ksh93 -c 'a=0; a() { local a; a=12; } ; a; echo $a'
> ksh93[1]: local: not found [No such file or directory]
> 12

It doesn't work even if you use `typeset', which is accepted.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Joerg Schilling]

Chet Ramey  wrote:

> > ksh88 has POSIX compliant functions, see:
> > 
> > ksh -c 'a=0; a() { a=12; } ; a; echo $a'
>
> Nobody is arguing whether or not unqualified assignment statements in ksh88
> modify global variables. This is not the behavior that makes ksh88
> functions non-conformant.
>
> I included the text above to note that Korn doesn't consider ksh88
> functions conformant and that he needed to modify ksh behavior for
> conformance.
>
> The portion about "local variables" in the quoted text means that
> the presence of the `typeset' command doesn't make a variable local:
> there are no local variables in ksh93 posix functions at all.

The problem with ksh93 is that it does not allow to create variables with 
local scope inside POSIX/Bourne functions while ksh88 allows to make a variable 
"local" by using the "local builtin" that really is a builtin alias to 
"typeset":

ksh -c 'a=0; a() { local a; a=12; } ; a; echo $a'

prints 0 with ksh88.

But this is what you get with ksh93:

ksh93 -c 'a=0; a() { local a; a=12; } ; a; echo $a'
ksh93[1]: local: not found [No such file or directory]
12

Linux distros decided (while moving to "dash" as /bin/sh) that they "need" a 
basic POSIX shell that in addition supports the "local" feature.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Joerg Schilling]

Jilles Tjoelker  wrote:

> However, things like "${a-"*"}" are unspecified (in practice, the
> asterisk may either be literal or generate pathnames, depending on the
> shell). With
>
> cat < ${a-"*"}
> EOF

echo "${a-"*"}"

is equivalent to "echo *" for the Bourne Shell, ksh88, ksh93 and bosh.

${a-"*"} is different, as this quotes "*". Even with this, 
Bourne Shell, ksh88 and bosh behave identical, only ksh93 differs.

What I have in mind for "${a-"*"}" is that there is no special treatment for '"'
in such a string and the only difference between the Bourne Shell and POSIX is 
that ${a... needs to be parsed in a way that does not let the word end before 
the related "}" is seen. This results in ${a- } to be treated as a single word 
(and not as two words as done by the Bourne Shell) and thus permits the macro 
expander to do something useful.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Joerg Schilling]

Martijn Dekker  wrote:

> Op 01-10-17 om 14:43 schreef Joerg Schilling:
> > The standard does not claim that a function has to be executed the same way 
> > as  
> > a compound ist would have been executed, or do you see such a claim in the 
> > standard?
> > 
> > The way I read the standard just mentions that the function body in a 
> > function 
> > definition is parsed like a compund list.
>
> 2.9.5 Function Definition Command
> [...]
> fname ( ) compound-command [io-redirect ...]
> [...]
> "The argument /compound-command/ represents a compound command, as
> described in Compound Commands."
>
> It "represents" a compound command. This does not mean it is only parsed
> as a compound command; it means it is to be treated as a compound
> command in every way described under "Compound Commands".

Then it is strange, that the deviation from ksh88 is not mentioned in the POSIX 
standard.

Maybe, we should add a related bug and enhance the POSIX text.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Geoff Clare]

Alan Coopersmith  wrote, on 02 Oct 2017:
>
> On 10/ 1/17 04:10 AM, Joerg Schilling wrote:
> >Is Oracle planning to run a SUSv7-tc2 certification?
> 
> I don't know which version we plan to certify against next, and probably
> couldn't pre-announce it here if I did.
> 
> >My impression is that as a result of various modification in the standard, 
> >more
> >than trivial bug-fixes need to be applied on ksh88 to make it SUSv7-tc2
> >compiant.
> 
> Have the test suites been updated to catch all of these cases?

The changes to align VSC with SUSv4 TC2 are in VSC5.3.13betaNW which
just finished its beta review.

P.S. Joerg - you need to subtract 3 from the Issue number to get
the SUSvX number, i.e. Issue 7 is SUSv4 not SUSv7.

-- 
Geoff Clare 
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Alan Coopersmith]

On 10/ 2/17 04:28 PM, Thor Lancelot Simon wrote:

On Mon, Oct 02, 2017 at 04:07:37PM -0700, Alan Coopersmith wrote:

On 10/ 1/17 04:10 AM, Joerg Schilling wrote:

Is Oracle planning to run a SUSv7-tc2 certification?


I don't know which version we plan to certify against next, and probably
couldn't pre-announce it here if I did.


I have to ask, what is the likelihood this will actually happen?  Hasn't
Oracle essentially ended Solaris development?


The rumors of our demise have been somewhat exaggerated.

While we lost a lot of engineers, there's still hundreds of us left working
on Solaris 11.4, which our VP's have recently publicly confirmed, during both
the SPARC M8 product launch and presentations this week at Oracle OpenWorld,
is planned to ship in calendar year 2018.

--
-Alan Coopersmith-   alan.coopersm...@oracle.com
 Oracle Solaris Engineering - https://blogs.oracle.com/alanc

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Alan Coopersmith]

On 10/ 1/17 04:10 AM, Joerg Schilling wrote:

Is Oracle planning to run a SUSv7-tc2 certification?


I don't know which version we plan to certify against next, and probably 
couldn't pre-announce it here if I did.



My impression is that as a result of various modification in the standard, more
than trivial bug-fixes need to be applied on ksh88 to make it SUSv7-tc2
compiant.


Have the test suites been updated to catch all of these cases?

--
-Alan Coopersmith-   alan.coopersm...@oracle.com
 Oracle Solaris Engineering - https://blogs.oracle.com/alanc

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Chet Ramey]

On 10/1/17 9:15 AM, Martijn Dekker wrote:

> 2.9.4 Compound Commands
> [...]
> { compound-list ; }
> Execute compound-list in the current process environment.
> 
> This means any compound-command in that form (which therefore includes
> any such compound-command that is the body of a function definition), is
> executed in the current process environment.

This sentence: "The compound-command shall be executed whenever the
function name is specified as the name of a simple command" from the
function definition description supports that interpretation.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Chet Ramey]

On 10/1/17 8:43 AM, Joerg Schilling wrote:
> Martijn Dekker  wrote:
> 
>>> Such a text is however missing in POSIX.
>>
>> It is not missing. As previously shown (and you removed from the quoted
>> text), the POSIX definition of a "function" incorporates the POSIX
>> definition of a "Compound Command". And that says:
>>
>> { compound-list ; }
>> Execute compound-list in the current process environment.
>>
>> (see link given previously).
>>
>> "In the current process environment" seems pretty clear to me.
> 
> The standard does not claim that a function has to be executed the same way 
> as  
> a compound ist would have been executed, or do you see such a claim in the 
> standard.

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_12

enumerates the cases where a subshell environment is used:

"Command substitution, commands that are grouped with parentheses, and
asynchronous lists shall be executed in a subshell environment.
Additionally, each command of a multi-command pipeline is in a subshell
environment; as an extension, however, any or all commands in a pipeline
may be executed in the current environment. All other commands shall be
executed in the current shell environment."

And while it does not explicitly say that shell functions are executed in
the calling environment, between this text and the text in the description
of function definitions, it's clearly implied.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Chet Ramey]

On 9/30/17 9:01 AM, Joerg Schilling wrote:

>> If I read the spec correctly, one should expect
>>
>> noglob() { set -f; }
>> glob() { set +f; }
>>
>> noglob; echo *; glob
>>
>> to output *, but that doesn't work with /usr/xpg4/bin/sh on
>> Solaris.
>>
>> Local scope for options is useful in practice (though a
>> lot more useful when it doesn't affect other functions called
>> from the function as with static scoping) but here breaks
>> compliance and expectations when done by default. At least with
>> ash, zsh, bash4.4, ksh93, you get the choice. ksh88 doesn't seem
>> to support ksh93's ". f" syntax to avoid the local scope.
> 
> This is an interesting behavior, but it is not related to local/dynamic 
> scoping, since this is a test whether a function is run in a sub-shell.

Sort of. It's a test of whether or not modifying shell options in a
shell function affects the calling shell environment. In ksh88, this is
independent of whether or not the function is run in a subshell.

> I am not sure whether this is covered by POSIX at all.

It is, and shell functions are not run in a subshell environment.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Chet Ramey]

On 9/30/17 8:58 AM, Joerg Schilling wrote:
> Chet Ramey  wrote:
> 
>> This is from the ksh93 release documentation:
>>
>> 1.  Functions, defined with name() with ksh-93 are compatible with
>> the POSIX standard, not with ksh-88.  No local variables are
>> permitted, and there is no separate scope.  Functions defined
>> with the function name syntax, have local variables as in ksh-88
>> but are statically scoped as in C so that a function does not
>> automatically have access to local variables of the caller.
>> This change also affects function traces.
>>
>> I have to check my ksh88 book, but I believe that ksh88 functions had
>> local scope for traps and options as well as variables.
> 
> ksh88 has POSIX compliant functions, see:
> 
> ksh -c 'a=0; a() { a=12; } ; a; echo $a'

Nobody is arguing whether or not unqualified assignment statements in ksh88
modify global variables. This is not the behavior that makes ksh88
functions non-conformant.

I included the text above to note that Korn doesn't consider ksh88
functions conformant and that he needed to modify ksh behavior for
conformance.

The portion about "local variables" in the quoted text means that
the presence of the `typeset' command doesn't make a variable local:
there are no local variables in ksh93 posix functions at all.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Chet Ramey]

On 9/30/17 8:56 AM, Joerg Schilling wrote:

> It may be that ksh86 did not support Bourne Shell functions, but ksh88 
> definitely does.

I believe that the name() { command; } syntax was in ksh86.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Martijn Dekker]

Op 01-10-17 om 14:43 schreef Joerg Schilling:
> The standard does not claim that a function has to be executed the same way 
> as  
> a compound ist would have been executed, or do you see such a claim in the 
> standard?
> 
> The way I read the standard just mentions that the function body in a 
> function 
> definition is parsed like a compund list.

2.9.5 Function Definition Command
[...]
fname ( ) compound-command [io-redirect ...]
[...]
"The argument /compound-command/ represents a compound command, as
described in Compound Commands."

It "represents" a compound command. This does not mean it is only parsed
as a compound command; it means it is to be treated as a compound
command in every way described under "Compound Commands".

2.9.4 Compound Commands
[...]
{ compound-list ; }
Execute compound-list in the current process environment.

This means any compound-command in that form (which therefore includes
any such compound-command that is the body of a function definition), is
executed in the current process environment.

I honestly don't see how your reading of it makes sense.

- Martijn

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Joerg Schilling]

Martijn Dekker  wrote:

> > Such a text is however missing in POSIX.
>
> It is not missing. As previously shown (and you removed from the quoted
> text), the POSIX definition of a "function" incorporates the POSIX
> definition of a "Compound Command". And that says:
>
> { compound-list ; }
> Execute compound-list in the current process environment.
>
> (see link given previously).
>
> "In the current process environment" seems pretty clear to me.

The standard does not claim that a function has to be executed the same way as  
a compound ist would have been executed, or do you see such a claim in the 
standard?

The way I read the standard just mentions that the function body in a function 
definition is parsed like a compund list.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Martijn Dekker]

Op 01-10-17 om 12:58 schreef Joerg Schilling:
> Martijn Dekker  wrote:
>> So the shell options don't change outside of the function, but the
>> variable does. Which means that the function is not run in a subshell,
>> and that shell options have a mandatory local scope, so /usr/xpg4/bin/sh
>> functions are not POSIX compliant.
>>
>>> I am not sure whether this is covered by POSIX at all.
>>
>> It is. See:
>> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_05
> 
> My impression is that POSIX would need to define that a function has to be 
> executed completely inside the environment of the calling shell process.
> 
> Such a text is however missing in POSIX.

It is not missing. As previously shown (and you removed from the quoted
text), the POSIX definition of a "function" incorporates the POSIX
definition of a "Compound Command". And that says:

{ compound-list ; }
Execute compound-list in the current process environment.

(see link given previously).

"In the current process environment" seems pretty clear to me.

- M.

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Joerg Schilling]

Martijn Dekker  wrote:

> > This is an interesting behavior, but it is not related to local/dynamic 
> > scoping, since this is a test whether a function is run in a sub-shell.
>
> That's easy to disprove. I just did the the following on my Solaris 11.3
> VM using /usr/xpg4/bin/sh:
>
> $ noglob() { set -f; g=no; }
> $ glob() { set +f; g=yes; }
> $ noglob; echo "$- $g"
> ism no
> $ glob; echo "$- $g"
> ism yes
>
> So the shell options don't change outside of the function, but the
> variable does. Which means that the function is not run in a subshell,
> and that shell options have a mandatory local scope, so /usr/xpg4/bin/sh
> functions are not POSIX compliant.
>
> > I am not sure whether this is covered by POSIX at all.
>
> It is. See:
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_05

My impression is that POSIX would need to define that a function has to be 
executed completely inside the environment of the calling shell process.

Such a text is however missing in POSIX.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Martijn Dekker]

Op 30-09-17 om 15:01 schreef Joerg Schilling:
> Stephane Chazelas  wrote:
> 
>> I knew that was the case for ksh93's functions defined with the
>> function f {...;} syntax (though using static scoping), but
>> hadn't realised ksh88 also did local (dynamic) scoping for
>> options and traps even with the f(){...;} syntax.
>>
>> So it looks like that's another non-compliance.
>>
>> If I read the spec correctly, one should expect
>>
>> noglob() { set -f; }
>> glob() { set +f; }
>>
>> noglob; echo *; glob
>>
>> to output *, but that doesn't work with /usr/xpg4/bin/sh on
>> Solaris.
[...]
> This is an interesting behavior, but it is not related to local/dynamic 
> scoping, since this is a test whether a function is run in a sub-shell.

That's easy to disprove. I just did the the following on my Solaris 11.3
VM using /usr/xpg4/bin/sh:

$ noglob() { set -f; g=no; }
$ glob() { set +f; g=yes; }
$ noglob; echo "$- $g"
ism no
$ glob; echo "$- $g"
ism yes

So the shell options don't change outside of the function, but the
variable does. Which means that the function is not run in a subshell,
and that shell options have a mandatory local scope, so /usr/xpg4/bin/sh
functions are not POSIX compliant.

> I am not sure whether this is covered by POSIX at all.

It is. See:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_05

This says that part of the function definition is a Compound Command:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_04

which says that a compound command like ( some commands here ) is
executed in a subshell whereas one like { some commands here; } is not.

Hence, according to POSIX, a function defined as

fn() ( some commands here )

is executed in a subshell, whereas a function defined as

fn() { some commands here; }

is not.

- Martijn

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Alan Coopersmith]

On 09/29/17 06:38 PM, Martijn Dekker wrote:

Op 29-09-17 om 14:16 schreef Stephane Chazelas:

2017-09-29 11:59:52 +0200, Martijn Dekker:
[...]

Ouch, quite of lot of nasty bugs. Have you reported them to
Solaris?


No, because I thought ksh88 is no longer being developed. Is that incorrect?


It is true that ksh88 is no longer being developed, but the Solaris
/usr/xpg4/bin/sh is a fork of ksh88, modified for standards compliance,
that is still being maintained, and bugs fixed as necessary.  (Make sure
you're testing with that version, not stock ksh88 or /usr/bin/ksh, as you
will get different results in areas historical ksh behavior deviates from
the standards.)


Where/how would I report Solaris bugs?


Customers with support contracts can report bugs via Oracle Support.

The people responsible for running the standards conformance test suites
on Solaris are also on this mailing list, and can file bugs if they see
issues we need to fix.

--
-Alan Coopersmith-   alan.coopersm...@oracle.com
 Oracle Solaris Engineering - https://blogs.oracle.com/alanc

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Chet Ramey]

On 9/29/17 11:29 AM, Stephane Chazelas wrote:
> 2017-09-29 10:32:33 -0400, Chet Ramey:
>> On 9/29/17 5:59 AM, Martijn Dekker wrote:
>>> I've been in correspondence with a few people who still seem to believe
>>> that AT ksh88 is POSIX compliant because POSIX was originally based on
>>> ksh88. On Solaris, to this day, /usr/xpg4/bin/sh is ksh88 and is
>>> considered "the POSIX compliant shell".
>>
>> It's not and never has been.
> 
> Well AFAIK /usr/xpg4/bin/sh on Solaris 10 has been certified as
> Unix compliant. And the POSIX sh specification is "based" on
> (a subset of) ksh88 even if it deviates.
> 
>> The most obvious example is its lack of the "Posix functions" that Korn
>> introduced in ksh93 to make it conformant.
> 
> What do you mean?

This is from the ksh93 release documentation:

1.  Functions, defined with name() with ksh-93 are compatible with
the POSIX standard, not with ksh-88.  No local variables are
permitted, and there is no separate scope.  Functions defined
with the function name syntax, have local variables as in ksh-88
but are statically scoped as in C so that a function does not
automatically have access to local variables of the caller.
This change also affects function traces.

I have to check my ksh88 book, but I believe that ksh88 functions had
local scope for traps and options as well as variables.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant

[Chet Ramey]

On 9/29/17 5:59 AM, Martijn Dekker wrote:
> I've been in correspondence with a few people who still seem to believe
> that AT ksh88 is POSIX compliant because POSIX was originally based on
> ksh88. On Solaris, to this day, /usr/xpg4/bin/sh is ksh88 and is
> considered "the POSIX compliant shell".

It's not and never has been.

The most obvious example is its lack of the "Posix functions" that Korn
introduced in ksh93 to make it conformant. There are a number of other
non-conformant behaviors, even without considering bugs (for instance, I
believe the `test' builtin isn't conformant with the Posix number-of-
arguments algorithm, but I don't have a version handy to test).

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/