Date: Mon, 14 Jan 2019 16:24:24 +0100
From: Martijn Dekker <mart...@inlv.org>
Message-ID: <8e92dc0f-1629-caa0-6ae0-064f78033...@inlv.org>
| This "mangled syntax" qualification is gratuitous disapproval and
| nothing more. You have still not come up with any actual argument
| against it.
It is disaproval, though I am not sure it is gratuitous, I think there are good
reasons for it, and wrt the 2nd sentence, I thought I did, but more
about that below.
You know my feelings on aliases, we have discussed that off list before...
| And, I repeat, it didn't require any "inventing".
I was referring to the particular syntax, not the idea - the idea has existed,
and been used, for many decades - Korn didn't invent it either - think back
to RATFOR (wich was a preprocessor, rather than langage macros, but to a
similar effrect, and is one of cases with less issues, as it was better
specified,
but still has all the problems caused because what gets executed, and from
which any coding bugs need to be debugged, is still FORTRAN) , and then
the initial C++; and has almost invariably turned out in hindsight to have been
the wrong thing to do (the wrong way to accomplish the goal, though at the
time it might have seemed to be the only practical solution.)
| Yes, there are implementation differences. Many other shell
| functionality has implementation differences as well.
Yes, they do, which is why (unfortunately) the standard has so much
listed as unspecified. It would be good if shell authors could agree on
what should be done in some of these cases, so there are less differences
and less that needs to be unspecified - at least when the relevant shells'
posix environment is enabled, when there is one. However, no-one wants
to risk breaking their users' scripts, so it is difficult to get changes made.
Even something so useful as "local" cannot be specified, because
of the variations in how it works (and the underlying model of what the
namespace for variables looks like).
| But defining a shell block using two aliases, e.g. one starting with a
| '{' and the other ending with a '}', is very straightforward, and works
| consistently on *all* the current shells that support aliases. Fact.
Agreed ... but write down clearly, and precisely, in standards type language
exactly what does work, and what does not, within a reasonable amount of
text so that it can be understood, and then perhaps there might be a way to
get it included.
If you just want to include '{' and '}' as two special case legal alias
strings, that could probably be done, but what is (or would be) the point?
It could even be specified what happens if an alias produces any single
reserved word, I suspect, but again, doing so isn't really useful.
As soon as you start adding more that is allowed you get both harder
to specify, and less likely to work the same with existing shells.
| By now, you know better than many that my library regression tests tend
| to test the shell (as well as the library) to its limits.
Yes, I also know that you deal with, and hide, many of the places where
shells do things differently. So you should know just how many variations
there are, in all areas, and aliases really aren't any different. Making some
of the uses unspecified is no different than any of the other unsppeciied
areas in the standard.
That is, it would be much better if everything currently unspecified was
specified, but it had better be specified MY way, not the way someone
else does it..... (no, that is not going to happen.)
| (I'm sure you'll agree that triggering segfaults in your NetBSD sh
| development version doesn't count as my method not working,
Of course not.
| and Isuspect aliases have nothing to do with that anyway
They did, as I'm sure you'll remember, you made core dumps from the
alias code which was bug riddled, for > 20 years, and no-one had ever
cared about until you noticed it all ... which kind of demonstrates just
how little aliases are used in scripts in general, and how unimportant
specifying them in POSIX is, to anyone except you.
Since aliases will not be going away from anyone's shells (or not in any
forseeable period anyway - for the same reason that the shells won't agree
to fix the (sometimes trivial) differences in other areas, or come to some
kind of compromise to allow a "local" builtin to be specified)
I suspect that you will be able to deal with it, and find the parts that work,
even if unspecified in the standard, and work around (or just avoid)
the parts that don't - just like you do for everything else which is
unspecified,
and yet which can be made to work if you know what the difference are, and
are prepared to deal with them.
| -- the problem appeared when my new loop construct started using
| FIFOs intensively; I was making use of aliases long before that.)
I doubt that fifo's have anything to do with it, I suspect (and this is
just a guess for now) that the issue relates to the agressive use of
pushing and popping file descriptors (n>&m on compound statements)
which increased a lot with the use of fifos.
While aliases are very unlikely to be directly influencing anything,
and that the same bug would exist if the code were simply written
out longhand (with the aliases all expanded) here, the problem is that
all of that file descriptor manipulation is buried in multiple layers of
aliases, which do more than just manipulate file descriptors, and
unravelling all of that to work out what is going on is exceedingly
difficult and tedious, as while I suspect I know the area in which the
bug exists, that's currently little more than intuition - and it certainly
takes something quite complex to trigger it, so working out what is
going on is exceedingly difficult.
That's what I was referring to in an earlier message, and the "any actual
argument against it" from the beginning of this message. Everything is
so convoluted, and so intertwined, that determining what is really happening
in terms of the underlying shell constructs is not at all easy. If there had
been no aliases, and the same code was just written, then the particular
failing construct would be much easier to spot, and then to work out what
had actually led to the situation where the bug exists.
I have been attempting to do that to your code - undo all the aliases, and
write actual shell code, for the area where the problem seems to occur,
so I have something to work on, but it is not at all easy.
| So, any outright claim that the new draft text "documents what actually
| works in the wild" would be false.
No it isn't, you're just rreading it as "documents all that works in the wild"
which is not what I said. What is documented works (we believe).
| I think you know this, hence your
| added qualifier, "(and is reasonable to specify)".
Yes, as in everything , one can be more precise, by adding millions of "except
when", "unless", "but only if", "provided that" ... type clauses all over
the place, in which case more could be included in what is specified.
The problem with this is that it is very difficult to do correctly, and avoid
the
wording unexpectedly saying something that was not intended - I expect
the authors of the current published alias wording thought they were
describing what, and how, things actually work too - they did not set
out to write nonsense.
My guess is that the best that could be done woult be to include a long
list of alias strings (the names used are obvious;y irrelevant) which are
expected to work, and do work in current shells - but there is no way that
such a list could be complete, and I doubt it would satisfy you, or
anyone else.
| Somehow I get the feeling you're not going to consider anything that
| sanctions an alias usage you disapprove of "reasonable".
What I consider has very little to do with anything, I do little more than
make comments here, I don't participate when the decisions are made.
Sometimes my opinions seem to carry some weight, other times they're
ignored.
| The historical practice with defining shell blocks using aliases is
| clearly useful,
I'm not sure I wold go that far, sh already has '{' '}' for making
blocks, which do work, and are all that is really needed. That is,
there is a way to accomplish what is desired (which there must be,
as aliases just transform one set of input tokens into another)
and providing multiple ways to achieve the same thing seems to
be overkill, rather than particularly useful. It certainly requires
that for anyone to understand the result, they need to first understand
sh syntax (that's required in all cases of course) but then also
understand all of the aliases that have been added and what they
do, and how and when they can be used, and their interactions,
in order to be able to understand a script using them. I don't
really consider that as being useful, unless you're entering a
shell code obscurification contest.
| clearly portable, and would be perfectly reasonable to specify.
Perhaps, but ...
| The standard developers just need to find the accurate wording
| for it.
No, as you're the one who wants this (perhaps almost the only one),
you need to do that - and expect the result to be pulled apart to make
sure it specifies things correctly, and only what really does work.
If no-one supplies wording, and there is no reason for anyone who
doesn't care about this, or actively believes that it is not a good idea,
to do it, then it simply will not happen.
| Then, please, show me the problems.
|
| Particularly the real, concrete, practical problems with this:
|
| alias TEST='{ testFn() {'
| alias ENDT='}; doTest; }'
|
| doTest() {
| function calling testFn and handling result goes here
| }
|
| TEST
| test code goes here
| ENDT # and many more of these
|
| What problems does this cause for the regression test scripts?
I did not ever say that there were problems (other than obscurification) in
those 2 aliases, what I said was that writing wording that allows those
(and similar) cases, which work, and excudes the ones that don't, is not
at all easy to accomplish. What's more, those only work because you
know what is in them, and how to use them properly, for example, do
you expect
TEST ENDT (or with a newline instead of the space)
or
TEST ; ENDT
to enter an empty, or placeholder, test to work?
What happens if I embed one TEST ENDT pair inside another ?
And lots of similar issues. Care to explain any of that to users?
Note: these are not examples of the aliases not working, they are illustrations
of why, even if they do work, using aliases this way is not a good idea.
If those aliases are just for your covenience writing the tests, then once
that's done, you can simply go through and do a global substitute in the
editor, and replace the aliases with their values, changing nothing, yet
avoiding exposing others to constructs that they do not understand.
| It is not the job of the POSIX standard to stop users from shooting
| themselves in the foot.
No, but it is to tell users what they can do, and expect to work, and what,
when if they do it (even when something useful happens in one shell)
they cannot necessarily expect to work (and also what they cannot do
at all).
| That is a job I intend to take on with the modernish library, with the
| safe mode, pervasive command hardening, etc.
All of that could be done with no aliases at all. The aliases just
implement the frills.
| > One more data point, I am sure you're aware that I am trying to find
| > a bug in the NetBSD sh that your scripts trigger (but apparently has
| > never been seen elsewhere).
|
| ...yet, as far as you know.
Yes, of course.
| Modernish consistently triggers segfaults.
Actually, that's not the current problem, if it were, it would be easier,
as the core file would provide clues. If you reall the cirrent problem
is lots of deadlocked subshells, that just sit there, doing nothing
(maybe rather than idle, producing more - particularly when some
of them are killed, I have also forgtten the exact issue! (but I still
have the e-mail)).
| Anything could trigger those intermittently, without you necessarily
| hearing about it. Users restart the computer, find it works the second
| time around, shrug, and move on.
They could, except that this doesn't work the second time around either.
There is an issue, it is just very very difficult to trigger. But when
the code exists that makes it happen, it happens reliably (if it weren't
for that, with something this complex, I probably would give up on the
hunt, until someone found a reliable trigger, or I fluked an understanding.)
I am not disputing that we (the NetBSD sh) have a bug, and I would
certainly like to find and fix it, the current issue is ...
| This is because you're running the regression test suite, which is
| designed to do exactly that
exactly, that's what it takes to trigger this problem. As you said above,
it is unlikely that aliases have anything to do with the actual bug, but
they are making it very difficult to find. If those tests had been written
in straight out shell code, I believe (I hope) I would have found the
problem weeks ago and once found, fixing usually takes very little time.)
And if you ask what the difference would be, then it is because in
plain sh code, I can add (unavailable to normal users) debug points
in the script, to watch (very closely) what happens in the area where
the problem occurs, and then if needed, go bacwards to look for
whatever happened earlier which caused the problem. I can do
that in the aliases too - but then they are used so often, all over
the place, that way too much output to comprehend is produced.
The same happens if I step backwards over one alias call - it
prouuces code that does so much that the output is meaningless.
| In the end, that has been for the benefit of both modernish and many
| shells, as while fixing modernish bugs I also report all the shell bugs
| I find.
Agreed, and as I said, I, and I would assume, the authors of other
shells, are grateful for all of that - having bugs reported, is always
useful., and even better if there's a relatively simple trigger method,
which usually you do the work to provide. Here, you gave up on that
(and I agree, it isn't your job) - but you understand what is happening
in all your aliases, and the functions they involke behind the scenes,
where no-one else does.
| Individual functionality on my current code (which has been developed
| and refactored quite a lot since last time I sent you a tarball) seems
| to be working fine, on the latest NetBSD 8.0 sh binary you sent me:
What changed? (we ought to be having this discussion off list ...)
If I knew what you altered, which avoided the hang forever that was
there in November/December, that might allow me to narrow down the
cause of the problem (which I very much doubt has magically fixed itself).
| So it looks like the basics definitely work. In the next few days, I'll
| try to test all the module functionality individually, let you know if
| anything breaks, and send you reports of any shell bugs I find. If
| anything changed since 2018-11-30, please send me your latest sh binary
| compiled for NetBSD 8.0.
I will do that, as some other obscure alias bugs have (very) recently been
corrected. But nothing relevant to the issue in question, as (with our shared,
and take/provide type history) these issues were also in the FreeBSD sh,
and that one did not have the problem I'm currently hunting (slowly...)
apparently (you told me that what was failing on NetBSD was working
with FreeBSD), so the recently uncovered alias iussues - found by testing
inspired by examples discussed on this list - cannot be related, and as
you said above, and I think I agree, aliases aren't directly related to the
bug, they just make it harder to find.
kre
