Re: [Toybox] making ./configure executable.

[Rob Landley]

On 02/06/2018 11:04 AM, enh wrote:
>> So when I say "a proper bash replacement" I'm _mostly_ targeting bash
>> 2.x with proper utf8 and the occasional new feature like ~= (regex
>> matches) that are easy to do. Stuff like <(command) and file{1,2}.txt
>> are over 20 years old.
> 
> <(command) isn't supported in mksh (used by Android) --- i think
> that's the only thing missing in mksh that anyone's really noticed.

If mksh was available under the toybox license I'd start with that
codebase, but alas it isn't. :( (Yes, I've asked mirablos.)

If android doesn't use toysh, it doesn't use toysh. But I need one for
my hermetic build environment goal, and might as well do it right...

But first, lemme get development back up to speed and clear this backlog
of half-merged stuff...

Rob
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] making ./configure executable.

[enh]

On Tue, Feb 6, 2018 at 8:45 AM, Rob Landley  wrote:
> On 02/05/2018 09:41 PM, Robert Thompson wrote:
>> Yeah, I ran into some similar issues years ago...  In my case,  I got
>> burned into learning not to assume that bash (or any other shell) is
>> correct, or even necessarily *self* consistent(even on linux), so I
>> tend to reflexively do differential checks. It's a very small amount
>> of effort in the normal case, but months of project time and a major
>> missed deadline tend to leave a bit of a scar :)
>>
>> there was a "posix, we swear" vendor /bin/sh, a classic /bin/bash, a
>> "new and sexy" /bin/bash, and the same version/same configuration
>> /bin/bash running on a different hardware/os...
>>
>> We wasted months assuming that the newer bash was most likely correct
>> (it was a heisenbug, both rare and inconsistent). In that case, turned
>> out that the posix sh was technically correct and consistent, and
>> *all* of the bashes exhibited *different* incompatible heisenbugs. And
>> they tended to show up only about a day into a massive
>> un-checkpointable processing run...
>
> The FSF is bad at writing software, yes. :)
>
> Part of my formative experience with Linux is that the FSF's software
> development efforts basically died for five years (1999-2004), so
> distros were using fixed stable versions of the userspace stuff. It was
> all basically "done". There were some ubiquitous local vendor patches
> (adding -j to tar and such in the 5 year gap between 1.13 and 1.14 being
> released, for example), and things that DID need to change like compiler
> development forked off as other projects (egcs, libc5), but there was a
> de-facto set of tools everybody agreed on, and that mostly included
> specific _versions_, and nobody had to care what the FSF thought about
> anything.
>
> Then the FSF noticed its ftp server had been broken into and was
> distributing malware, and they took all the source tarballs down and
> replaced them with a note asking if anybody had backup copies (of course
> the FSF itself didn't: it's the FSF). I noticed this had happened
> because I was doing Linux From Scratch, and three weeks later nobody'd
> cared enough to do anything about it so I poked slashdot
> (https://developers.slashdot.org/story/03/08/13/1530239/fsf-ftp-site-cracked-looking-for-md5-sums)
> and that whole debacle seems to have gotten people to notice the FSF was
> dead, but instead of _replacing_ them they volunteered at them and tried
> to work through their horrible crazy lens, and when that round of
> excitement subsided and the FSF went back to dying they tried suing
> people to reclaim attention
> (https://www.linux.com/news/gpl-requirement-could-have-chilling-effect-derivative-distros)
> and that "all we've got to squeeze money out of people is control of the
> license" nonsense snowballed into GPLv3.
>
> Not a net win, really.
>
> Anyway, even back at busybox I was mostly targeting equivalents of those
> 5-year "stable" versions from 1999, and only cherry-picking newer
> features when people asked for them or the kernel grew some new thing I
> wanted to take advantage of. These days my baseline is posix-2008 plus
> any Y2k-era Linux extensions that posix rejected because That's Not
> Solaris I Want Solaris La La La Solaris.
>
> So when I say "a proper bash replacement" I'm _mostly_ targeting bash
> 2.x with proper utf8 and the occasional new feature like ~= (regex
> matches) that are easy to do. Stuff like <(command) and file{1,2}.txt
> are over 20 years old.

<(command) isn't supported in mksh (used by Android) --- i think
that's the only thing missing in mksh that anyone's really noticed.

> I really don't care what the FSF's done since about 2005. As far as I'm
> concerned the Mepis thing I linked above was a career limiting move
> _before_ they split "the GPL" into incompatible versions so the Linux
> kernel and Samba can't share code even though they implement 2 ends of
> the same protocol and are both "GPL".
>
> (None of this excuses dash, though. :)
>
> Rob
>
> P.S. Yes, I did my own GPLv2 enforcement lawsuits circa 2006. That's
> another story...
> ___
> Toybox mailing list
> Toybox@lists.landley.net
> http://lists.landley.net/listinfo.cgi/toybox-landley.net
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] making ./configure executable.

[Rob Landley]

On 02/05/2018 09:41 PM, Robert Thompson wrote:
> Yeah, I ran into some similar issues years ago...  In my case,  I got
> burned into learning not to assume that bash (or any other shell) is
> correct, or even necessarily *self* consistent(even on linux), so I
> tend to reflexively do differential checks. It's a very small amount
> of effort in the normal case, but months of project time and a major
> missed deadline tend to leave a bit of a scar :)
> 
> there was a "posix, we swear" vendor /bin/sh, a classic /bin/bash, a
> "new and sexy" /bin/bash, and the same version/same configuration
> /bin/bash running on a different hardware/os...
> 
> We wasted months assuming that the newer bash was most likely correct
> (it was a heisenbug, both rare and inconsistent). In that case, turned
> out that the posix sh was technically correct and consistent, and
> *all* of the bashes exhibited *different* incompatible heisenbugs. And
> they tended to show up only about a day into a massive
> un-checkpointable processing run...

The FSF is bad at writing software, yes. :)

Part of my formative experience with Linux is that the FSF's software
development efforts basically died for five years (1999-2004), so
distros were using fixed stable versions of the userspace stuff. It was
all basically "done". There were some ubiquitous local vendor patches
(adding -j to tar and such in the 5 year gap between 1.13 and 1.14 being
released, for example), and things that DID need to change like compiler
development forked off as other projects (egcs, libc5), but there was a
de-facto set of tools everybody agreed on, and that mostly included
specific _versions_, and nobody had to care what the FSF thought about
anything.

Then the FSF noticed its ftp server had been broken into and was
distributing malware, and they took all the source tarballs down and
replaced them with a note asking if anybody had backup copies (of course
the FSF itself didn't: it's the FSF). I noticed this had happened
because I was doing Linux From Scratch, and three weeks later nobody'd
cared enough to do anything about it so I poked slashdot
(https://developers.slashdot.org/story/03/08/13/1530239/fsf-ftp-site-cracked-looking-for-md5-sums)
and that whole debacle seems to have gotten people to notice the FSF was
dead, but instead of _replacing_ them they volunteered at them and tried
to work through their horrible crazy lens, and when that round of
excitement subsided and the FSF went back to dying they tried suing
people to reclaim attention
(https://www.linux.com/news/gpl-requirement-could-have-chilling-effect-derivative-distros)
and that "all we've got to squeeze money out of people is control of the
license" nonsense snowballed into GPLv3.

Not a net win, really.

Anyway, even back at busybox I was mostly targeting equivalents of those
5-year "stable" versions from 1999, and only cherry-picking newer
features when people asked for them or the kernel grew some new thing I
wanted to take advantage of. These days my baseline is posix-2008 plus
any Y2k-era Linux extensions that posix rejected because That's Not
Solaris I Want Solaris La La La Solaris.

So when I say "a proper bash replacement" I'm _mostly_ targeting bash
2.x with proper utf8 and the occasional new feature like ~= (regex
matches) that are easy to do. Stuff like <(command) and file{1,2}.txt
are over 20 years old.

I really don't care what the FSF's done since about 2005. As far as I'm
concerned the Mepis thing I linked above was a career limiting move
_before_ they split "the GPL" into incompatible versions so the Linux
kernel and Samba can't share code even though they implement 2 ends of
the same protocol and are both "GPL".

(None of this excuses dash, though. :)

Rob

P.S. Yes, I did my own GPLv2 enforcement lawsuits circa 2006. That's
another story...
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] making ./configure executable.

[Robert Thompson]

Yeah, I ran into some similar issues years ago...  In my case,  I got
burned into learning not to assume that bash (or any other shell) is
correct, or even necessarily *self* consistent(even on linux), so I
tend to reflexively do differential checks. It's a very small amount
of effort in the normal case, but months of project time and a major
missed deadline tend to leave a bit of a scar :)

there was a "posix, we swear" vendor /bin/sh, a classic /bin/bash, a
"new and sexy" /bin/bash, and the same version/same configuration
/bin/bash running on a different hardware/os...

We wasted months assuming that the newer bash was most likely correct
(it was a heisenbug, both rare and inconsistent). In that case, turned
out that the posix sh was technically correct and consistent, and
*all* of the bashes exhibited *different* incompatible heisenbugs. And
they tended to show up only about a day into a massive
un-checkpointable processing run...

that eval *still* smarts, and worse, it was supposedly-legacy crap
that we were just supposed to "update a bit" (it had always been doing
this, we were just the first people who noticed)...



On 2/5/18, Rob Landley  wrote:
> On 02/05/2018 04:01 PM, Robert Thompson wrote:
>> It seems that the single-equals is POSIX, while the double-equals is a
>> bash extension that at one point bash preferred and extended the match
>> behavior of.
>>
>> But, the bash manpage now documents double-equals  (when in the
>> context of the builtin test or its '[' alias) as equivalent to
>> single-equals.
>
> I researched all this crud at one point, but it was a few years ago now. :)
>
>> The manpage notes that when using test (rather than '['), one should
>> always use single-equals, due to POSIX compatibility issues, in case
>> the script gets run by a shell without a builtin test.
>>
>> Bash has another extended test, '[[', which does significantly
>> different, and expanded, things, but its behavior was inconsistent
>> across bash versions the last time it was relevant to my work.
>>
>> Apparently bash's interpretation of single-equals isn't always exactly
>> POSIX-standard, depending on which bash settings are in effect. A
>> quick skim shows that it may or may not be case-insensitive, may or
>> may not be glob-active (older version), *is* glob-active (newer
>> versions, with caveats?)...
>>
>> dash's behavior isn't saying "unknown operator '['". It's saying
>> "builtin-test-via-left-square-bracket: unknown operator '=='". I think
>> it's just phrasing it really poorly due to overly simplistic
>> backtracking in its parser (most recent symbol being assumed to be the
>> cause of the error, thus printed).
>
> Dash is crap. The switch to dash was because Ubuntu wanted to run its
> init scripts faster and thought changing all the init scripts to start
> with #!/bin/dash was too intrusive a change. No really:
>
>   https://wiki.ubuntu.com/DashAsBinSh
>
> This broke (among other things) the Linux kernel build. (Bash was the
> first program Linux ever ran, Linus extended his term program to run
> bash specifically. /bin/sh being bash was the one thing all Linux
> distros agreed on, until Ubuntu did a stupid.)
>
> Then, of course, the redirect didn't speed init up enough so they went
> down the parallelizing route and created "upstart", but didn't back out
> the /bin/sh change because that would be admitting they made a mistake.
> (They did relent enough to change the default so user accounts would
> have bash as their default shell in /etc/passwd.)
>
> Yes, it is a pet peeve of mine. :)
>
> Rob
>
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] making ./configure executable.

[Robert Thompson]

It seems that the single-equals is POSIX, while the double-equals is a
bash extension that at one point bash preferred and extended the match
behavior of.

But, the bash manpage now documents double-equals  (when in the
context of the builtin test or its '[' alias) as equivalent to
single-equals.

The manpage notes that when using test (rather than '['), one should
always use single-equals, due to POSIX compatibility issues, in case
the script gets run by a shell without a builtin test.

Bash has another extended test, '[[', which does significantly
different, and expanded, things, but its behavior was inconsistent
across bash versions the last time it was relevant to my work.

Apparently bash's interpretation of single-equals isn't always exactly
POSIX-standard, depending on which bash settings are in effect. A
quick skim shows that it may or may not be case-insensitive, may or
may not be glob-active (older version), *is* glob-active (newer
versions, with caveats?)...

dash's behavior isn't saying "unknown operator '['". It's saying
"builtin-test-via-left-square-bracket: unknown operator '=='". I think
it's just phrasing it really poorly due to overly simplistic
backtracking in its parser (most recent symbol being assumed to be the
cause of the error, thus printed).



On 2/5/18, Ryan Prichard  wrote:
> FWIW: I think the problem is the double-equals operator:
>
> $ /bin/dash -c 'if [ x = x ]; then echo foo; fi'
> foo
>
> $ /bin/dash -c 'if [ x == x ]; then echo foo; fi'
> /bin/dash: 1: [: x: unexpected operator
>
> $ /bin/dash -c 'if /usr/bin/[ x == x ]; then echo foo; fi'
> foo
>
> POSIX hasn't specified ==. I'm guessing it's strictly equivalent to =?
>
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html
>
> -Ryan
>
>
> On Sun, Feb 4, 2018 at 6:11 PM, Rob Landley  wrote:
>
>> I tried adding this to configure:
>>
>> if [ "$(basename "$0")" == configure ]
>> then
>>   echo "This file is just a persistent place to set environment
>> variables."
>>   echo "You probably want to run 'make defconfig'."
>>   echo "Type 'make help' for more options, or see the README."
>>   exit 1
>> fi
>>
>> But if your #!/bin/sh points to the Defective Annoying SHell it
>> goes "unexpected operator [" EVEN THOUGH "[" IS IN THE $PATH,
>> and I'm just not going there.
>>
>> So I've put #!/bin/bash at the top, set the executable bit, and
>> I'm keeping the complaint to one line and running "make defconfig"
>> anyway on the theory making it easier to use is better than trying
>> to educating people with a pop-up window error.
>>
>> Rob
>> ___
>> Toybox mailing list
>> Toybox@lists.landley.net
>> http://lists.landley.net/listinfo.cgi/toybox-landley.net
>>
>
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] making ./configure executable.

[Ryan Prichard]

FWIW: I think the problem is the double-equals operator:

$ /bin/dash -c 'if [ x = x ]; then echo foo; fi'
foo

$ /bin/dash -c 'if [ x == x ]; then echo foo; fi'
/bin/dash: 1: [: x: unexpected operator

$ /bin/dash -c 'if /usr/bin/[ x == x ]; then echo foo; fi'
foo

POSIX hasn't specified ==. I'm guessing it's strictly equivalent to =?

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html

-Ryan


On Sun, Feb 4, 2018 at 6:11 PM, Rob Landley  wrote:

> I tried adding this to configure:
>
> if [ "$(basename "$0")" == configure ]
> then
>   echo "This file is just a persistent place to set environment variables."
>   echo "You probably want to run 'make defconfig'."
>   echo "Type 'make help' for more options, or see the README."
>   exit 1
> fi
>
> But if your #!/bin/sh points to the Defective Annoying SHell it
> goes "unexpected operator [" EVEN THOUGH "[" IS IN THE $PATH,
> and I'm just not going there.
>
> So I've put #!/bin/bash at the top, set the executable bit, and
> I'm keeping the complaint to one line and running "make defconfig"
> anyway on the theory making it easier to use is better than trying
> to educating people with a pop-up window error.
>
> Rob
> ___
> Toybox mailing list
> Toybox@lists.landley.net
> http://lists.landley.net/listinfo.cgi/toybox-landley.net
>
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net