Re: [Toybox] Test design issue.

[Rob Landley]

On 04/26/2017 04:56 PM, enh wrote:
> On Wed, Apr 26, 2017 at 2:41 PM, Rob Landley  > wrote:
> 
> On 04/25/2017 12:28 PM, enh wrote:
> > note that the existing SKIP_HOST mechanism is very error prone. iirc
> > none of the sed tests actually run because one near the top needs
> > SKIP_HOST and it never gets unset.
> 
> I thought that "VARIABLE=value command" worked the same for shell
> functions as it does for external commands? Hmmm...
> 
> $ make test_sed > one.txt
> $ TEST_HOST=1 make test_sed > two.txt
> $ diff -u one.txt two.txt | grep '^[+-]'
> ...
> -PASS: sed sed - - twice
> -PASS: sed match \n literal n
> -PASS: sed c empty continuation
> -PASS: sed 'y/a\bc/de\f/'
> +FAIL: sed 'y/a\bc/de\f/'
> -PASS: sed skip start of range
> +FAIL: sed skip start of range
> 
> The first three are the ones we skip, the second the host version is
> failing but the target version isn't so possibly they should be
> annotated too. (Or at least have a comment...)
> 
> What shell are you using to run this? (Not bash?)
> 
> hmm, maybe this got fixed since i last saw it?

Not that I recall, and commit log of sed.test isn't saying so either.

 with TEST_HOST=1 make
> test_sed i'm not seeing any "SKIP" output, whereas when i ran in to this
> i saw a whole wall of yellow "SKIP" lines.

Hmmm... I just tried "BLAH=42 echo hello; echo $BLAH" on the android M
shell on my phone and it didn't persistently set it, but I don't have an
O test environment lying around... (I really need to work out how to
build individual AOSP packages outside of AOSP. I'd like a static
version of mksh and such to compare things against. And a bionic
toolchain I can build toybox against on an ubuntu host...)

I don't recall fixing this here, and am not seeing anything obvious in
the logs, but that doesn't mean much. Remember my recent "I should teach
tests to ignore whitespace" and fixing ls -m output to space
differently? A year ago I did:

commit fcbf55ad565861e22e1cb297e6954b9d82f5094b
Author: Rob Landley 
Date:   Thu Jun 30 10:39:23 2016 -0500

Add NOSPACE=1 to allow tests to pass with diff -b

Which is in news.html but not code.html. I should add a section
documenting the test suite...

One of the reasons there's a whole lotta "squirrel!" in my development
process is when I get a reproducible bug I tend to _stop_ and chase that
sucker. Reproducible bugs are gold, and tend to submerge again if you
let the environment they reproduce in drift. Submerge is NOT the same as
get fixed, it's like roaches. You see it, you kill it, then you spray
for it. Do not let it breed. So fixing reproducible bugs can take
priority of implementing features, even halfway through...

> > SKIP_HOST is also not very intention-revealing. i'd prefer to explicitly
> > distinguish "host tool is just plain broken" from "we deliberately chose
> > to diverge [because ...]".
> 
> Which is why I added comments before them, ala:
> 
> # This segfaults ubuntu 12.04's sed. No really.
> SKIP_HOST=1 testing 'sed - - twice' 'sed "" - -' "hello\n" "" "hello\n"
> 
> But I see I haven't kept up with that. Blah. (Presumably git annotate
> should give a commit comment, and/or let me know what date to look for a
> mailing list entry or blog post. I'll throw it on the todo list.)
> 
> exactly: having the skip mechanism require a string would help keep us
> honest.

In theory you could do SKIP_HOST="this is why" now, it only cares that
it's not an empty string. It'd be a bit _more_ error prone though
because you'd have to do:

  SKIP_HOST="Tea not hot enough to drive atomic vector plotter" \
  testing "improbability drive" "whale" "petunias" "" ""

And if you miss the \ at the end of the line you get the trailing
SKIP_HOST again. (That said, you'd presumably _notice_ that rather a
lot. But only if you remember to TEST_HOST as well as target. :)

If I get the test suite in a reasonable shape, I can add a line to my
release.txt checklist, which is currently:

news.html - release notes
Update version in main.c
  - git commit main.c www/news.html
tag repo
  - git tag $VER
  - git push --tags
source tarball
  git archive --prefix=toybox-$VER/ $VER | gzip -9 > toybox-$VER.tar.gz
  scp toybox-$VER.tar.gz landley.net:
  scp www/news.html landley.net:
binaries
  cd ../clean
  git pull ../toy3
  ../relbuild.sh
update help.html
  ./toybox help -ah > www/help.html
  scp www/help.html landley.net:landley.net/toybox/
scripts/mkstatus.py -> status.html
  - scp www/status.gen landley.net:landley.net/toybox/
roadmap.html
code.html
Inform mailing list

This _used_ to have "build linux from scratch under aboriginal linux" on
all targets, but aboriginal fell by the wayside because
(http://lists.landley.net/pipermail/aboriginal-landley.net/2016-May/002567.html)
and I'm finally doing mkroot to replace it now...

(You may have noticed 

Re: [Toybox] Test design issue.

[enh]

On Wed, Apr 26, 2017 at 2:41 PM, Rob Landley  wrote:

> On 04/25/2017 12:28 PM, enh wrote:
> > note that the existing SKIP_HOST mechanism is very error prone. iirc
> > none of the sed tests actually run because one near the top needs
> > SKIP_HOST and it never gets unset.
>
> I thought that "VARIABLE=value command" worked the same for shell
> functions as it does for external commands? Hmmm...
>
> $ make test_sed > one.txt
> $ TEST_HOST=1 make test_sed > two.txt
> $ diff -u one.txt two.txt | grep '^[+-]'
> ...
> -PASS: sed sed - - twice
> -PASS: sed match \n literal n
> -PASS: sed c empty continuation
> -PASS: sed 'y/a\bc/de\f/'
> +FAIL: sed 'y/a\bc/de\f/'
> -PASS: sed skip start of range
> +FAIL: sed skip start of range
>
> The first three are the ones we skip, the second the host version is
> failing but the target version isn't so possibly they should be
> annotated too. (Or at least have a comment...)
>
> What shell are you using to run this? (Not bash?)


hmm, maybe this got fixed since i last saw it? with TEST_HOST=1 make
test_sed i'm not seeing any "SKIP" output, whereas when i ran in to this i
saw a whole wall of yellow "SKIP" lines.


>
> > SKIP_HOST is also not very intention-revealing. i'd prefer to explicitly
> > distinguish "host tool is just plain broken" from "we deliberately chose
> > to diverge [because ...]".
>
> Which is why I added comments before them, ala:
>
> # This segfaults ubuntu 12.04's sed. No really.
> SKIP_HOST=1 testing 'sed - - twice' 'sed "" - -' "hello\n" "" "hello\n"
>
> But I see I haven't kept up with that. Blah. (Presumably git annotate
> should give a commit comment, and/or let me know what date to look for a
> mailing list entry or blog post. I'll throw it on the todo list.)


exactly: having the skip mechanism require a string would help keep us
honest.


>
> > cases like sh.test are awkward too...
>
> Yeah shellit() tries to compensate a bit but I haven't opened the toysh
> can of worms yet. (Tim Bird asked me what it would take to do toysh, and
> my estimate was 6 months full-time devoted to it, but he hasn't got a
> budget to buy my time from my employer. Meanwhile $DAYJOB is focused on
> winning solar/wind grid integration monitoring contracts. The market
> opportunity for that is likely to explode in about 3 years and we need
> to deploy stuff people can buy before then and be ready to scale up, so
> getting any time to spend on toybox takes a bit of pushing...)
>
> On the bright side I got https://github.com/landley/mkroot a bit more
> fluffed out this week. Once I get all the musl-cross-make targets
> booting, I can coordinate with Rich to get him to tag and ship an actual
> release with binaries you can download (maybe he'll even put the
> mcm-buildall.sh script in _his_ repo instead of mine).
>
> Building kernels with mkroot needs toybox commit 9be140ebb29f so I'll
> have to cut a new release here to coincide with the mkroot release, but
> it's coming up on time for that anyway...
>
> > exponentiation is a bash extension
> > to the more widespread arithmetic extension (which is all the test
> > tests).
>
> Oh toysh is going to need quite possibly a thousand lines of tests. The
> tests I have _now_ are just ones that I had lying around because I was
> poking at something that was doing this...
>

mksh has a thousand line perl script to run 13k lines of tests that look
like

name: xtrace-2
description:
Check that "set -x" is off during PS4 expansion
stdin:
f() {
print -n "(f1:$-)"
set -x
print -n "(f2:$-)"
}
PS4='[(p:$-)$(f)] '
print "(o0:$-)"
set -x -o inherit-xtrace
print "(o1:$-)"
set +x
print "(o2:$-)"
expected-stdout:
(o0:sh)
(o1:shx)
(o2:sh)
expected-stderr:
[(p:sh)(f1:sh)(f2:sh)] print '(o1:shx)'
[(p:sh)(f1:sh)(f2:sh)] set +x



> I thought it was bc/dc but:
>
>   $ dc "1+2"
>   dc: Could not open file 1+2
>   $ bc "1+2"
>   File 1+2 is unavailable.
>
> Those are both _way_ too stupid to use as a quick and dirty calculator
> from the command line. (I usually do echo $((1+2)) when I don't need
> floating point, and fire up python when I do. But there should be a
> better way, possibly I should have toysh $(()) support floating point?
> Hmmm...)
>
> > so although it's "sh.test", it explicitly invokes "bash".
>
> Because toysh should provide the bash alias along with bash functionality.
>
> Ubuntu's epically stupid https://wiki.ubuntu.com/DashAsBinSh broke the
> #!/bin/sh symlink to point to a broken shell (seriously, at the time it
> segfaulted and got signal handling wrong and...), so scripts explicitly
> say #!/bin/bash instead. I intend to provide a reasonable bash
> replacement that runs those scripts.
>
> Bash was the first program linux ever ran (and running bash was the
> reason Linus implemented system calls in his term program), so it was
> the standard shell of Linux 

Re: [Toybox] Test design issue.

[Rob Landley]

On 04/25/2017 12:28 PM, enh wrote:
> note that the existing SKIP_HOST mechanism is very error prone. iirc
> none of the sed tests actually run because one near the top needs
> SKIP_HOST and it never gets unset.

I thought that "VARIABLE=value command" worked the same for shell
functions as it does for external commands? Hmmm...

$ make test_sed > one.txt
$ TEST_HOST=1 make test_sed > two.txt
$ diff -u one.txt two.txt | grep '^[+-]'
...
-PASS: sed sed - - twice
-PASS: sed match \n literal n
-PASS: sed c empty continuation
-PASS: sed 'y/a\bc/de\f/'
+FAIL: sed 'y/a\bc/de\f/'
-PASS: sed skip start of range
+FAIL: sed skip start of range

The first three are the ones we skip, the second the host version is
failing but the target version isn't so possibly they should be
annotated too. (Or at least have a comment...)

What shell are you using to run this? (Not bash?)

> SKIP_HOST is also not very intention-revealing. i'd prefer to explicitly
> distinguish "host tool is just plain broken" from "we deliberately chose
> to diverge [because ...]".

Which is why I added comments before them, ala:

# This segfaults ubuntu 12.04's sed. No really.
SKIP_HOST=1 testing 'sed - - twice' 'sed "" - -' "hello\n" "" "hello\n"

But I see I haven't kept up with that. Blah. (Presumably git annotate
should give a commit comment, and/or let me know what date to look for a
mailing list entry or blog post. I'll throw it on the todo list.)

> cases like sh.test are awkward too...

Yeah shellit() tries to compensate a bit but I haven't opened the toysh
can of worms yet. (Tim Bird asked me what it would take to do toysh, and
my estimate was 6 months full-time devoted to it, but he hasn't got a
budget to buy my time from my employer. Meanwhile $DAYJOB is focused on
winning solar/wind grid integration monitoring contracts. The market
opportunity for that is likely to explode in about 3 years and we need
to deploy stuff people can buy before then and be ready to scale up, so
getting any time to spend on toybox takes a bit of pushing...)

On the bright side I got https://github.com/landley/mkroot a bit more
fluffed out this week. Once I get all the musl-cross-make targets
booting, I can coordinate with Rich to get him to tag and ship an actual
release with binaries you can download (maybe he'll even put the
mcm-buildall.sh script in _his_ repo instead of mine).

Building kernels with mkroot needs toybox commit 9be140ebb29f so I'll
have to cut a new release here to coincide with the mkroot release, but
it's coming up on time for that anyway...

> exponentiation is a bash extension
> to the more widespread arithmetic extension (which is all the test
> tests).

Oh toysh is going to need quite possibly a thousand lines of tests. The
tests I have _now_ are just ones that I had lying around because I was
poking at something that was doing this...

I thought it was bc/dc but:

  $ dc "1+2"
  dc: Could not open file 1+2
  $ bc "1+2"
  File 1+2 is unavailable.

Those are both _way_ too stupid to use as a quick and dirty calculator
from the command line. (I usually do echo $((1+2)) when I don't need
floating point, and fire up python when I do. But there should be a
better way, possibly I should have toysh $(()) support floating point?
Hmmm...)

> so although it's "sh.test", it explicitly invokes "bash".

Because toysh should provide the bash alias along with bash functionality.

Ubuntu's epically stupid https://wiki.ubuntu.com/DashAsBinSh broke the
#!/bin/sh symlink to point to a broken shell (seriously, at the time it
segfaulted and got signal handling wrong and...), so scripts explicitly
say #!/bin/bash instead. I intend to provide a reasonable bash
replacement that runs those scripts.

Bash was the first program linux ever ran (and running bash was the
reason Linus implemented system calls in his term program), so it was
the standard shell of Linux from 1991 to 2006. Ubuntu's stated reason
for changing this (speeding up init scripts) didn't work (they then
implemented upstart to get parallelism), but they never admitted the
mistake and reverted it. They said that changing the #! at the start of
each init script was too intrusive a change, so they broke the kernel
build instead.

That's why all my scripts say #!/bin/bash and not #!/bin/sh. I need to
replace _that_.

> and if you "fix" it by running sh, bash-as-sh fails too. for now i just have 
> an
> explicit case to skip sh.test in my Android toybox test runner, but in
> an ideal world it would be more fine-grained than that.

My infrastructure checks .config and should skip tests for commands that
are disabled?

Alas, I don't have a "pending" for the tests directory. In an ideal
world I'd be making progress fast enough this wouldn't be an issue. In
practice we're coming up on the one year anniversary of $DAYJOB's
"funding knothole" where we've been in perpetual crisis mode ever since.
(It's compelling technology I really want to see ship, but the pay sucks
and it's eating my life. Doing what I 

Re: [Toybox] Test design issue.

[enh]

note that the existing SKIP_HOST mechanism is very error prone. iirc none
of the sed tests actually run because one near the top needs SKIP_HOST and
it never gets unset.

SKIP_HOST is also not very intention-revealing. i'd prefer to explicitly
distinguish "host tool is just plain broken" from "we deliberately chose to
diverge [because ...]".

cases like sh.test are awkward too... exponentiation is a bash extension to
the more widespread arithmetic extension (which is all the test tests). so
although it's "sh.test", it explicitly invokes "bash". and if you "fix" it
by running sh, bash-as-sh fails too. for now i just have an explicit case
to skip sh.test in my Android toybox test runner, but in an ideal world it
would be more fine-grained than that.

here though, is there a strong reason to prefer \e? otherwise why not just
do what the host does in this case?

On Mon, Apr 24, 2017 at 3:18 PM, Rob Landley  wrote:

> I want an ls.test checking "ls -b" via some variant of:
>
> mkdir blah
> touch blah/"$(echo -e "$(X=0;while [ $X -lt 255 ];do X=$(($X+1));[ $X
> -eq 47 ]&& continue;printf '\\x%02x' $X; done)")"
> ls -b blah
>
> And then make sure it's escaping everything it should, but the ubuntu
> 14.04 ls -b  and toybox's ls -b have one difference: for ascii 27 (the
> escape char) ubuntu says "\033" and toybox says "\e".
>
> I guess I can break it into 3 tests with the middle one SKIP_HOST? Seems
> awkward...
>
> Rob
> ___
> Toybox mailing list
> Toybox@lists.landley.net
> http://lists.landley.net/listinfo.cgi/toybox-landley.net
>



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net