On 8/1/23 20:01, enh wrote:
> On Tue, Aug 1, 2023 at 5:56 PM enh <e...@google.com> wrote:
>> yes, with the obvious line added to run-tests-on-android.sh, all the
>> tests pass on my hwasan build (and the sed test only takes a couple of
>> seconds). (for reference, my linux/x86-64 hardware that timed out was
>> a work amd threadripper box, not my personal 10 year old laptop!)
>
> hmm... missed one failure. my idea of what $TESTDIR is meant to be and> yours
> obviously not exactly the same...
TESTDIR=generated/testdir is where the $PATH is, not generated/testdir/testdir
where the tests run.
From scripts/test.sh:
export FILES="$PWD"/tests/files PREFIX=generated/testdir
rm -rf "$PREFIX"
mkdir -p "$PREFIX"/testdir
...
export -n PREFIX
cd "$PREFIX"
PATH="$PWD:$PATH" TESTDIR="$PWD"
I.E:
PREFIX=generated/testdir
cd "$PREFIX"
TESTDIR="$PWD"
> -- ls
> FAIL: ls no argument
> echo -ne '' | "/system/bin/ls"
> --- expected 2023-08-02 00:40:11.231545936 +0000
> +++ actual 2023-08-02 00:40:11.327545936 +0000
> @@ -1,4 +1,6 @@
> +actual
> dir1
> dir2
> +expected
> file1.txt
> file2.txt
That was commit 4acaebea5a08 that started me down this path. Well, the current
branch of it. Ok, backstory:
1) In the beginning there was generated/testdir which gets populated with test
files. When testing toybox, that directory gets added to the start of your $PATH
and the commands being tested are installed/symlinked into there. The tests
themselves originally ran in there too.
2) Running tests in the directory with all the commands BEING tested is awkward:
you're adding random crap to your $PATH, it's easy to delete or collide with
something you shouldn't, trash accumulates between tests if you're not careful
because you can't just delete everything in the directory... so I made a second
"testdir" subdirectory under that (generated/testdir/testdir) that gets deleted
and recreated between each command being tested.
So: generated/testdir = first thing in $PATH so it finds test binaries
generated/testdir/testdir = $PWD when a test is running
3) The test plumbing has to source each tests/command.test file instead of
running them as a child process, so they can inherit the testcmd shell functions
and such. But that means leftover environment variables or shell functions
defined by commands accumulate... So put parentheses around the test call so it
runs in a subshell, and doesn't contaminate the host shell's context. (Making
this work right so ctrl-c would reliably stop the tests took MANY ATTEMPTS, I
forget why. In theory you can export shell functions in bash, but in practice it
didn't work reliably.)
So now the filesystem AND shell environment cleaned up after themselves, and I
could yank a lot of "rm" and "unset" at the end of tests...
4) Now that we start in a clean directory with a clean environment... I still
couldn't "tar x test.tar && ls | sort" because there were other files in the
directory, namely "expected" and "actual". The first is testing argument $3 with
the expected output, and the the second is the captured stdout, at the end the
test diffs the two together to see if it got the right output. So they're
needed, but they're not needed THERE.
So I moved them to ../expected and ../actual in commit 4acaebea5a08 so the
directory started _empty_. Except for the file "input" which is only present
when "testing" argument 4 isn't blank:
[ ! -z "$4" ] && echo -ne "$4" > input || rm -f input
This means I can tell ls or tar or zip to just use the current directory
contents without worrying about extraneous files. (I'd have to put a sort on
either the input or the output because filesystem order varies with hashing, but
that was already the case.)
5) A test can cd to a new directory, and if it doesn't cd _back_ then relative
paths won't be able to find expected/actual afterwards. In fact if a group of
tests cd into a subdirectory, then expected/actual get created in the otherwise
"clean" second testdir directory. And the one that really broke stuff was a test
that did:
ln -s name .; cd name
Which added "name" to $PWD for the shell, but as far as the filesystem was
concerned .. was 2 levels up from there because that traverses actual the mount
tree not "how we got here", so "cd .." took the "name" off but did NOT put us in
the directory containing expected/actual and the script broke (silently! Because
I wasn't redirecting stderr into $DIFF yet, so diff producing no output to
stdout was taken as their being no differences and the test succeeded. Commit
03e1cc1e45b6 added the 2>&1 to catch that in future.)
So I used an absolute path sticking expected/actual them in generated/testdir
along with all those $PATH entries, which isn't ideal but that's where the ..
was putting them before, and I had a variable already set to point at that.
"And that is the story of the first thanksgiving." - Lazerpig
Rob
P.S. When I say I could easily spend years JUST ON THE TEST SUITE, this is not
hyperbole...
P.P.S. Lazerpig is... Imagine if Sheogorath, Daedric Prince of Madness from
Skyrim, got very drunk and started making youtube videos about military history
and the Ukraine conflict using a photo of a large pig as his avatar, with titles
like "The A10 sucks and I can prove it mathematically", "The T-34 is not as good
as you think it is", "Shut up about the F-35", and "Something Something Tiger
Tank". For a while his outros were him singing things like "It's raining men"
and the Spongebob Squarepants song very badly, but youtube starting copyright
claiming his videos. Anyway, that signoff was at the end of
https://www.youtube.com/watch?v=FI0KUZiZp1Q if you were wondering. It doesn't
make any more sense in context.
_______________________________________________
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net
