Re: [Toybox] [PATCH] Add the gzip/gunzip/zcat tests I wrote for toolbox gzip/gunzip/zcat.

2017-04-26 Thread enh 
On Wed, Apr 26, 2017 at 2:43 PM, Rob Landley  wrote:

> On 04/24/2017 03:43 PM, enh wrote:
> > Bringing the zlib-based gzip/gunzip/zcat over to toybox is a problem for
> > another day,
>
> Blah, I should take this one.
>
> I have deflate decompression side implemented already (and working at
> one point, although I just tested "make zcat" and got a bad crc
> decompressing the musl-1.1.16.gz tarball). I have the first 1/3 or so of
> compression side implemented and somewhere around here is a version
> that's about 2/3 through. I stopped at "where do dictionary resets
> happen" and basically go the answer "nobody seems to agree, just do it
> every 250k". (Which means my gzip and other gzip won't necessarily
> produce the same binary, but it should extract the same. There's
> actually a format hitch you can do sticking in gratuitous resets to
> allow parallel decompression, it's a bit like mp4 streaming mode
> sticking in extra keyframes, only noticeably cheaper. :)
>
> Adding zlib mode to that is basically the same plumbing as
> toys/*/md5sum.c does for libcrypto.
>
> Lemme try to find a chunk of time to devote to that...


if you're actually going to start to look, i'll attach my port of the
current toolbox implementation.

---
 Config.in   |   6 ++
 scripts/make.sh |   2 +-
 toys/pending/compress.c |  93 --
 toys/pending/gzip.c | 199

 4 files changed, 206 insertions(+), 94 deletions(-)
 create mode 100644 toys/pending/gzip.c



>
> > but at least the tests are easy...
> >
> > (These tests pass with TEST_HOST and on the toolbox versions, but the
> > toybox toys are in pending and very broken.)
>
> The problem is this is just testing that gzip and gunzip reverse each
> other. If they share infrastructure and screw something up the same way,
> what have you proved? Also you can't test gunzip without gzip being
> present.
>
> For bzip I added a tests/files/bzcat (and already had a lot of
> tests/files/blkid/*.bz2). I'd like to have at least a couple canned .gz
> files for reference under tests/files.
>
> I'll try to take a closer look at this later...
>
> Rob
>



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
From ab661916a64d67d9b07591c2edb20be21b2e7095 Mon Sep 17 00:00:00 2001
From: Elliott Hughes 
Date: Wed, 26 Apr 2017 15:01:41 -0700
Subject: [PATCH] Implement zlib-based gzip/gunzip/zcat.

---
 Config.in   |   6 ++
 scripts/make.sh |   2 +-
 toys/pending/compress.c |  93 --
 toys/pending/gzip.c | 199 
 4 files changed, 206 insertions(+), 94 deletions(-)
 create mode 100644 toys/pending/gzip.c

diff --git a/Config.in b/Config.in
index 2bfe30a..0524ce3 100644
--- a/Config.in
+++ b/Config.in
@@ -77,6 +77,12 @@ config TOYBOX_LIBCRYPTO
help
  Use faster hash functions out of exteral -lcrypto library.
 
+config TOYBOX_LIBZ
+   bool "Use libz (zlib)"
+   default n
+   help
+ Use libz for gz support.
+
 config TOYBOX_FLOAT
 	bool "Floating point support"
 	default y
diff --git a/scripts/make.sh b/scripts/make.sh
index 5792847..2d33be4 100755
--- a/scripts/make.sh
+++ b/scripts/make.sh
@@ -109,7 +109,7 @@ then
   # for it.
 
   > generated/optlibs.dat
-  for i in util crypt m resolv selinux smack attr rt crypto
+  for i in util crypt m resolv selinux smack attr rt crypto z
   do
 echo "int main(int argc, char *argv[]) {return 0;}" | \
 ${CROSS_COMPILE}${CC} $CFLAGS -xc - -o generated/libprobe -Wl,--as-needed -l$i > /dev/null 2>/dev/null &&
diff --git a/toys/pending/compress.c b/toys/pending/compress.c
index 1749ee4..5605a35 100644
--- a/toys/pending/compress.c
+++ b/toys/pending/compress.c
@@ -16,9 +16,6 @@
 // Leave Lrg at end so flag values line up.
 
 USE_COMPRESS(NEWTOY(compress, "zcd9lrg[-cd][!zgLr]", TOYFLAG_USR|TOYFLAG_BIN))
-USE_GZIP(NEWTOY(gzip, USE_GZIP_D("d")"19dcflqStvgLRz[!gLRz]", TOYFLAG_USR|TOYFLAG_BIN))
-USE_ZCAT(NEWTOY(zcat, 0, TOYFLAG_USR|TOYFLAG_BIN))
-USE_GUNZIP(NEWTOY(gunzip, "cflqStv", TOYFLAG_USR|TOYFLAG_BIN))
 
 //zip unzip gzip gunzip zcat
 
@@ -37,40 +34,6 @@ config COMPRESS
 -R	raw
 -z	zip
 
-config GZIP
-  bool "gzip"
-  default y
-  depends on COMPRESS
-  help
-usage: gzip [-19cfqStvzgLR] [FILE...]
-
-Compess (deflate) file(s). With no files, compress stdin to stdout.
-
-On successful decompression, compressed files are replaced with the
-uncompressed version. The input file is removed and replaced with
-a new file without the .gz extension (with same ownership/permissions).
-
--1	Minimal compression (fastest)
--9	Max compression (default)
--c	cat to stdout (act as zcat)
--f	force (if output file exists, input is tty, unrecognized extension)
--q	quiet (no warnings)
--S	specify exension

Re: [Toybox] Test design issue.

2017-04-26 Thread enh 
On Wed, Apr 26, 2017 at 2:41 PM, Rob Landley  wrote:

> On 04/25/2017 12:28 PM, enh wrote:
> > note that the existing SKIP_HOST mechanism is very error prone. iirc
> > none of the sed tests actually run because one near the top needs
> > SKIP_HOST and it never gets unset.
>
> I thought that "VARIABLE=value command" worked the same for shell
> functions as it does for external commands? Hmmm...
>
> $ make test_sed > one.txt
> $ TEST_HOST=1 make test_sed > two.txt
> $ diff -u one.txt two.txt | grep '^[+-]'
> ...
> -PASS: sed sed - - twice
> -PASS: sed match \n literal n
> -PASS: sed c empty continuation
> -PASS: sed 'y/a\bc/de\f/'
> +FAIL: sed 'y/a\bc/de\f/'
> -PASS: sed skip start of range
> +FAIL: sed skip start of range
>
> The first three are the ones we skip, the second the host version is
> failing but the target version isn't so possibly they should be
> annotated too. (Or at least have a comment...)
>
> What shell are you using to run this? (Not bash?)


hmm, maybe this got fixed since i last saw it? with TEST_HOST=1 make
test_sed i'm not seeing any "SKIP" output, whereas when i ran in to this i
saw a whole wall of yellow "SKIP" lines.


>
> > SKIP_HOST is also not very intention-revealing. i'd prefer to explicitly
> > distinguish "host tool is just plain broken" from "we deliberately chose
> > to diverge [because ...]".
>
> Which is why I added comments before them, ala:
>
> # This segfaults ubuntu 12.04's sed. No really.
> SKIP_HOST=1 testing 'sed - - twice' 'sed "" - -' "hello\n" "" "hello\n"
>
> But I see I haven't kept up with that. Blah. (Presumably git annotate
> should give a commit comment, and/or let me know what date to look for a
> mailing list entry or blog post. I'll throw it on the todo list.)


exactly: having the skip mechanism require a string would help keep us
honest.


>
> > cases like sh.test are awkward too...
>
> Yeah shellit() tries to compensate a bit but I haven't opened the toysh
> can of worms yet. (Tim Bird asked me what it would take to do toysh, and
> my estimate was 6 months full-time devoted to it, but he hasn't got a
> budget to buy my time from my employer. Meanwhile $DAYJOB is focused on
> winning solar/wind grid integration monitoring contracts. The market
> opportunity for that is likely to explode in about 3 years and we need
> to deploy stuff people can buy before then and be ready to scale up, so
> getting any time to spend on toybox takes a bit of pushing...)
>
> On the bright side I got https://github.com/landley/mkroot a bit more
> fluffed out this week. Once I get all the musl-cross-make targets
> booting, I can coordinate with Rich to get him to tag and ship an actual
> release with binaries you can download (maybe he'll even put the
> mcm-buildall.sh script in _his_ repo instead of mine).
>
> Building kernels with mkroot needs toybox commit 9be140ebb29f so I'll
> have to cut a new release here to coincide with the mkroot release, but
> it's coming up on time for that anyway...
>
> > exponentiation is a bash extension
> > to the more widespread arithmetic extension (which is all the test
> > tests).
>
> Oh toysh is going to need quite possibly a thousand lines of tests. The
> tests I have _now_ are just ones that I had lying around because I was
> poking at something that was doing this...
>

mksh has a thousand line perl script to run 13k lines of tests that look
like

name: xtrace-2
description:
Check that "set -x" is off during PS4 expansion
stdin:
f() {
print -n "(f1:$-)"
set -x
print -n "(f2:$-)"
}
PS4='[(p:$-)$(f)] '
print "(o0:$-)"
set -x -o inherit-xtrace
print "(o1:$-)"
set +x
print "(o2:$-)"
expected-stdout:
(o0:sh)
(o1:shx)
(o2:sh)
expected-stderr:
[(p:sh)(f1:sh)(f2:sh)] print '(o1:shx)'
[(p:sh)(f1:sh)(f2:sh)] set +x



> I thought it was bc/dc but:
>
>   $ dc "1+2"
>   dc: Could not open file 1+2
>   $ bc "1+2"
>   File 1+2 is unavailable.
>
> Those are both _way_ too stupid to use as a quick and dirty calculator
> from the command line. (I usually do echo $((1+2)) when I don't need
> floating point, and fire up python when I do. But there should be a
> better way, possibly I should have toysh $(()) support floating point?
> Hmmm...)
>
> > so although it's "sh.test", it explicitly invokes "bash".
>
> Because toysh should provide the bash alias along with bash functionality.
>
> Ubuntu's epically stupid https://wiki.ubuntu.com/DashAsBinSh broke the
> #!/bin/sh symlink to point to a broken shell (seriously, at the time it
> segfaulted and got signal handling wrong and...), so scripts explicitly
> say #!/bin/bash instead. I intend to provide a reasonable bash
> replacement that runs those scripts.
>
> Bash was the first program linux ever ran (and running bash was the
> reason Linus implemented system calls in his term program), so it was
> the standard shell of Linux

Re: [Toybox] [PATCH] Add the gzip/gunzip/zcat tests I wrote for toolbox gzip/gunzip/zcat.

2017-04-26 Thread Rob Landley 
On 04/24/2017 03:43 PM, enh wrote:
> Bringing the zlib-based gzip/gunzip/zcat over to toybox is a problem for
> another day,

Blah, I should take this one.

I have deflate decompression side implemented already (and working at
one point, although I just tested "make zcat" and got a bad crc
decompressing the musl-1.1.16.gz tarball). I have the first 1/3 or so of
compression side implemented and somewhere around here is a version
that's about 2/3 through. I stopped at "where do dictionary resets
happen" and basically go the answer "nobody seems to agree, just do it
every 250k". (Which means my gzip and other gzip won't necessarily
produce the same binary, but it should extract the same. There's
actually a format hitch you can do sticking in gratuitous resets to
allow parallel decompression, it's a bit like mp4 streaming mode
sticking in extra keyframes, only noticeably cheaper. :)

Adding zlib mode to that is basically the same plumbing as
toys/*/md5sum.c does for libcrypto.

Lemme try to find a chunk of time to devote to that...

> but at least the tests are easy...
> 
> (These tests pass with TEST_HOST and on the toolbox versions, but the
> toybox toys are in pending and very broken.)

The problem is this is just testing that gzip and gunzip reverse each
other. If they share infrastructure and screw something up the same way,
what have you proved? Also you can't test gunzip without gzip being present.

For bzip I added a tests/files/bzcat (and already had a lot of
tests/files/blkid/*.bz2). I'd like to have at least a couple canned .gz
files for reference under tests/files.

I'll try to take a closer look at this later...

Rob
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] Test design issue.

2017-04-26 Thread Rob Landley 
On 04/25/2017 12:28 PM, enh wrote:
> note that the existing SKIP_HOST mechanism is very error prone. iirc
> none of the sed tests actually run because one near the top needs
> SKIP_HOST and it never gets unset.

I thought that "VARIABLE=value command" worked the same for shell
functions as it does for external commands? Hmmm...

$ make test_sed > one.txt
$ TEST_HOST=1 make test_sed > two.txt
$ diff -u one.txt two.txt | grep '^[+-]'
...
-PASS: sed sed - - twice
-PASS: sed match \n literal n
-PASS: sed c empty continuation
-PASS: sed 'y/a\bc/de\f/'
+FAIL: sed 'y/a\bc/de\f/'
-PASS: sed skip start of range
+FAIL: sed skip start of range

The first three are the ones we skip, the second the host version is
failing but the target version isn't so possibly they should be
annotated too. (Or at least have a comment...)

What shell are you using to run this? (Not bash?)

> SKIP_HOST is also not very intention-revealing. i'd prefer to explicitly
> distinguish "host tool is just plain broken" from "we deliberately chose
> to diverge [because ...]".

Which is why I added comments before them, ala:

# This segfaults ubuntu 12.04's sed. No really.
SKIP_HOST=1 testing 'sed - - twice' 'sed "" - -' "hello\n" "" "hello\n"

But I see I haven't kept up with that. Blah. (Presumably git annotate
should give a commit comment, and/or let me know what date to look for a
mailing list entry or blog post. I'll throw it on the todo list.)

> cases like sh.test are awkward too...

Yeah shellit() tries to compensate a bit but I haven't opened the toysh
can of worms yet. (Tim Bird asked me what it would take to do toysh, and
my estimate was 6 months full-time devoted to it, but he hasn't got a
budget to buy my time from my employer. Meanwhile $DAYJOB is focused on
winning solar/wind grid integration monitoring contracts. The market
opportunity for that is likely to explode in about 3 years and we need
to deploy stuff people can buy before then and be ready to scale up, so
getting any time to spend on toybox takes a bit of pushing...)

On the bright side I got https://github.com/landley/mkroot a bit more
fluffed out this week. Once I get all the musl-cross-make targets
booting, I can coordinate with Rich to get him to tag and ship an actual
release with binaries you can download (maybe he'll even put the
mcm-buildall.sh script in _his_ repo instead of mine).

Building kernels with mkroot needs toybox commit 9be140ebb29f so I'll
have to cut a new release here to coincide with the mkroot release, but
it's coming up on time for that anyway...

> exponentiation is a bash extension
> to the more widespread arithmetic extension (which is all the test
> tests).

Oh toysh is going to need quite possibly a thousand lines of tests. The
tests I have _now_ are just ones that I had lying around because I was
poking at something that was doing this...

I thought it was bc/dc but:

  $ dc "1+2"
  dc: Could not open file 1+2
  $ bc "1+2"
  File 1+2 is unavailable.

Those are both _way_ too stupid to use as a quick and dirty calculator
from the command line. (I usually do echo $((1+2)) when I don't need
floating point, and fire up python when I do. But there should be a
better way, possibly I should have toysh $(()) support floating point?
Hmmm...)

> so although it's "sh.test", it explicitly invokes "bash".

Because toysh should provide the bash alias along with bash functionality.

Ubuntu's epically stupid https://wiki.ubuntu.com/DashAsBinSh broke the
#!/bin/sh symlink to point to a broken shell (seriously, at the time it
segfaulted and got signal handling wrong and...), so scripts explicitly
say #!/bin/bash instead. I intend to provide a reasonable bash
replacement that runs those scripts.

Bash was the first program linux ever ran (and running bash was the
reason Linus implemented system calls in his term program), so it was
the standard shell of Linux from 1991 to 2006. Ubuntu's stated reason
for changing this (speeding up init scripts) didn't work (they then
implemented upstart to get parallelism), but they never admitted the
mistake and reverted it. They said that changing the #! at the start of
each init script was too intrusive a change, so they broke the kernel
build instead.

That's why all my scripts say #!/bin/bash and not #!/bin/sh. I need to
replace _that_.

> and if you "fix" it by running sh, bash-as-sh fails too. for now i just have 
> an
> explicit case to skip sh.test in my Android toybox test runner, but in
> an ideal world it would be more fine-grained than that.

My infrastructure checks .config and should skip tests for commands that
are disabled?

Alas, I don't have a "pending" for the tests directory. In an ideal
world I'd be making progress fast enough this wouldn't be an issue. In
practice we're coming up on the one year anniversary of $DAYJOB's
"funding knothole" where we've been in perpetual crisis mode ever since.
(It's compelling technology I really want to see ship, but the pay sucks
and it's eating my life. Doing what I