Re: [Toybox] [PATCH] Add the gzip/gunzip/zcat tests I wrote for toolbox gzip/gunzip/zcat.

2017-05-22 Thread enh 
On Tue, May 9, 2017 at 9:02 AM, enh  wrote:
> On Wed, Apr 26, 2017 at 2:43 PM, Rob Landley  wrote:
>> On 04/24/2017 03:43 PM, enh wrote:
>>> Bringing the zlib-based gzip/gunzip/zcat over to toybox is a problem for
>>> another day,
>>
>> Blah, I should take this one.
>>
>> I have deflate decompression side implemented already (and working at
>> one point, although I just tested "make zcat" and got a bad crc
>> decompressing the musl-1.1.16.gz tarball). I have the first 1/3 or so of
>> compression side implemented and somewhere around here is a version
>> that's about 2/3 through. I stopped at "where do dictionary resets
>> happen" and basically go the answer "nobody seems to agree, just do it
>> every 250k". (Which means my gzip and other gzip won't necessarily
>> produce the same binary, but it should extract the same. There's
>> actually a format hitch you can do sticking in gratuitous resets to
>> allow parallel decompression, it's a bit like mp4 streaming mode
>> sticking in extra keyframes, only noticeably cheaper. :)
>>
>> Adding zlib mode to that is basically the same plumbing as
>> toys/*/md5sum.c does for libcrypto.
>>
>> Lemme try to find a chunk of time to devote to that...
>>
>>> but at least the tests are easy...
>>>
>>> (These tests pass with TEST_HOST and on the toolbox versions, but the
>>> toybox toys are in pending and very broken.)
>>
>> The problem is this is just testing that gzip and gunzip reverse each
>> other. If they share infrastructure and screw something up the same way,
>> what have you proved? Also you can't test gunzip without gzip being present.
>
> yeah, despite being the person who's trying to run the toybox tests on
> a toybox-only system, i relied on testing on the host.
>
> commit these as better than nothing and i'll follow up with a patch
> adding pre-canned inputs for each of them, so each test only needs one
> tool?

ping? (and next question too for the follow-on patch.)

> when i do create the pre-canned inputs, do you have any preference
> about which tool (toybox/busybox/gnu) generates them?
>
>> For bzip I added a tests/files/bzcat (and already had a lot of
>> tests/files/blkid/*.bz2). I'd like to have at least a couple canned .gz
>> files for reference under tests/files.
>>
>> I'll try to take a closer look at this later...
>>
>> Rob
>
>
>
> --
> Elliott Hughes - http://who/enh - http://jessies.org/~enh/
> Android native code/tools questions? Mail me/drop by/add me as a reviewer.



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] [PATCH] Add the gzip/gunzip/zcat tests I wrote for toolbox gzip/gunzip/zcat.

2017-05-09 Thread enh 
On Wed, Apr 26, 2017 at 2:43 PM, Rob Landley  wrote:
> On 04/24/2017 03:43 PM, enh wrote:
>> Bringing the zlib-based gzip/gunzip/zcat over to toybox is a problem for
>> another day,
>
> Blah, I should take this one.
>
> I have deflate decompression side implemented already (and working at
> one point, although I just tested "make zcat" and got a bad crc
> decompressing the musl-1.1.16.gz tarball). I have the first 1/3 or so of
> compression side implemented and somewhere around here is a version
> that's about 2/3 through. I stopped at "where do dictionary resets
> happen" and basically go the answer "nobody seems to agree, just do it
> every 250k". (Which means my gzip and other gzip won't necessarily
> produce the same binary, but it should extract the same. There's
> actually a format hitch you can do sticking in gratuitous resets to
> allow parallel decompression, it's a bit like mp4 streaming mode
> sticking in extra keyframes, only noticeably cheaper. :)
>
> Adding zlib mode to that is basically the same plumbing as
> toys/*/md5sum.c does for libcrypto.
>
> Lemme try to find a chunk of time to devote to that...
>
>> but at least the tests are easy...
>>
>> (These tests pass with TEST_HOST and on the toolbox versions, but the
>> toybox toys are in pending and very broken.)
>
> The problem is this is just testing that gzip and gunzip reverse each
> other. If they share infrastructure and screw something up the same way,
> what have you proved? Also you can't test gunzip without gzip being present.

yeah, despite being the person who's trying to run the toybox tests on
a toybox-only system, i relied on testing on the host.

commit these as better than nothing and i'll follow up with a patch
adding pre-canned inputs for each of them, so each test only needs one
tool?

when i do create the pre-canned inputs, do you have any preference
about which tool (toybox/busybox/gnu) generates them?

> For bzip I added a tests/files/bzcat (and already had a lot of
> tests/files/blkid/*.bz2). I'd like to have at least a couple canned .gz
> files for reference under tests/files.
>
> I'll try to take a closer look at this later...
>
> Rob



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Re: [Toybox] [PATCH] Add the gzip/gunzip/zcat tests I wrote for toolbox gzip/gunzip/zcat.

2017-04-26 Thread enh 
On Wed, Apr 26, 2017 at 2:43 PM, Rob Landley  wrote:

> On 04/24/2017 03:43 PM, enh wrote:
> > Bringing the zlib-based gzip/gunzip/zcat over to toybox is a problem for
> > another day,
>
> Blah, I should take this one.
>
> I have deflate decompression side implemented already (and working at
> one point, although I just tested "make zcat" and got a bad crc
> decompressing the musl-1.1.16.gz tarball). I have the first 1/3 or so of
> compression side implemented and somewhere around here is a version
> that's about 2/3 through. I stopped at "where do dictionary resets
> happen" and basically go the answer "nobody seems to agree, just do it
> every 250k". (Which means my gzip and other gzip won't necessarily
> produce the same binary, but it should extract the same. There's
> actually a format hitch you can do sticking in gratuitous resets to
> allow parallel decompression, it's a bit like mp4 streaming mode
> sticking in extra keyframes, only noticeably cheaper. :)
>
> Adding zlib mode to that is basically the same plumbing as
> toys/*/md5sum.c does for libcrypto.
>
> Lemme try to find a chunk of time to devote to that...


if you're actually going to start to look, i'll attach my port of the
current toolbox implementation.

---
 Config.in   |   6 ++
 scripts/make.sh |   2 +-
 toys/pending/compress.c |  93 --
 toys/pending/gzip.c | 199

 4 files changed, 206 insertions(+), 94 deletions(-)
 create mode 100644 toys/pending/gzip.c



>
> > but at least the tests are easy...
> >
> > (These tests pass with TEST_HOST and on the toolbox versions, but the
> > toybox toys are in pending and very broken.)
>
> The problem is this is just testing that gzip and gunzip reverse each
> other. If they share infrastructure and screw something up the same way,
> what have you proved? Also you can't test gunzip without gzip being
> present.
>
> For bzip I added a tests/files/bzcat (and already had a lot of
> tests/files/blkid/*.bz2). I'd like to have at least a couple canned .gz
> files for reference under tests/files.
>
> I'll try to take a closer look at this later...
>
> Rob
>



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
From ab661916a64d67d9b07591c2edb20be21b2e7095 Mon Sep 17 00:00:00 2001
From: Elliott Hughes 
Date: Wed, 26 Apr 2017 15:01:41 -0700
Subject: [PATCH] Implement zlib-based gzip/gunzip/zcat.

---
 Config.in   |   6 ++
 scripts/make.sh |   2 +-
 toys/pending/compress.c |  93 --
 toys/pending/gzip.c | 199 
 4 files changed, 206 insertions(+), 94 deletions(-)
 create mode 100644 toys/pending/gzip.c

diff --git a/Config.in b/Config.in
index 2bfe30a..0524ce3 100644
--- a/Config.in
+++ b/Config.in
@@ -77,6 +77,12 @@ config TOYBOX_LIBCRYPTO
help
  Use faster hash functions out of exteral -lcrypto library.
 
+config TOYBOX_LIBZ
+   bool "Use libz (zlib)"
+   default n
+   help
+ Use libz for gz support.
+
 config TOYBOX_FLOAT
 	bool "Floating point support"
 	default y
diff --git a/scripts/make.sh b/scripts/make.sh
index 5792847..2d33be4 100755
--- a/scripts/make.sh
+++ b/scripts/make.sh
@@ -109,7 +109,7 @@ then
   # for it.
 
   > generated/optlibs.dat
-  for i in util crypt m resolv selinux smack attr rt crypto
+  for i in util crypt m resolv selinux smack attr rt crypto z
   do
 echo "int main(int argc, char *argv[]) {return 0;}" | \
 ${CROSS_COMPILE}${CC} $CFLAGS -xc - -o generated/libprobe -Wl,--as-needed -l$i > /dev/null 2>/dev/null &&
diff --git a/toys/pending/compress.c b/toys/pending/compress.c
index 1749ee4..5605a35 100644
--- a/toys/pending/compress.c
+++ b/toys/pending/compress.c
@@ -16,9 +16,6 @@
 // Leave Lrg at end so flag values line up.
 
 USE_COMPRESS(NEWTOY(compress, "zcd9lrg[-cd][!zgLr]", TOYFLAG_USR|TOYFLAG_BIN))
-USE_GZIP(NEWTOY(gzip, USE_GZIP_D("d")"19dcflqStvgLRz[!gLRz]", TOYFLAG_USR|TOYFLAG_BIN))
-USE_ZCAT(NEWTOY(zcat, 0, TOYFLAG_USR|TOYFLAG_BIN))
-USE_GUNZIP(NEWTOY(gunzip, "cflqStv", TOYFLAG_USR|TOYFLAG_BIN))
 
 //zip unzip gzip gunzip zcat
 
@@ -37,40 +34,6 @@ config COMPRESS
 -R	raw
 -z	zip
 
-config GZIP
-  bool "gzip"
-  default y
-  depends on COMPRESS
-  help
-usage: gzip [-19cfqStvzgLR] [FILE...]
-
-Compess (deflate) file(s). With no files, compress stdin to stdout.
-
-On successful decompression, compressed files are replaced with the
-uncompressed version. The input file is removed and replaced with
-a new file without the .gz extension (with same ownership/permissions).
-
--1	Minimal compression (fastest)
--9	Max compression (default)
--c	cat to stdout (act as zcat)
--f	force (if output file exists, input is tty, unrecognized extension)
--q	quiet (no warnings)
--S	specify exension

Re: [Toybox] [PATCH] Add the gzip/gunzip/zcat tests I wrote for toolbox gzip/gunzip/zcat.

2017-04-26 Thread Rob Landley 
On 04/24/2017 03:43 PM, enh wrote:
> Bringing the zlib-based gzip/gunzip/zcat over to toybox is a problem for
> another day,

Blah, I should take this one.

I have deflate decompression side implemented already (and working at
one point, although I just tested "make zcat" and got a bad crc
decompressing the musl-1.1.16.gz tarball). I have the first 1/3 or so of
compression side implemented and somewhere around here is a version
that's about 2/3 through. I stopped at "where do dictionary resets
happen" and basically go the answer "nobody seems to agree, just do it
every 250k". (Which means my gzip and other gzip won't necessarily
produce the same binary, but it should extract the same. There's
actually a format hitch you can do sticking in gratuitous resets to
allow parallel decompression, it's a bit like mp4 streaming mode
sticking in extra keyframes, only noticeably cheaper. :)

Adding zlib mode to that is basically the same plumbing as
toys/*/md5sum.c does for libcrypto.

Lemme try to find a chunk of time to devote to that...

> but at least the tests are easy...
> 
> (These tests pass with TEST_HOST and on the toolbox versions, but the
> toybox toys are in pending and very broken.)

The problem is this is just testing that gzip and gunzip reverse each
other. If they share infrastructure and screw something up the same way,
what have you proved? Also you can't test gunzip without gzip being present.

For bzip I added a tests/files/bzcat (and already had a lot of
tests/files/blkid/*.bz2). I'd like to have at least a couple canned .gz
files for reference under tests/files.

I'll try to take a closer look at this later...

Rob
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net

[Toybox] [PATCH] Add the gzip/gunzip/zcat tests I wrote for toolbox gzip/gunzip/zcat.

2017-04-24 Thread enh 
Bringing the zlib-based gzip/gunzip/zcat over to toybox is a problem for
another day, but at least the tests are easy...

(These tests pass with TEST_HOST and on the toolbox versions, but the
toybox toys are in pending and very broken.)
---
 tests/gunzip.test | 50 +++
 tests/gzip.test   | 78
+++
 tests/zcat.test   | 34 ++--
 3 files changed, 142 insertions(+), 20 deletions(-)
 create mode 100644 tests/gunzip.test
 create mode 100644 tests/gzip.test
 mode change 100755 => 100644 tests/zcat.test
From 820e8e5b7c10e96664a2615d95e554069656c384 Mon Sep 17 00:00:00 2001
From: Elliott Hughes 
Date: Mon, 24 Apr 2017 13:29:30 -0700
Subject: [PATCH] Add the gzip/gunzip/zcat tests I wrote for toolbox
 gzip/gunzip/zcat.

Bringing the zlib-based gzip/gunzip/zcat over to toybox is a problem for
another day, but at least the tests are easy...

(These tests pass with TEST_HOST and on the toolbox versions, but the
toybox toys are in pending and very broken.)
---
 tests/gunzip.test | 50 +++
 tests/gzip.test   | 78 +++
 tests/zcat.test   | 34 ++--
 3 files changed, 142 insertions(+), 20 deletions(-)
 create mode 100644 tests/gunzip.test
 create mode 100644 tests/gzip.test
 mode change 100755 => 100644 tests/zcat.test

diff --git a/tests/gunzip.test b/tests/gunzip.test
new file mode 100644
index 000..9f9ef5e
--- /dev/null
+++ b/tests/gunzip.test
@@ -0,0 +1,50 @@
+#!/bin/bash
+
+[ -f testing.sh ] && . testing.sh
+
+#testing "name" "command" "result" "infile" "stdin"
+
+# Decompress files.
+# On success, the input files are removed and replaced by new
+# files without the .gz suffix.
+echo -n "foo " | gzip > f1.gz
+echo "bar" | gzip > f2.gz
+testing "with input files" "gunzip f1.gz f2.gz && 
+! test -f f1.gz && ! test -f f2.gz && 
+test -f f1 && test -f f2 && 
+cat f1 f2" "foo bar\n" "" ""
+rm -f f1 f2 f1.gz f2.gz
+
+# With no files, decompresses stdin to stdout.
+echo "hello world" | gzip > f.gz
+testing "no files (stdin to stdout)" "cat f.gz | gunzip > f && 
+test -f f.gz && cat f" "hello world\n" "" ""
+rm -f f f.gz
+
+# -c	Output to stdout
+echo -n "foo " | gzip > f1.gz
+echo "bar" | gzip > f2.gz
+testing "with input files and -c" "gunzip -c f1.gz f2.gz > out && 
+test -f f1.gz && test -f f2.gz && 
+! test -f f1 && ! test -f f2 && 
+cat out" "foo bar\n" "" ""
+rm -f f1.gz f2.gz out
+
+# TODO: how to test "gunzip -f"?
+
+# -k	Keep input files (don't remove)
+echo "hello world" | gzip > f1.gz
+testing "-k" "gunzip -k f1.gz && cat f1 && zcat f1.gz" \
+"hello world\nhello world\n" "" ""
+rm -f f1 f1.gz
+
+# Test that gunzip preserves permissions and times.
+export TZ=UTC
+echo "hello world" | gzip > f1.gz
+chmod 0411 f1.gz
+touch -a -t 197801020304 f1.gz
+touch -m -t 198704030201 f1.gz
+testing "permissions/times preservation" \
+"gunzip -k f1.gz && stat -c '%a %X %Y' f1 && stat -c '%a %Y' f1.gz" \
+"411 252558240 544413660\n411 544413660\n" "" ""
+rm -f f1 f1.gz
diff --git a/tests/gzip.test b/tests/gzip.test
new file mode 100644
index 000..24bd01e
--- /dev/null
+++ b/tests/gzip.test
@@ -0,0 +1,78 @@
+#!/bin/bash
+
+[ -f testing.sh ] && . testing.sh
+
+#testing "name" "command" "result" "infile" "stdin"
+
+# Compress files.
+# On success, the input files are removed and replaced by new
+# files with the .gz suffix.
+echo -n "foo " > f1
+echo "bar" > f2
+testing "with input files" "gzip f1 f2 && 
+test -f f1.gz && test -f f2.gz && 
+! test -f f1 && ! test -f f2 && 
+zcat f1.gz f2.gz" "foo bar\n" "" ""
+rm -f f1 f2 f1.gz f2.gz
+
+# With no files, compresses stdin to stdout.
+testing "no files (stdin to stdout)" "echo hello world | gzip > f.gz && 
+test -f f.gz && zcat f.gz" "hello world\n" "" ""
+rm -f f.gz
+
+# -c	Output to stdout
+echo -n "foo " > f1
+echo "bar" > f2
+testing "with input files and -c" "gzip -c f1 f2 > out.gz && 
+! test -f f1.gz && ! test -f f2.gz && 
+test -f f1 && test -f f2 && 
+zcat out.gz" "foo bar\n" "" ""
+rm -f f1 f2 out.gz
+
+# -d	Decompress (act as gunzip)
+echo "hello world" | gzip > f.gz
+testing "-d (act as gunzip)" "gzip -d f.gz && 
+test -f f && ! test -f f.gz && cat f" "hello world\n" "" ""
+rm -f f.gz f
+
+echo "hello world" | gzip > f.gz
+testing "-dc (act as zcat)" "gzip -dc f.gz && 
+! test -f f && test -f f.gz" "hello world\n" "" ""
+rm -f f.gz f
+
+# -f	Force: allow overwrite of output file
+echo "hello world" > f1
+echo "precious data" > f1.gz
+testing "no overwrite without -f" \
+"gzip f1 2>/dev/null || echo refused && cat f1 f1.gz" \
+"refused\nhello world\nprecious data\n" "" ""
+testing "overwrite with -f" \
+"gzip -f f1 && echo allowed && ! test -f f1 && zcat f1.gz" \
+"allowed\nhello world\n" "" ""
+rm -f f1 f1.gz
+
+# -k	Keep input files (don't remove)
+echo