On 10/30/2018 11:32 AM, 'Paulo Matos' via Racket Users wrote:
I have quite a few large files that I want to gzip to a single file (without an intermediate concatenation) and then later gunzip.
I don't think you can do that - at least not without other software. gzip/gunzip are meant to work only with a single file. gzip is a compression format, not an archive format - the compressed stream is assumed to contain a single object, it has no index or other metadata to handle multiple objects. Typically you would tar the files and then zip the tar.
Interestingly the gunzipping is blocking on a read-line. I wonder if this is because I cannot use gzip-through-ports the way I am doing it or if there's a bug somewhere. Generate 3 files with: $ base64 /dev/urandom | head -c 1000000 > foo3 $ base64 /dev/urandom | head -c 1000000 > foo2 $ base64 /dev/urandom | head -c 1000000 > foo1 Now run the code: ``` #lang racket (require file/gzip file/gunzip) (define paths '("foo1" "foo2" "foo3")) ;; compress (printf "compressing~n") (call-with-atomic-output-file "foo.gz" (lambda (op p) (for ([f (in-list paths)]) (call-with-input-file f (lambda (i) (gzip-through-ports i op #false (current-seconds))) #:mode 'binary)))) ;; decompress (printf "decompressing~n") (define-values (in out) (make-pipe)) (void (thread (thunk (call-with-input-file "foo.gz" (lambda (cin) (gunzip-through-ports cin out)) #:mode 'binary)))) (call-with-atomic-output-file "foo.txt" (lambda (op p) (let loop ([l (read-line in)]) (unless (eof-object? l) (write l op) (loop (read-line in)))))) ``` This is going to block in a read-line, and I have a suspicion that it blocks at the end of a compressed file. Is there a reason for it blocking? Note that if you `zcat foo.gz | less` you can see the whole file, so I am suspicious that something might be wrong with gunzip-through-ports. Any suggestions to improve this? Thanks,
I think you can recover by closing the port when gunzip finishes, but in this case you will only receive the 1st file - additional files will be lost. If you want to send multiple objects in the same stream, you need to use some protocol to delimit them. I'm not sure how [or if] you can duplicate the port to gunzip-through-ports, but you can read the port yourself and pass the data to gunzip.
George -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.