On 2020/04/10 20:21, Olivier Taïbi wrote:
> Here is further progress on this port, following versions by Enric
> Morales and Stuart Henderson. I hope that this does not duplicate your
> efforts.
>
> There were three zlib-related bugs in notmuch:
> 1) second argument (&enum) of gzerror() cannot be NULL, as gzerror()
> will write to this address. This was causing SIGSEGV.
> 2) gzerror() cannot be called after gzclose() (use after free). This
> was causing SIGBUS.
> Patching these two got rid of all the SIGSEGV and SIGBUS in tests.
great, that's a big improvement :)
> 3) these two errors were not spotted before because no one was reaching
> them with a recent zlib (they are bugs both for the latest zlib and
> OpenBSD's zlib though). It seems that gzerror() in OpenBSD's older zlib
> will return (or rather set errnum to) Z_STREAM_END when reaching EOF,
> and this was handled as an error in notmuch's gz_getline(), when in fact
> it seems to be equivalent to Z_OK plus the indication that we reached
> EOF. Unfortunately the manual of zlib does not say much about the
> meaning of Z_* error codes, but that's the most obvious meaning (to me
> at least).
>
> I reported these bugs on the notmuch mailing list, but at least one
> issue remains. OpenBSD's zlib does not have a relatively recent feature
> of zlib: the ability to write uncompressed files, or "T" (for
> "transparent") mode. The result is equivalent to "--gzip" being the
> default (and the only choice!) for notmuch dump. To run the tests I
> patched them by adding "| zcat" after each notmuch dump which is not
> followed by --gzip, and a gunzip [...] line after each notmuch dump
> --output. I also replaced gpg by gpg2 in the tests, and imposed the
> python3 FLAVOR because it is necessary for a few tests (I may have done
> this the wrong way). There are still 9 tests that fail, but I am not
for a standalone port, use "MODPY_VERSION = ${MODPY_DEFAULT_VERSION_3}",
the FLAVOR setup is for python modules (py-* ports).
> sure that all of them are real failures. That's not counting the tests
> that Stuart Henderson forced to skip (I haven't looked at them yet) and
> the tests requiring large files (databases to be downloaded from
> notmuch's website).
>
> I see 3 ways to solve this outstanding problem:
> 1) add the "T" mode to OpenBSD's zlib, for example by merging with a
> more recent zlib from upstream. Unfortunately this seems to be
> a lot of work: see
> https://marc.info/?l=openbsd-tech&m=141936824422127&w=2
The update of the main copy in src/lib/libz has been done at least twice
(though I don't think it happened for the other copies in sys/lib/libz and
src/gnu/usr.bin/cvs - amusingly the fourth copy, in perl, is already at
1.2.11) - but not made it into the tree. The thing I remember putting
some people off was the "new" z_off64_t (10 years old today in the
public api).
But updating it is the only option that fixes the various pain / patching
/ workarounds / using bundled copies in various things in ports that
has been a problem at least going back to 2012.
> One could also just add the "T" mode.
It does help notmuch but not most other ports where we ran into
problems with old zlib (gzoffset was one of the more common ones iirc).
> 2) just accept that notmuch dump will only output gzipped data, i.e.
> only patch the man page to indicate that --gzip is the one and only
> choice. Not very nice...
Not a fan of this.
> 3) patch notmuch to remove the compression feature (--gzip), essentially
> replacing zlib by stdio. I like that because the gzip feature is not
> really useful anyway (a pipe to any compression program is simpler and
> more flexible), but that will mean merging work for each port update.
> Perhaps this could be suggested upstream?
That does sound like a good idea (and probably more acceptable to
upstream than having separate code for gzopen/gzFile/gzprintf vs
fopen/FILE */fprintf just to cope with OS with old zlib..)
It's probably not _too_ much of a maintenance burden in ports either,
certainly less of a burden than patching the tests to use zcat/gunzip.
