Re: wip: mail/notmuch
On 5/17/20 5:40 AM, Stuart Henderson wrote:
> See what was committed.
>
Wow this is great news!
Thanks a lot to everyone involved for their hard work :)
Aisha
> --
> Sent from a phone, apologies for poor formatting.
>
> On 16 May 2020 23:44:32 Andrea Fleckenstein wrote:
>
>> Hello,
>>
>> I had a bit of an issue compiling this again, I had change
>> patch_configure to use gpg2 instead of gpg in the GMime check in order
>> to get it to compile. Am I missing something here?
>>
>> I've been using this with emacs integration for a while now, tagging and
>> searching seems to be working just fine.
>>
>> Best,
>> Andrea
>>
>> --- patch-configure.orig Sat May 16 17:46:56 2020
>> +++ patch-configure Sat May 16 18:35:02 2020
>> @@ -3,6 +3,15 @@
>> Index: configure
>> --- configure.orig
>> +++ configure
>> +@@ -533,7 +533,7 @@ EOF
>> + printf 'No.\nCould not make tempdir for testing session-key
>> support.\n'
>> + errors=$((errors + 1))
>> + elif ${CC} ${CFLAGS} ${gmime_cflags} _check_session_keys.c
>> ${gmime_ldflags} -o _check_session_keys \
>> +- && GNUPGHOME=${TEMP_GPG} gpg --batch --quiet --import <
>> test/gnupg-secret-key.asc \
>> ++ && GNUPGHOME=${TEMP_GPG} gpg2 --batch --quiet --import <
>> test/gnupg-secret-key.asc \
>> + && SESSION_KEY=$(GNUPGHOME=${TEMP_GPG} ./_check_session_keys) \
>> + && [ $SESSION_KEY =
>> 9:0BACD64099D1468AB07C796F0C0AC4851948A658A15B34E803865E9FC635F2F5 ]
>> + then
>> @@ -588,9 +588,9 @@ if ! pkg-config --exists zlib; then
>> rm -f compat/gen_zlib_pc
>> fi
>>
>> Stuart Henderson writes:
>>
>>> Anyone want to import this now (or give me an ok to do so)?
>>> Reattached.
>>>
>>> (Worked on by various people including me, Olivier Taïbi, and Enric Morales)
>>>
>>>
>>>
>>> On 2020/04/19 22:32, Stuart Henderson wrote:
This is now OK sthen@ to import. As there has been a fair bit of
interest in this over time, I think it might be worth considering this
for commit before 6.7 if another committer agrees, but I would not like
to add the dependency to neomutt until after release (because then
notmuch build failures for any arches would knock out building neomutt
on those arches).
On 2020/04/19 23:14, Olivier Taïbi wrote:
> Oops, just realised that the addition of MODPY_VERSION as Stuart
> Henderson suggested means that sphinx-build-3 is installed as a
> dependency, instead of sphinx-build previously. The attached port
> reflects this (sphinx-build -> sphinx-build-3 in two files).
>
> On Thu, Apr 16, 2020 at 08:47:50PM +0200, Olivier Taïbi wrote:
> > On Fri, Apr 10, 2020 at 09:47:35PM +0100, Stuart Henderson wrote:
> > > for a standalone port, use "MODPY_VERSION =
> > > ${MODPY_DEFAULT_VERSION_3}",
> > > the FLAVOR setup is for python modules (py-* ports).
> >
> > Thanks, I did that.
> >
> > > The update of the main copy in src/lib/libz has been done at least
> > > twice
> > > (though I don't think it happened for the other copies in
> > > sys/lib/libz and
> > > src/gnu/usr.bin/cvs - amusingly the fourth copy, in perl, is already
> > > at
> > > 1.2.11) - but not made it into the tree. The thing I remember putting
> > > some people off was the "new" z_off64_t (10 years old today in the
> > > public api).
> > >
> > > But updating it is the only option that fixes the various pain /
> > > patching
> > > / workarounds / using bundled copies in various things in ports that
> > > has been a problem at least going back to 2012.
> >
> > Would incremental changes converging towards API compatibility with
> > newer zlib stand a better chance of being committed?
> >
> > In any case, I patched notmuch-dump.c (mostly) to replace zlib by stdio
> > and added crude error checking. Thanks to the notmuch tests I believe
> > I
> > came across a bug in src/lib/libz/gzio.c, see
> > https://marc.info/?l=openbsd-bugs=158697346006702
> > With this patch to base, almost all tests now pass, except:
> > - in T350-crypto, the first decryption test fails with "Failed to
> > decrypt part: Decryption failed: No secret key". Oddly, the next
> >ones
> > succeed, and if I permute them, then only the first one fails. I was
> > not able to reproduce this bug (even running the test on another
> > machine made it disappear, go figure...), and I would guess that it
> > comes from gmime30 (which is responsible for looking for said secret
> > key) rather than notmuch. Probably related to an environment
> >variable
> > or something else particular to the chroot in which I build.
> > - In T455-emacs-charsets, two tests related to the Yen character fail,
> > but I think they shouldn't: the output is the "Yen sign" U+00A5, i.e.
Re: wip: mail/notmuch
See what was committed.
--
Sent from a phone, apologies for poor formatting.
On 16 May 2020 23:44:32 Andrea Fleckenstein wrote:
Hello,
I had a bit of an issue compiling this again, I had change
patch_configure to use gpg2 instead of gpg in the GMime check in order
to get it to compile. Am I missing something here?
I've been using this with emacs integration for a while now, tagging and
searching seems to be working just fine.
Best,
Andrea
--- patch-configure.origSat May 16 17:46:56 2020
+++ patch-configure Sat May 16 18:35:02 2020
@@ -3,6 +3,15 @@
Index: configure
--- configure.orig
+++ configure
+@@ -533,7 +533,7 @@ EOF
+ printf 'No.\nCould not make tempdir for testing session-key
support.\n'
+ errors=$((errors + 1))
+ elif ${CC} ${CFLAGS} ${gmime_cflags} _check_session_keys.c
${gmime_ldflags} -o _check_session_keys \
+- && GNUPGHOME=${TEMP_GPG} gpg --batch --quiet --import <
test/gnupg-secret-key.asc \
++ && GNUPGHOME=${TEMP_GPG} gpg2 --batch --quiet --import <
test/gnupg-secret-key.asc \
+&& SESSION_KEY=$(GNUPGHOME=${TEMP_GPG} ./_check_session_keys) \
+&& [ $SESSION_KEY =
9:0BACD64099D1468AB07C796F0C0AC4851948A658A15B34E803865E9FC635F2F5 ]
+ then
@@ -588,9 +588,9 @@ if ! pkg-config --exists zlib; then
rm -f compat/gen_zlib_pc
fi
Stuart Henderson writes:
Anyone want to import this now (or give me an ok to do so)?
Reattached.
(Worked on by various people including me, Olivier Taïbi, and Enric Morales)
On 2020/04/19 22:32, Stuart Henderson wrote:
This is now OK sthen@ to import. As there has been a fair bit of
interest in this over time, I think it might be worth considering this
for commit before 6.7 if another committer agrees, but I would not like
to add the dependency to neomutt until after release (because then
notmuch build failures for any arches would knock out building neomutt
on those arches).
On 2020/04/19 23:14, Olivier Taïbi wrote:
> Oops, just realised that the addition of MODPY_VERSION as Stuart
> Henderson suggested means that sphinx-build-3 is installed as a
> dependency, instead of sphinx-build previously. The attached port
> reflects this (sphinx-build -> sphinx-build-3 in two files).
>
> On Thu, Apr 16, 2020 at 08:47:50PM +0200, Olivier Taïbi wrote:
> > On Fri, Apr 10, 2020 at 09:47:35PM +0100, Stuart Henderson wrote:
> > > for a standalone port, use "MODPY_VERSION = ${MODPY_DEFAULT_VERSION_3}",
> > > the FLAVOR setup is for python modules (py-* ports).
> >
> > Thanks, I did that.
> >
> > > The update of the main copy in src/lib/libz has been done at least twice
> > > (though I don't think it happened for the other copies in
sys/lib/libz and
> > > src/gnu/usr.bin/cvs - amusingly the fourth copy, in perl, is already at
> > > 1.2.11) - but not made it into the tree. The thing I remember putting
> > > some people off was the "new" z_off64_t (10 years old today in the
> > > public api).
> > >
> > > But updating it is the only option that fixes the various pain / patching
> > > / workarounds / using bundled copies in various things in ports that
> > > has been a problem at least going back to 2012.
> >
> > Would incremental changes converging towards API compatibility with
> > newer zlib stand a better chance of being committed?
> >
> > In any case, I patched notmuch-dump.c (mostly) to replace zlib by stdio
> > and added crude error checking. Thanks to the notmuch tests I believe I
> > came across a bug in src/lib/libz/gzio.c, see
> > https://marc.info/?l=openbsd-bugs=158697346006702
> > With this patch to base, almost all tests now pass, except:
> > - in T350-crypto, the first decryption test fails with "Failed to
> > decrypt part: Decryption failed: No secret key". Oddly, the next ones
> > succeed, and if I permute them, then only the first one fails. I was
> > not able to reproduce this bug (even running the test on another
> > machine made it disappear, go figure...), and I would guess that it
> > comes from gmime30 (which is responsible for looking for said secret
> > key) rather than notmuch. Probably related to an environment variable
> > or something else particular to the chroot in which I build.
> > - In T455-emacs-charsets, two tests related to the Yen character fail,
> > but I think they shouldn't: the output is the "Yen sign" U+00A5, i.e.
> > 0xC2 0xA5, when the test expected the "Fullwidth Yen sign" U+FFE5,
> > i.e. 0xEF 0xBF 0xA5. What we get seems to be more correct.
> > - in T610-message-property, the "prefix" test fails, but this seems to
> > be because the test is incorrect (it expects a certain order that is
> > not guaranteed by the implementation: there is a key-multiple values
> > map in which the keys are sorted, but not the values). The test
> > itself should be fixed.
> > I also ran the tests in T530-upgrade, which require a file to be
> > downloaded separately (I did not see a way to achieve this automatically
> >
Re: wip: mail/notmuch
Hello,
I had a bit of an issue compiling this again, I had change
patch_configure to use gpg2 instead of gpg in the GMime check in order
to get it to compile. Am I missing something here?
I've been using this with emacs integration for a while now, tagging and
searching seems to be working just fine.
Best,
Andrea
--- patch-configure.origSat May 16 17:46:56 2020
+++ patch-configure Sat May 16 18:35:02 2020
@@ -3,6 +3,15 @@
Index: configure
--- configure.orig
+++ configure
+@@ -533,7 +533,7 @@ EOF
+ printf 'No.\nCould not make tempdir for testing session-key
support.\n'
+ errors=$((errors + 1))
+ elif ${CC} ${CFLAGS} ${gmime_cflags} _check_session_keys.c
${gmime_ldflags} -o _check_session_keys \
+- && GNUPGHOME=${TEMP_GPG} gpg --batch --quiet --import <
test/gnupg-secret-key.asc \
++ && GNUPGHOME=${TEMP_GPG} gpg2 --batch --quiet --import <
test/gnupg-secret-key.asc \
+&& SESSION_KEY=$(GNUPGHOME=${TEMP_GPG} ./_check_session_keys) \
+&& [ $SESSION_KEY =
9:0BACD64099D1468AB07C796F0C0AC4851948A658A15B34E803865E9FC635F2F5 ]
+ then
@@ -588,9 +588,9 @@ if ! pkg-config --exists zlib; then
rm -f compat/gen_zlib_pc
fi
Stuart Henderson writes:
> Anyone want to import this now (or give me an ok to do so)?
> Reattached.
>
> (Worked on by various people including me, Olivier Taïbi, and Enric Morales)
>
>
>
> On 2020/04/19 22:32, Stuart Henderson wrote:
>> This is now OK sthen@ to import. As there has been a fair bit of
>> interest in this over time, I think it might be worth considering this
>> for commit before 6.7 if another committer agrees, but I would not like
>> to add the dependency to neomutt until after release (because then
>> notmuch build failures for any arches would knock out building neomutt
>> on those arches).
>>
>>
>>
>> On 2020/04/19 23:14, Olivier Taïbi wrote:
>> > Oops, just realised that the addition of MODPY_VERSION as Stuart
>> > Henderson suggested means that sphinx-build-3 is installed as a
>> > dependency, instead of sphinx-build previously. The attached port
>> > reflects this (sphinx-build -> sphinx-build-3 in two files).
>> >
>> > On Thu, Apr 16, 2020 at 08:47:50PM +0200, Olivier Taïbi wrote:
>> > > On Fri, Apr 10, 2020 at 09:47:35PM +0100, Stuart Henderson wrote:
>> > > > for a standalone port, use "MODPY_VERSION =
>> > > > ${MODPY_DEFAULT_VERSION_3}",
>> > > > the FLAVOR setup is for python modules (py-* ports).
>> > >
>> > > Thanks, I did that.
>> > >
>> > > > The update of the main copy in src/lib/libz has been done at least
>> > > > twice
>> > > > (though I don't think it happened for the other copies in sys/lib/libz
>> > > > and
>> > > > src/gnu/usr.bin/cvs - amusingly the fourth copy, in perl, is already at
>> > > > 1.2.11) - but not made it into the tree. The thing I remember putting
>> > > > some people off was the "new" z_off64_t (10 years old today in the
>> > > > public api).
>> > > >
>> > > > But updating it is the only option that fixes the various pain /
>> > > > patching
>> > > > / workarounds / using bundled copies in various things in ports that
>> > > > has been a problem at least going back to 2012.
>> > >
>> > > Would incremental changes converging towards API compatibility with
>> > > newer zlib stand a better chance of being committed?
>> > >
>> > > In any case, I patched notmuch-dump.c (mostly) to replace zlib by stdio
>> > > and added crude error checking. Thanks to the notmuch tests I believe I
>> > > came across a bug in src/lib/libz/gzio.c, see
>> > > https://marc.info/?l=openbsd-bugs=158697346006702
>> > > With this patch to base, almost all tests now pass, except:
>> > > - in T350-crypto, the first decryption test fails with "Failed to
>> > > decrypt part: Decryption failed: No secret key". Oddly, the next ones
>> > > succeed, and if I permute them, then only the first one fails. I was
>> > > not able to reproduce this bug (even running the test on another
>> > > machine made it disappear, go figure...), and I would guess that it
>> > > comes from gmime30 (which is responsible for looking for said secret
>> > > key) rather than notmuch. Probably related to an environment variable
>> > > or something else particular to the chroot in which I build.
>> > > - In T455-emacs-charsets, two tests related to the Yen character fail,
>> > > but I think they shouldn't: the output is the "Yen sign" U+00A5, i.e.
>> > > 0xC2 0xA5, when the test expected the "Fullwidth Yen sign" U+FFE5,
>> > > i.e. 0xEF 0xBF 0xA5. What we get seems to be more correct.
>> > > - in T610-message-property, the "prefix" test fails, but this seems to
>> > > be because the test is incorrect (it expects a certain order that is
>> > > not guaranteed by the implementation: there is a key-multiple values
>> > > map in which the keys are sorted, but not the values). The test
>> > > itself should be fixed.
>> > > I also ran the tests in T530-upgrade, which
Re: wip: mail/notmuch
On Thu, May 14, 2020 at 10:30:01PM +0100, Stuart Henderson wrote: > Anyone want to import this now (or give me an ok to do so)? > Reattached. > (Worked on by various people including me, Olivier Taïbi, and Enric Morales) No reason not to import, but I was curious at not making the python extensions python3-only when the main port is python3-only. That could always be tweaked after import though. ok kmos to import --Kurt
Re: wip: mail/notmuch
Anyone want to import this now (or give me an ok to do so)?
Reattached.
(Worked on by various people including me, Olivier Taïbi, and Enric Morales)
On 2020/04/19 22:32, Stuart Henderson wrote:
> This is now OK sthen@ to import. As there has been a fair bit of
> interest in this over time, I think it might be worth considering this
> for commit before 6.7 if another committer agrees, but I would not like
> to add the dependency to neomutt until after release (because then
> notmuch build failures for any arches would knock out building neomutt
> on those arches).
>
>
>
> On 2020/04/19 23:14, Olivier Taïbi wrote:
> > Oops, just realised that the addition of MODPY_VERSION as Stuart
> > Henderson suggested means that sphinx-build-3 is installed as a
> > dependency, instead of sphinx-build previously. The attached port
> > reflects this (sphinx-build -> sphinx-build-3 in two files).
> >
> > On Thu, Apr 16, 2020 at 08:47:50PM +0200, Olivier Taïbi wrote:
> > > On Fri, Apr 10, 2020 at 09:47:35PM +0100, Stuart Henderson wrote:
> > > > for a standalone port, use "MODPY_VERSION = ${MODPY_DEFAULT_VERSION_3}",
> > > > the FLAVOR setup is for python modules (py-* ports).
> > >
> > > Thanks, I did that.
> > >
> > > > The update of the main copy in src/lib/libz has been done at least twice
> > > > (though I don't think it happened for the other copies in sys/lib/libz
> > > > and
> > > > src/gnu/usr.bin/cvs - amusingly the fourth copy, in perl, is already at
> > > > 1.2.11) - but not made it into the tree. The thing I remember putting
> > > > some people off was the "new" z_off64_t (10 years old today in the
> > > > public api).
> > > >
> > > > But updating it is the only option that fixes the various pain /
> > > > patching
> > > > / workarounds / using bundled copies in various things in ports that
> > > > has been a problem at least going back to 2012.
> > >
> > > Would incremental changes converging towards API compatibility with
> > > newer zlib stand a better chance of being committed?
> > >
> > > In any case, I patched notmuch-dump.c (mostly) to replace zlib by stdio
> > > and added crude error checking. Thanks to the notmuch tests I believe I
> > > came across a bug in src/lib/libz/gzio.c, see
> > > https://marc.info/?l=openbsd-bugs=158697346006702
> > > With this patch to base, almost all tests now pass, except:
> > > - in T350-crypto, the first decryption test fails with "Failed to
> > > decrypt part: Decryption failed: No secret key". Oddly, the next ones
> > > succeed, and if I permute them, then only the first one fails. I was
> > > not able to reproduce this bug (even running the test on another
> > > machine made it disappear, go figure...), and I would guess that it
> > > comes from gmime30 (which is responsible for looking for said secret
> > > key) rather than notmuch. Probably related to an environment variable
> > > or something else particular to the chroot in which I build.
> > > - In T455-emacs-charsets, two tests related to the Yen character fail,
> > > but I think they shouldn't: the output is the "Yen sign" U+00A5, i.e.
> > > 0xC2 0xA5, when the test expected the "Fullwidth Yen sign" U+FFE5,
> > > i.e. 0xEF 0xBF 0xA5. What we get seems to be more correct.
> > > - in T610-message-property, the "prefix" test fails, but this seems to
> > > be because the test is incorrect (it expects a certain order that is
> > > not guaranteed by the implementation: there is a key-multiple values
> > > map in which the keys are sorted, but not the values). The test
> > > itself should be fixed.
> > > I also ran the tests in T530-upgrade, which require a file to be
> > > downloaded separately (I did not see a way to achieve this automatically
> > > using bsd.port.mk, so I just downloaded the file on the side and ran the
> > > test manually), and the tests pass (after a small patch in
> > > notmuch-new.c). It probably does not matter much anyway because these
> > > tests concern upgrading from an old database.
> > >
> > > So I am now happy with the result of the tests.
> > >
> > > I did not touch the python bindings part of the port. In fact I did not
> > > even look at it, but the test pass so hopefully they're ok. Grepping
> > > for dump does not yield any match, so I guess that the removal of the
> > > --gzip option makes no difference for them.
> > >
> > > Note that there are also ruby bindings, which are currently not built by
> > > the port, and that are presumably required for the vim plugin. I do not
> > > intend to look at them in the near future.
> > >
> > > I hope that this port can now be commited. I have been using notmuch on
> > > OpenBSD for more than a year without issues now, but I have not tagged
> > > anything (too lazy) and only used searching, and so I have never had any
> > > use for the dump/restore feature. I hope that Enric, Andrea and others
> > > can test tagging more extensively now.
> > >
> > > Next, there is a trivial patch
Re: wip: mail/notmuch
This is now OK sthen@ to import. As there has been a fair bit of
interest in this over time, I think it might be worth considering this
for commit before 6.7 if another committer agrees, but I would not like
to add the dependency to neomutt until after release (because then
notmuch build failures for any arches would knock out building neomutt
on those arches).
On 2020/04/19 23:14, Olivier Taïbi wrote:
> Oops, just realised that the addition of MODPY_VERSION as Stuart
> Henderson suggested means that sphinx-build-3 is installed as a
> dependency, instead of sphinx-build previously. The attached port
> reflects this (sphinx-build -> sphinx-build-3 in two files).
>
> On Thu, Apr 16, 2020 at 08:47:50PM +0200, Olivier Taïbi wrote:
> > On Fri, Apr 10, 2020 at 09:47:35PM +0100, Stuart Henderson wrote:
> > > for a standalone port, use "MODPY_VERSION = ${MODPY_DEFAULT_VERSION_3}",
> > > the FLAVOR setup is for python modules (py-* ports).
> >
> > Thanks, I did that.
> >
> > > The update of the main copy in src/lib/libz has been done at least twice
> > > (though I don't think it happened for the other copies in sys/lib/libz and
> > > src/gnu/usr.bin/cvs - amusingly the fourth copy, in perl, is already at
> > > 1.2.11) - but not made it into the tree. The thing I remember putting
> > > some people off was the "new" z_off64_t (10 years old today in the
> > > public api).
> > >
> > > But updating it is the only option that fixes the various pain / patching
> > > / workarounds / using bundled copies in various things in ports that
> > > has been a problem at least going back to 2012.
> >
> > Would incremental changes converging towards API compatibility with
> > newer zlib stand a better chance of being committed?
> >
> > In any case, I patched notmuch-dump.c (mostly) to replace zlib by stdio
> > and added crude error checking. Thanks to the notmuch tests I believe I
> > came across a bug in src/lib/libz/gzio.c, see
> > https://marc.info/?l=openbsd-bugs=158697346006702
> > With this patch to base, almost all tests now pass, except:
> > - in T350-crypto, the first decryption test fails with "Failed to
> > decrypt part: Decryption failed: No secret key". Oddly, the next ones
> > succeed, and if I permute them, then only the first one fails. I was
> > not able to reproduce this bug (even running the test on another
> > machine made it disappear, go figure...), and I would guess that it
> > comes from gmime30 (which is responsible for looking for said secret
> > key) rather than notmuch. Probably related to an environment variable
> > or something else particular to the chroot in which I build.
> > - In T455-emacs-charsets, two tests related to the Yen character fail,
> > but I think they shouldn't: the output is the "Yen sign" U+00A5, i.e.
> > 0xC2 0xA5, when the test expected the "Fullwidth Yen sign" U+FFE5,
> > i.e. 0xEF 0xBF 0xA5. What we get seems to be more correct.
> > - in T610-message-property, the "prefix" test fails, but this seems to
> > be because the test is incorrect (it expects a certain order that is
> > not guaranteed by the implementation: there is a key-multiple values
> > map in which the keys are sorted, but not the values). The test
> > itself should be fixed.
> > I also ran the tests in T530-upgrade, which require a file to be
> > downloaded separately (I did not see a way to achieve this automatically
> > using bsd.port.mk, so I just downloaded the file on the side and ran the
> > test manually), and the tests pass (after a small patch in
> > notmuch-new.c). It probably does not matter much anyway because these
> > tests concern upgrading from an old database.
> >
> > So I am now happy with the result of the tests.
> >
> > I did not touch the python bindings part of the port. In fact I did not
> > even look at it, but the test pass so hopefully they're ok. Grepping
> > for dump does not yield any match, so I guess that the removal of the
> > --gzip option makes no difference for them.
> >
> > Note that there are also ruby bindings, which are currently not built by
> > the port, and that are presumably required for the vim plugin. I do not
> > intend to look at them in the near future.
> >
> > I hope that this port can now be commited. I have been using notmuch on
> > OpenBSD for more than a year without issues now, but I have not tagged
> > anything (too lazy) and only used searching, and so I have never had any
> > use for the dump/restore feature. I hope that Enric, Andrea and others
> > can test tagging more extensively now.
> >
> > Next, there is a trivial patch for neomutt (notmuch is a configure
> > option), which I also have been happily using. Would that be better as
> > a flavor or is notmuch small enough to be a dependency of neomutt?
> >
> > PS: The three zlib-related bugs in notmuch mentioned in a previous email
> > are now fixed upstream, so they should vanish in the next version.
>
>
Re: wip: mail/notmuch
Oops, just realised that the addition of MODPY_VERSION as Stuart
Henderson suggested means that sphinx-build-3 is installed as a
dependency, instead of sphinx-build previously. The attached port
reflects this (sphinx-build -> sphinx-build-3 in two files).
On Thu, Apr 16, 2020 at 08:47:50PM +0200, Olivier Taïbi wrote:
> On Fri, Apr 10, 2020 at 09:47:35PM +0100, Stuart Henderson wrote:
> > for a standalone port, use "MODPY_VERSION = ${MODPY_DEFAULT_VERSION_3}",
> > the FLAVOR setup is for python modules (py-* ports).
>
> Thanks, I did that.
>
> > The update of the main copy in src/lib/libz has been done at least twice
> > (though I don't think it happened for the other copies in sys/lib/libz and
> > src/gnu/usr.bin/cvs - amusingly the fourth copy, in perl, is already at
> > 1.2.11) - but not made it into the tree. The thing I remember putting
> > some people off was the "new" z_off64_t (10 years old today in the
> > public api).
> >
> > But updating it is the only option that fixes the various pain / patching
> > / workarounds / using bundled copies in various things in ports that
> > has been a problem at least going back to 2012.
>
> Would incremental changes converging towards API compatibility with
> newer zlib stand a better chance of being committed?
>
> In any case, I patched notmuch-dump.c (mostly) to replace zlib by stdio
> and added crude error checking. Thanks to the notmuch tests I believe I
> came across a bug in src/lib/libz/gzio.c, see
> https://marc.info/?l=openbsd-bugs=158697346006702
> With this patch to base, almost all tests now pass, except:
> - in T350-crypto, the first decryption test fails with "Failed to
> decrypt part: Decryption failed: No secret key". Oddly, the next ones
> succeed, and if I permute them, then only the first one fails. I was
> not able to reproduce this bug (even running the test on another
> machine made it disappear, go figure...), and I would guess that it
> comes from gmime30 (which is responsible for looking for said secret
> key) rather than notmuch. Probably related to an environment variable
> or something else particular to the chroot in which I build.
> - In T455-emacs-charsets, two tests related to the Yen character fail,
> but I think they shouldn't: the output is the "Yen sign" U+00A5, i.e.
> 0xC2 0xA5, when the test expected the "Fullwidth Yen sign" U+FFE5,
> i.e. 0xEF 0xBF 0xA5. What we get seems to be more correct.
> - in T610-message-property, the "prefix" test fails, but this seems to
> be because the test is incorrect (it expects a certain order that is
> not guaranteed by the implementation: there is a key-multiple values
> map in which the keys are sorted, but not the values). The test
> itself should be fixed.
> I also ran the tests in T530-upgrade, which require a file to be
> downloaded separately (I did not see a way to achieve this automatically
> using bsd.port.mk, so I just downloaded the file on the side and ran the
> test manually), and the tests pass (after a small patch in
> notmuch-new.c). It probably does not matter much anyway because these
> tests concern upgrading from an old database.
>
> So I am now happy with the result of the tests.
>
> I did not touch the python bindings part of the port. In fact I did not
> even look at it, but the test pass so hopefully they're ok. Grepping
> for dump does not yield any match, so I guess that the removal of the
> --gzip option makes no difference for them.
>
> Note that there are also ruby bindings, which are currently not built by
> the port, and that are presumably required for the vim plugin. I do not
> intend to look at them in the near future.
>
> I hope that this port can now be commited. I have been using notmuch on
> OpenBSD for more than a year without issues now, but I have not tagged
> anything (too lazy) and only used searching, and so I have never had any
> use for the dump/restore feature. I hope that Enric, Andrea and others
> can test tagging more extensively now.
>
> Next, there is a trivial patch for neomutt (notmuch is a configure
> option), which I also have been happily using. Would that be better as
> a flavor or is notmuch small enough to be a dependency of neomutt?
>
> PS: The three zlib-related bugs in notmuch mentioned in a previous email
> are now fixed upstream, so they should vanish in the next version.
notmuch.tgz
Description: Binary data
Re: wip: mail/notmuch
On Fri, Apr 10, 2020 at 09:47:35PM +0100, Stuart Henderson wrote:
> for a standalone port, use "MODPY_VERSION = ${MODPY_DEFAULT_VERSION_3}",
> the FLAVOR setup is for python modules (py-* ports).
Thanks, I did that.
> The update of the main copy in src/lib/libz has been done at least twice
> (though I don't think it happened for the other copies in sys/lib/libz and
> src/gnu/usr.bin/cvs - amusingly the fourth copy, in perl, is already at
> 1.2.11) - but not made it into the tree. The thing I remember putting
> some people off was the "new" z_off64_t (10 years old today in the
> public api).
>
> But updating it is the only option that fixes the various pain / patching
> / workarounds / using bundled copies in various things in ports that
> has been a problem at least going back to 2012.
Would incremental changes converging towards API compatibility with
newer zlib stand a better chance of being committed?
In any case, I patched notmuch-dump.c (mostly) to replace zlib by stdio
and added crude error checking. Thanks to the notmuch tests I believe I
came across a bug in src/lib/libz/gzio.c, see
https://marc.info/?l=openbsd-bugs=158697346006702
With this patch to base, almost all tests now pass, except:
- in T350-crypto, the first decryption test fails with "Failed to
decrypt part: Decryption failed: No secret key". Oddly, the next ones
succeed, and if I permute them, then only the first one fails. I was
not able to reproduce this bug (even running the test on another
machine made it disappear, go figure...), and I would guess that it
comes from gmime30 (which is responsible for looking for said secret
key) rather than notmuch. Probably related to an environment variable
or something else particular to the chroot in which I build.
- In T455-emacs-charsets, two tests related to the Yen character fail,
but I think they shouldn't: the output is the "Yen sign" U+00A5, i.e.
0xC2 0xA5, when the test expected the "Fullwidth Yen sign" U+FFE5,
i.e. 0xEF 0xBF 0xA5. What we get seems to be more correct.
- in T610-message-property, the "prefix" test fails, but this seems to
be because the test is incorrect (it expects a certain order that is
not guaranteed by the implementation: there is a key-multiple values
map in which the keys are sorted, but not the values). The test
itself should be fixed.
I also ran the tests in T530-upgrade, which require a file to be
downloaded separately (I did not see a way to achieve this automatically
using bsd.port.mk, so I just downloaded the file on the side and ran the
test manually), and the tests pass (after a small patch in
notmuch-new.c). It probably does not matter much anyway because these
tests concern upgrading from an old database.
So I am now happy with the result of the tests.
I did not touch the python bindings part of the port. In fact I did not
even look at it, but the test pass so hopefully they're ok. Grepping
for dump does not yield any match, so I guess that the removal of the
--gzip option makes no difference for them.
Note that there are also ruby bindings, which are currently not built by
the port, and that are presumably required for the vim plugin. I do not
intend to look at them in the near future.
I hope that this port can now be commited. I have been using notmuch on
OpenBSD for more than a year without issues now, but I have not tagged
anything (too lazy) and only used searching, and so I have never had any
use for the dump/restore feature. I hope that Enric, Andrea and others
can test tagging more extensively now.
Next, there is a trivial patch for neomutt (notmuch is a configure
option), which I also have been happily using. Would that be better as
a flavor or is notmuch small enough to be a dependency of neomutt?
PS: The three zlib-related bugs in notmuch mentioned in a previous email
are now fixed upstream, so they should vanish in the next version.
notmuch.tgz
Description: Binary data
Re: wip: mail/notmuch
Olivier Taïbi writes: > Here is further progress on this port, following versions by Enric > Morales and Stuart Henderson. I hope that this does not duplicate your > efforts. builds and installs on amd64 -current.
Re: wip: mail/notmuch
Hi Olivier, On 2020-04-10 20:21, Olivier Taïbi wrote: Here is further progress on this port, following versions by Enric Morales and Stuart Henderson. I hope that this does not duplicate your efforts. Thanks a lot for providing your effort. It's great that you are working on this too, since I haven't had the time lately nor the knowledge. How is it working on your end? Any unstability so far?
Re: wip: mail/notmuch
On 2020/04/10 20:21, Olivier Taïbi wrote:
> Here is further progress on this port, following versions by Enric
> Morales and Stuart Henderson. I hope that this does not duplicate your
> efforts.
>
> There were three zlib-related bugs in notmuch:
> 1) second argument () of gzerror() cannot be NULL, as gzerror()
> will write to this address. This was causing SIGSEGV.
> 2) gzerror() cannot be called after gzclose() (use after free). This
> was causing SIGBUS.
> Patching these two got rid of all the SIGSEGV and SIGBUS in tests.
great, that's a big improvement :)
> 3) these two errors were not spotted before because no one was reaching
> them with a recent zlib (they are bugs both for the latest zlib and
> OpenBSD's zlib though). It seems that gzerror() in OpenBSD's older zlib
> will return (or rather set errnum to) Z_STREAM_END when reaching EOF,
> and this was handled as an error in notmuch's gz_getline(), when in fact
> it seems to be equivalent to Z_OK plus the indication that we reached
> EOF. Unfortunately the manual of zlib does not say much about the
> meaning of Z_* error codes, but that's the most obvious meaning (to me
> at least).
>
> I reported these bugs on the notmuch mailing list, but at least one
> issue remains. OpenBSD's zlib does not have a relatively recent feature
> of zlib: the ability to write uncompressed files, or "T" (for
> "transparent") mode. The result is equivalent to "--gzip" being the
> default (and the only choice!) for notmuch dump. To run the tests I
> patched them by adding "| zcat" after each notmuch dump which is not
> followed by --gzip, and a gunzip [...] line after each notmuch dump
> --output. I also replaced gpg by gpg2 in the tests, and imposed the
> python3 FLAVOR because it is necessary for a few tests (I may have done
> this the wrong way). There are still 9 tests that fail, but I am not
for a standalone port, use "MODPY_VERSION = ${MODPY_DEFAULT_VERSION_3}",
the FLAVOR setup is for python modules (py-* ports).
> sure that all of them are real failures. That's not counting the tests
> that Stuart Henderson forced to skip (I haven't looked at them yet) and
> the tests requiring large files (databases to be downloaded from
> notmuch's website).
>
> I see 3 ways to solve this outstanding problem:
> 1) add the "T" mode to OpenBSD's zlib, for example by merging with a
> more recent zlib from upstream. Unfortunately this seems to be
> a lot of work: see
> https://marc.info/?l=openbsd-tech=141936824422127=2
The update of the main copy in src/lib/libz has been done at least twice
(though I don't think it happened for the other copies in sys/lib/libz and
src/gnu/usr.bin/cvs - amusingly the fourth copy, in perl, is already at
1.2.11) - but not made it into the tree. The thing I remember putting
some people off was the "new" z_off64_t (10 years old today in the
public api).
But updating it is the only option that fixes the various pain / patching
/ workarounds / using bundled copies in various things in ports that
has been a problem at least going back to 2012.
> One could also just add the "T" mode.
It does help notmuch but not most other ports where we ran into
problems with old zlib (gzoffset was one of the more common ones iirc).
> 2) just accept that notmuch dump will only output gzipped data, i.e.
> only patch the man page to indicate that --gzip is the one and only
> choice. Not very nice...
Not a fan of this.
> 3) patch notmuch to remove the compression feature (--gzip), essentially
> replacing zlib by stdio. I like that because the gzip feature is not
> really useful anyway (a pipe to any compression program is simpler and
> more flexible), but that will mean merging work for each port update.
> Perhaps this could be suggested upstream?
That does sound like a good idea (and probably more acceptable to
upstream than having separate code for gzopen/gzFile/gzprintf vs
fopen/FILE */fprintf just to cope with OS with old zlib..)
It's probably not _too_ much of a maintenance burden in ports either,
certainly less of a burden than patching the tests to use zcat/gunzip.
wip: mail/notmuch
Here is further progress on this port, following versions by Enric Morales and Stuart Henderson. I hope that this does not duplicate your efforts. There were three zlib-related bugs in notmuch: 1) second argument () of gzerror() cannot be NULL, as gzerror() will write to this address. This was causing SIGSEGV. 2) gzerror() cannot be called after gzclose() (use after free). This was causing SIGBUS. Patching these two got rid of all the SIGSEGV and SIGBUS in tests. 3) these two errors were not spotted before because no one was reaching them with a recent zlib (they are bugs both for the latest zlib and OpenBSD's zlib though). It seems that gzerror() in OpenBSD's older zlib will return (or rather set errnum to) Z_STREAM_END when reaching EOF, and this was handled as an error in notmuch's gz_getline(), when in fact it seems to be equivalent to Z_OK plus the indication that we reached EOF. Unfortunately the manual of zlib does not say much about the meaning of Z_* error codes, but that's the most obvious meaning (to me at least). I reported these bugs on the notmuch mailing list, but at least one issue remains. OpenBSD's zlib does not have a relatively recent feature of zlib: the ability to write uncompressed files, or "T" (for "transparent") mode. The result is equivalent to "--gzip" being the default (and the only choice!) for notmuch dump. To run the tests I patched them by adding "| zcat" after each notmuch dump which is not followed by --gzip, and a gunzip [...] line after each notmuch dump --output. I also replaced gpg by gpg2 in the tests, and imposed the python3 FLAVOR because it is necessary for a few tests (I may have done this the wrong way). There are still 9 tests that fail, but I am not sure that all of them are real failures. That's not counting the tests that Stuart Henderson forced to skip (I haven't looked at them yet) and the tests requiring large files (databases to be downloaded from notmuch's website). I see 3 ways to solve this outstanding problem: 1) add the "T" mode to OpenBSD's zlib, for example by merging with a more recent zlib from upstream. Unfortunately this seems to be a lot of work: see https://marc.info/?l=openbsd-tech=141936824422127=2 One could also just add the "T" mode. 2) just accept that notmuch dump will only output gzipped data, i.e. only patch the man page to indicate that --gzip is the one and only choice. Not very nice... 3) patch notmuch to remove the compression feature (--gzip), essentially replacing zlib by stdio. I like that because the gzip feature is not really useful anyway (a pipe to any compression program is simpler and more flexible), but that will mean merging work for each port update. Perhaps this could be suggested upstream? On Sun, Dec 22, 2019 at 12:35:34AM +, Stuart Henderson wrote: > Enric Morales has sent a newer version to ports@, but there are still > some problems - some segfaults and hangs connected with the notmuch > code using zlib (which has been patched to work with OpenBSD's rather > old version of zlib - I don't trust these patches, in particular the > one relating to gzclose_w). There are some other problems which might > also be connected with zlib but might be something else. > > It builds and packages and some things work, but I don't think it's > ready for import until these problems are figured out. (I'm not > expecting a fully clear "make test" run, but the tests which currently > fail point at some problems that really want fixing first). > > Here's a version I've worked on a bit. Compared to the last one that > was sent to ports@, it's a newer version, simplifies some patching, > and gets more of the tests running correctly. > notmuch.tgz Description: Binary data
Re: openbsd-wip/mail/notmuch fails to compile
Hi Stuart, ports@, Carolyn, Sorry for the late reply. Thank you for working on this. The Notmuch devs fixed the use-after-free bug after you spotted it and also released 0.29.3 which I see you are already building in this new Makefile. I haven't had time to try it but will do tomorrow when I'll have the time. Enric Stuart Henderson writes: > On 2019/12/21 14:40, Carolyn Knight-Serrano wrote: >> Howdy! I'm trying to build mail/notmuch but it fails to build > > openbsd-wip is a place to work on ports (possibly collaboratively) > before submitting them when they're closer to being ready. Ports there > are often not expected to work properly. > > Enric Morales has sent a newer version to ports@, but there are still > some problems - some segfaults and hangs connected with the notmuch > code using zlib (which has been patched to work with OpenBSD's rather > old version of zlib - I don't trust these patches, in particular the > one relating to gzclose_w). There are some other problems which might > also be connected with zlib but might be something else. > > It builds and packages and some things work, but I don't think it's > ready for import until these problems are figured out. (I'm not > expecting a fully clear "make test" run, but the tests which currently > fail point at some problems that really want fixing first). > > Here's a version I've worked on a bit. Compared to the last one that > was sent to ports@, it's a newer version, simplifies some patching, > and gets more of the tests running correctly. >
Re: openbsd-wip/mail/notmuch fails to compile
On 2019/12/21 14:40, Carolyn Knight-Serrano wrote: > Howdy! I'm trying to build mail/notmuch but it fails to build openbsd-wip is a place to work on ports (possibly collaboratively) before submitting them when they're closer to being ready. Ports there are often not expected to work properly. Enric Morales has sent a newer version to ports@, but there are still some problems - some segfaults and hangs connected with the notmuch code using zlib (which has been patched to work with OpenBSD's rather old version of zlib - I don't trust these patches, in particular the one relating to gzclose_w). There are some other problems which might also be connected with zlib but might be something else. It builds and packages and some things work, but I don't think it's ready for import until these problems are figured out. (I'm not expecting a fully clear "make test" run, but the tests which currently fail point at some problems that really want fixing first). Here's a version I've worked on a bit. Compared to the last one that was sent to ports@, it's a newer version, simplifies some patching, and gets more of the tests running correctly. notmuch.tgz Description: application/tar-gz
openbsd-wip/mail/notmuch fails to compile
Howdy! I'm trying to build mail/notmuch but it fails to build |In file included from /usr/local/include/xapian.h:56: In file included from /usr/local/include/xapian/error.h:30: In file included from /usr/include/c++/v1/string:505: In file included from /usr/include/c++/v1/string_view:181: ./version:1:1: error: expected unqualified-id 0.28.3 ^ In file included from lib/database.cc:21: In file included from ./lib/database-private.h:39: In file included from /usr/local/include/xapian.h:56: In file included from /usr/local/include/xapian/error.h:30: In file included from /usr/include/c++/v1/string:518: ./version:1:1: error: expected unqualified-id 0.28.3 ^ In file included from lib/database.cc:21: In file included from ./lib/database-private.h:39: In file included from /usr/local/include/xapian.h:60: In file included from /usr/local/include/xapian/database.h:37: In file included from /usr/include/c++/v1/vector:284: ./version:1:1: error: expected unqualified-id 0.28.3 ^ In file included from lib/database.cc:21: In file included from ./lib/database-private.h:39: In file included from /usr/local/include/xapian.h:75: In file included from /usr/local/include/xapian/expanddecider.h:28: In file included from /usr/include/c++/v1/set:427: In file included from /usr/include/c++/v1/__node_handle:16: In file included from /usr/include/c++/v1/optional:159: ./version:1:1: error: expected unqualified-id 0.28.3 ^ In file included from lib/database.cc:21: In file included from ./lib/database-private.h:39: In file included from /usr/local/include/xapian.h:75: In file included from /usr/local/include/xapian/expanddecider.h:28: In file included from /usr/include/c++/v1/set:429: ./version:1:1: error: expected unqualified-id 0.28.3 ^ In file included from lib/database.cc:21: In file included from ./lib/database-private.h:39: In file included from /usr/local/include/xapian.h:77: In file included from /usr/local/include/xapian/matchspy.h:36: In file included from /usr/include/c++/v1/map:488: ./version:1:1: error: expected unqualified-id 0.28.3 ^ fatal error: too many errors emitted, stopping now [-ferror-limit=] 20 errors generated. gmake: *** [Makefile.local:197: lib/database.o] Error 1 *** Error 2 in . (/usr/ports/infrastructure/mk/bsd.port.mk:2890 '/usr/ports/pobj/notmuch-0.28.3/.build_done') *** Error 1 in . (/usr/ports/infrastructure/mk/bsd.port.mk:2090 '/usr/ports/packages/amd64/all/notmuch-0.28.3.tgz') *** Error 1 in . (/usr/ports/infrastructure/mk/bsd.port.mk:2577 '_internal-package') *** Error 1 in . (/usr/ports/infrastructure/mk/bsd.port.mk:2556 'package') *** Error 1 in . (/usr/ports/infrastructure/mk/bsd.port.mk:2111 '/var/db/pkg/notmuch-0.28.3/+CONTENTS') *** Error 1 in /usr/ports/openbsd-wip/mail/notmuch (/usr/ports/infrastructure/mk/bsd.port.mk:2556 'install') |Also the version of notmuch is outdated ||
