Re: Corrupted database (notmuch 0.31, Emacs 27.1.50, afew 3.0.1, OfflineIMAP, Python)

[Jorge P . de Morais Neto]

Em [2020-11-18 qua 15:41:43-0400], David Bremner escreveu:

> OK, so the problem is at the notmuch level.  I don't really have any
> good ideas on how to debug your issues.  There are known (apparently
> unfixable) memory management issues with the old python bindings, it
> is possible that your issues are related to that.  You should probably
> consider migrating any python scripts to the new bindings (the module
> is called notmuch2).

OK, I intend to migrate my script when time permits.  It will be no
later than December.  In the meanwhile, I have disabled afew DKIM
filtering, which was giving errors.

Regards

-- 
- 
- If an email of mine arrives at your spam box, please notify me.
- Please adopt free/libre formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: Search Case Sensitivity

[David Bremner]

Wenlong Dai  writes:

> I later realised my statement was wrong. I did not test all search terms.
> I was only testing the folder: search term.
>
> It seems folder: search is always case-sensitive and it doesn't allow wild
> cards.
> So if I want to find messages in a specific folder, I'll have to spell the
> folder name case-sensitively,
> and I'll have to either remember the full path or use regex. Is this
> understanding correct?
> If it is, then I would say it's a bit unexpected and odd.

Yes, that sounds right. folder: (and path:) searches are based on the
paths on disk.  I have to admit I don't use folder searches (or upper
case path names ;)) much, so I leave it to others to discuss whether having
case-insensitive path searches would be worthwhile. Offhand I think it
would require indexing some additional terms, and we'd want to know how
much that increased the database size before deciding.

path: prefixes do support a limited form of wildcards to support
recursive search.  In general Xapian wildcard is somewhat limited,
currently only allowing wildcards at the end of terms. I think that may
change in the future, and could be useful for notmuch queries, since
people are used to globs in paths.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: Search Case Sensitivity

[Wenlong Dai]

I later realised my statement was wrong. I did not test all search terms.
I was only testing the folder: search term.

It seems folder: search is always case-sensitive and it doesn't allow wild
cards.
So if I want to find messages in a specific folder, I'll have to spell the
folder name case-sensitively,
and I'll have to either remember the full path or use regex. Is this
understanding correct?
If it is, then I would say it's a bit unexpected and odd.

On Wed, 18 Nov 2020 at 22:28, David Bremner  wrote:

> Wenlong Dai  writes:
>
> > Based on my usage and testing, it seems to me that all notmuch searches
> are
> > case-sensitive.
> > Is this true?
> > Is there a way to do case-insensitive search?
>
> Most searches (other than boolean prefixes and regex searches) are in
> fact case insensitive. The definition of case is English-centric. Can
> you give an example of a search that is case-sensitive that you are
> expecting to be case insensitive?
>
> d
>
>

-- 
Kind Regards,
Wenlong Dai
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: Corrupted database (notmuch 0.31, Emacs 27.1.50, afew 3.0.1, OfflineIMAP, Python)

[David Bremner]

Jorge P. de Morais Neto  writes:

> xapian-check (from buster xapian-tools, which is version 1.4.11) found
> no errors.  Note that the database directory is named "compact_backup"
> because, when I tried compacting the database to fix the error, I
> invoked
> $ notmuch compact --backup=compact_backup

OK, so the problem is at the notmuch level. I don't really have any good
ideas on how to debug your issues. There are known (apparently unfixable)
memory management issues with the old python bindings, it is possible
that your issues are related to that. You should probably consider
migrating any python scripts to the new bindings (the module is called
notmuch2). We have deprecated the "notmuch" module, and will drop it
completely at some point.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: Corrupted database (notmuch 0.31, Emacs 27.1.50, afew 3.0.1, OfflineIMAP, Python)

[Jorge P . de Morais Neto]

Em [2020-11-18 qua 10:52:43-0400], David Bremner escreveu:

> "Jorge P. de Morais Neto"  writes:
>>
>> I have also saved the binary corrupted database.  If you want to see it,
>> then tell me and I may upload it to Disroot's Lufi instance.  It should
>> probably be shown to as few people as possible for the sake of my
>> privacy.
>
> Can you run xapian-check on the corrupted database?
>
> You should not have to put it anywher that will interfere with notmuch
> for that, just point xapian-check to the directory containing the backup
> copy of the database.

xapian-check (from buster xapian-tools, which is version 1.4.11) found
no errors.  Note that the database directory is named "compact_backup"
because, when I tried compacting the database to fix the error, I
invoked
$ notmuch compact --backup=compact_backup

--8<---cut here---start->8---
$ xapian-check compact_backup/
docdata:
blocksize=8K items=84 firstunused=1 revision=20879 levels=0 root=0
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=125188 firstunused=15901 revision=20879 levels=2 root=15862
B-tree checked okay
termlist table structure checked OK

postlist:
blocksize=8K items=952837 firstunused=24179 revision=20879 levels=2 root=12823
B-tree checked okay
postlist table structure checked OK

position:
blocksize=8K items=12120294 firstunused=46553 revision=20879 levels=2 root=44626
B-tree checked okay
position table structure checked OK

spelling:
Lazily created, and not yet used.

synonym:
Lazily created, and not yet used.

No errors found
--8<---cut here---end--->8---

Regards
-- 
- 
- If an email of mine arrives at your spam box, please notify me.
- Please adopt free/libre formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Please discard---Corrupted database (notmuch 0.31, Emacs 27.1.50, afew 3.0.1, OfflineIMAP, Python)

[Jorge P . de Morais Neto]

Moderator, please discard the email cited below and mentioned in the
In-Reply-To header: Message-ID 87lfezpz9t@disroot.org.  Today I
submitted another copy (with the large attachment replaced by an URL to
a file upload service) and it already wen through.

Regards

Em [2020-11-17 ter 16:19:10-0300], Jorge P. de Morais Neto escreveu:

> Hi.  I use Notmuch 0.31.2 on Emacs 27.1.50 (manually compiled on
> 2020-11-02) with matching version-pinned MELPA Stable Notmuch package on
> updated Debian buster.  I have enabled buster-proposed-updates,
> buster-updates and buster-backports.  I manually backport notmuch
> according to .  I use
> OfflineIMAP 7.3.3 (Python 2 pip), afew 3.0.1 (pip3), Bogofilter 1.2.4
> (buster) and a custom Python 3 script based on the ~notmuch~ module.
>
> Yesterday (when still on Notmuch 0.31) I noticed that, when I tagged a
> message or thread, the fido-mode completion offered many weird candidate
> tags that shouldn't exist in the database.  Also, on the Notmuch Hello
> screen the ~All tags~ section would error out.  I then dumped the
> database (~notmuch dump~) and noticed many lines associating weird tags
> to weird message ids.
>
> In almost every case, both the weird tags and the weird Message-Id
> contained uncommon characters, often ASCII control characters.  One of
> the weird lines was " -- id:8"---specifying a message with Messaged-ID
> "8" and no tags.  I tried ~notmuch show id:8~ and got an internal
> error---something like "message with document ID  has no
> thread ID".
>
> I then upgraded Notmuch to 0.31.2 and compacted the database but the
> error persisted.  I then manually cleaned up the database dump, deleted
> the ~/offlineimap/Jorge-Disroot/.notmuch/xapian/ directory, invoked
> ~notmuch new~, and ~notmuch restore~.  I checked my backups from
> 2020-11-09 (no corruption) and 2011-11-16.  That latest backup was from
> before I /noticed/ the corruption, but it was affected too.  I then
> diffed backup 2020-11-09 with backup 2020-11-16; and then backup
> 2020-11-16 with the current dump.  The diffs suggest that the error
> involved only the addition of invalid information; I suspect and hope
> that valid information was not lost.
>
> I attached my post-new Bash script and the Python 3 script it invokes.
> So you can see the weird lines I mentioned, I also attached the
> xz-compressed output of the command:
>
> diff -u notmuch_dump--manually_fixed notmuch_dump--corrupted > 
> diff_notmuch_dump__manually_fixed--corrupted
>
> I have also saved the binary corrupted database.  If you want to see it,
> then tell me and I may upload it to Disroot's Lufi instance.  It should
> probably be shown to as few people as possible for the sake of my
> privacy.
>
> Finally, my notmuch config includes the following directives (the other
> directives are probably irrelevant to you):
>
> [new]
> tags=new
> ignore=
>
> [search]
> exclude_tags=deleted;spam;trash
>
> [maildir]
> synchronize_flags=true
>
> Regards
>
> #!/usr/bin/env python3
> """Mail filter (including anti-spam) and notifier for Notmuch.
>
> Track messages classified as spam (or ham) by Bogofilter via '.bf_spam'
> (resp. '.bf_ham' ) tags.  Since afew removes the `new' tag, when
> notifying mail we track new messages with a temporary tag (option
> '--tmp' of `filter' subcommand) which we assume not to preexist in the
> database.  These tags and that added by the user to spam messages can be
> customized via command-line options or, from Python, by modifying
> module-level constants or by via function arguments.
>
> This script is potentially affected by environment variables, files and
> directories that affect afew, Bogofilter, Notmuch or (obviously)
> Python3, including:
> 1. `NOTMUCH_CONFIG' – location of Notmuch configuration file – and that
>file itself.
> 2. `BOGOFILTER_DIR' – location of Bogofilter's database directory – and
>that directory itself.
> 3. afew configuration.
>
> WISH: Accept customizable "new" flags (currently we assume "new").
> """
> # WISH: Finish documenting the exceptions possibly raised by each function
> import logging
> import sys
> import time
> from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, Namespace
> from functools import partial
> from logging import handlers
> from subprocess import PIPE, STDOUT, Popen, run
> from typing import Any, Callable, Iterable, Optional, Tuple, Union
>
> from notmuch import Database, Message
>
> # https://wiki.archlinux.org/index.php/Desktop_notifications#Python
> import gi   # isort:skip
> gi.require_version('Notify', '0.7')
> # pylint: disable=wrong-import-position
> from gi.repository import Notify  # noqa: E402 isort:skip
>
> Tags = Union[str, Iterable[str]]
>
> LOG = logging.getLogger(__name__)
> FILTER_ACTIONS = {'spam', 'general', 'notify'}
>
> # Defaults for command-line options
> BF_HAM = '.bf_ham'
> BF_SPAM = '.bf_spam'
> 

Corrupted database (notmuch 0.31, Emacs 27.1.50, afew 3.0.1, OfflineIMAP, Python)

[Jorge P. de Morais Neto]

Hi.  I use Notmuch 0.31.2 on Emacs 27.1.50 (manually compiled on
2020-11-02) with matching version-pinned MELPA Stable Notmuch package on
updated Debian buster.  I have enabled buster-proposed-updates,
buster-updates and buster-backports.  I manually backport notmuch
according to .  I use
OfflineIMAP 7.3.3 (Python 2 pip), afew 3.0.1 (pip3), Bogofilter 1.2.4
(buster) and a custom Python 3 script based on the ~notmuch~ module.

This Monday (when still on Notmuch 0.31) I noticed that, when I tagged a
message or thread, the fido-mode completion offered many weird candidate
tags that shouldn't exist in the database.  Also, on the Notmuch Hello
screen the ~All tags~ section would error out.  I then dumped the
database (~notmuch dump~) and noticed many lines associating weird tags
to weird message ids.  In almost every case, both the weird tags and the
weird Message-Id contained uncommon characters, often ASCII control
characters.

One of the weird lines was " -- id:8"---specifying a message with
Messaged-ID "8" and no tags.  Invoking ~notmuch show id:8~ yielded
internal error---something like "message with document ID 
has no thread ID".

I then upgraded Notmuch to 0.31.2 and compacted the database but the
error persisted.  I then manually cleaned up the database dump, deleted
the ~/offlineimap/Jorge-Disroot/.notmuch/xapian/ directory, invoked
~notmuch new~, and ~notmuch restore~.  I checked my backups from
2020-11-09 (not affected) and 2011-11-16.  That latest backup was from
before I /noticed/ the corruption, but sadly it was affected too.  I
then diffed the latest backup with the previous; and then the latest
backup with the current dump.  The diffs suggest that the error involved
only the addition of invalid information; I suspect and hope that valid
information was not lost.

I attached my post-new Bash script and the Python 3 script it invokes.
So you can see the weird lines I mentioned, I provide for download (URL
below) the xz-compressed output of the command:

diff -u notmuch_dump--manually_fixed notmuch_dump--corrupted > 
diff_notmuch_dump__manually_fixed--corrupted

https://upload.disroot.org/r/vSGNbFrN#8OCTbQTDKsrpRNjBsFzkv6rimPqEFE/UhGm14MypY0o=

I have also saved the binary corrupted database.  If you want to see it,
then tell me and I may upload it to Disroot's Lufi instance.  It should
probably be shown to as few people as possible for the sake of my
privacy.

Finally, my notmuch config includes the following directives (the other
directives are probably irrelevant to you):

[new]
tags=new
ignore=

[search]
exclude_tags=deleted;spam;trash

[maildir]
synchronize_flags=true

Regards


post-new
Description: Notmuch post-new hook
#!/usr/bin/env python3
"""Mail filter (including anti-spam) and notifier for Notmuch.

Track messages classified as spam (or ham) by Bogofilter via '.bf_spam'
(resp. '.bf_ham' ) tags.  Since afew removes the `new' tag, when
notifying mail we track new messages with a temporary tag (option
'--tmp' of `filter' subcommand) which we assume not to preexist in the
database.  These tags and that added by the user to spam messages can be
customized via command-line options or, from Python, by modifying
module-level constants or via function arguments.

This script is potentially affected by environment variables, files and
directories that affect afew, Bogofilter, Notmuch or (obviously)
Python3, including:
1. `NOTMUCH_CONFIG'---location of Notmuch configuration file---and that
   file itself.
2. `BOGOFILTER_DIR'---location of Bogofilter's database directory---and
   that directory itself.
3. afew configuration.

WISH: Accept customizable "new" flags (currently we assume "new").
"""
# WISH: Finish documenting the exceptions possibly raised by each function
import logging
import sys
import time
from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, Namespace
from functools import partial
from logging import handlers
from subprocess import PIPE, STDOUT, Popen, run
from typing import Any, Callable, Iterable, Optional, Tuple, Union

from notmuch import Database, Message

# https://wiki.archlinux.org/index.php/Desktop_notifications#Python
import gi   # isort:skip
gi.require_version('Notify', '0.7')
# pylint: disable=wrong-import-position
from gi.repository import Notify  # noqa: E402 isort:skip

Tags = Union[str, Iterable[str]]

LOG = logging.getLogger(__name__)
FILTER_ACTIONS = {'spam', 'general', 'notify'}

# Defaults for command-line options
BF_HAM = '.bf_ham'
BF_SPAM = '.bf_spam'
USER_SPAM = 'spam'
TMP = '_simplemuch_tmp'


class SimplemuchError(Exception):
"""Base class for simplemuch exception classes"""


class NotmuchDatabaseNeedsUpgradeError(SimplemuchError):
"""needs_upgrade() returned True."""


# WISH Capture more information, e.g. return code and command line
class BogofilterError(SimplemuchError):
"""Error from Bogofilter"""


# def teste_mypy(i: int) 

Re: Corrupted database (notmuch 0.31, Emacs 27.1.50, afew 3.0.1, OfflineIMAP, Python)

[David Bremner]

"Jorge P. de Morais Neto"  writes:
>
> I have also saved the binary corrupted database.  If you want to see it,
> then tell me and I may upload it to Disroot's Lufi instance.  It should
> probably be shown to as few people as possible for the sake of my
> privacy.

Can you run xapian-check on the corrupted database?

You should not have to put it anywher that will interfere with notmuch
for that, just point xapian-check to the directory containing the backup
copy of the database.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: Search Case Sensitivity

[David Bremner]

Wenlong Dai  writes:

> Based on my usage and testing, it seems to me that all notmuch searches are
> case-sensitive.
> Is this true?
> Is there a way to do case-insensitive search?

Most searches (other than boolean prefixes and regex searches) are in
fact case insensitive. The definition of case is English-centric. Can
you give an example of a search that is case-sensitive that you are
expecting to be case insensitive?

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Search Case Sensitivity

[Wenlong Dai]

Hi Guys,

I did a bit of Google Search and reading of notmuch man pages.
I'm surprised to find no mention of this.

Based on my usage and testing, it seems to me that all notmuch searches are
case-sensitive.
Is this true?
Is there a way to do case-insensitive search?

Thanks
Dave
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: emacs: error decrypting s/mime

[Alexander Adolf]

Hello David,

first of all, many thanks for not giving up on this one!

David Bremner  writes:

> [...]
> I think this might be a deeper issue. Looking at the structure of
>
>   test/corpora/protected-headers/smime-sign+enc.eml
>
> it looks like there is an application/pkcs-7 part for the outer
> container with an encstatus, and one inside that (with the same mime
> type) with a sigstatus. So maybe the right thing is to just ignore
> missing encstatus?
> [...]

Conceptually, a typical s/mime message looks like this:
--
Received:
From:
To:
Subject:
Date:
MIME-Version: 1.0
Content-Disposition: attachment; filename="smime.p7m"
Content-Type: application/x-pkcs7-mime; smime-type=enveloped-data;
name="smime.p7m"
[more headers (opt.)]
  
gibberish (the CMS)
--

After decrypting the gibberish (CMS), you get a new mime tree structure,
in which the top-level entity can be a multipart/signed (most often), or
a message/rfc822 (sometimes), or something else (rarely seen) [1].

[1] https://tools.ietf.org/html/rfc8551

I.e. you decrypt, and further mime parts appear. No assumptions should
be made about the tree's structure. Quoting from [1]:

 Begin Quote -
3.1.  Preparing the MIME Entity for Signing, Enveloping, or Compressing

   S/MIME is used to secure MIME entities.  A MIME message is composed
   of a MIME header and a MIME body.  A body can consist of a single
   MIME entity or a tree of MIME entities (rooted with a multipart).
   S/MIME can be used to secure either a single MIME entity or a tree of
   MIME entities.  These entities can be in locations other than the
   root.  S/MIME can be applied multiple times to different entities in
   a single message. [...]
- End Quote --

After decrypting your "outer" container (the CMS), the result is a mime
tree, i.e. should start with "Content-Type:". Standard mime tree
processing should be applied (recursively).

The bodypart handler error message shows this "embedded (inner) mime
tree", and in my case the top-level entity is a multipart/signed. You
wrote "the outer container with an encstatus, and one inside that (with
the same mime type) with a sigstatus". So it seems that at that point
you have access to the root of the "inner tree" (multipart/signed, you
can access the sigstatus), but the content-type information is from the
"outer" container (the CMS) still. Perhaps the recursive mime re-parsing
after the decryption is not happening?


More new questions than answers...

  --alexander
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org