Encodings
Hiya,
I noticed that commit 687366b920caa5de6ea0b66b70cf2a11e5399f7b
breaks things with Database.get_all_tags:
-->%-
AttributeErrorTraceback (most recent call last)
/home/pazz/projects/alot/ in ()
/usr/local/lib/python2.7/dist-packages/notmuch/tag.pyc in next(self)
86 # No need to call nmlib.notmuch_tags_valid(self._tags);
87 # Tags._get safely returns None, if there is no more valid tag.
---> 88 tag = Tags._get(self._tags).decode('utf-8')
89 if tag is None:
90 self._tags = None
AttributeError: 'NoneType' object has no attribute 'decode'
%<---
The reason is that the Tags.next() tries to decode before it tests if tag is
None.
Now, we _could_ apply a patch like this one here:
-->%-
diff --git a/bindings/python/notmuch/tag.py b/bindings/python/notmuch/tag.py
index 65a9118..2ae670d 100644
--- a/bindings/python/notmuch/tag.py
+++ b/bindings/python/notmuch/tag.py
@@ -85,12 +85,12 @@ class Tags(object):
raise NotmuchError(STATUS.NOT_INITIALIZED)
# No need to call nmlib.notmuch_tags_valid(self._tags);
# Tags._get safely returns None, if there is no more valid tag.
-tag = Tags._get(self._tags).decode('utf-8')
+tag = Tags._get(self._tags)
if tag is None:
self._tags = None
raise StopIteration
nmlib.notmuch_tags_move_to_next(self._tags)
-return tag
+return tag.decode('utf-8')
def __nonzero__(self):
"""Implement bool(Tags) check that can be repeatedly used
---%<-
But as Carl sais, we cannot guarantee that a tag is utf8 encoded anyway.
So i'd suggest we just revore the commit in question.
best,
/p
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL:
<http://notmuchmail.org/pipermail/notmuch/attachments/20110712/ffd88188/attachment.pgp>
Slowness (search opens every email file?)
Quoth Istvan Marko on Jul 12 at 6:55 pm: > Austin Clements writes: > > > Istvan, did you make any progress on this patch since the last > > version? I seem to recall it just needed general cleanup (code style > > and such) and a better answer for backwards compatibility (the > > unfortunate " " thing). > > I have been using the version that encodes empty headers to " " but > another way to handle this is to simply not set a VALUE for empty > headers and then fall back to the original parsing method for these > messages. Emails without from/subject/message-id headers are not very > common so perhaps this is a good compromise. > > Below is the patch without the " " hack. Ah, clever. I was going to suggest adding another value to indicate the presence or absence of these Xapian values, but I like your solution better. The only downside I can think of is that it might not extend to other headers if we store more header values in the database in the future. I'd say this patch looks good other than coding style - Tab indentation - /* */ comments, starting with a capital letter - Space between function name and open paren - Space after comma in argument lists - Spaces around assignment operator
[python] pep8
hi all, I just read I'm a bit late with this as Sebastian is going on holiday for the next couple of weeks, but here it is anyway. The attached patches make notmuch's python bindings pep8 compliant. most of it is just chopping down long lines and removin trailing whitespaces. best, /p -- next part -- A non-text attachment was scrubbed... Name: 0002-pep8-compliance-for-message.py.patch Type: text/x-diff Size: 20551 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110712/bae74832/attachment-0007.patch> -- next part -- A non-text attachment was scrubbed... Name: 0003-pep8-compliance-for-database.py.patch Type: text/x-diff Size: 23602 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110712/bae74832/attachment-0008.patch> -- next part -- A non-text attachment was scrubbed... Name: 0004-pep8-compliance-for-filename.py.patch Type: text/x-diff Size: 2828 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110712/bae74832/attachment-0009.patch> -- next part -- A non-text attachment was scrubbed... Name: 0005-pep8-compliance-for-globals.py.patch Type: text/x-diff Size: 3253 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110712/bae74832/attachment-0010.patch> -- next part -- A non-text attachment was scrubbed... Name: 0006-pep8-compliance-for-tag.py.patch Type: text/x-diff Size: 2671 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110712/bae74832/attachment-0011.patch> -- next part -- A non-text attachment was scrubbed... Name: 0007-pep8-compliance-for-thread.py.patch Type: text/x-diff Size: 10355 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110712/bae74832/attachment-0012.patch> -- next part -- A non-text attachment was scrubbed... Name: 0008-pep8-compliance-for-__init__.py.patch Type: text/x-diff Size: 1517 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110712/bae74832/attachment-0013.patch> -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110712/bae74832/attachment-0001.pgp>
Encodings
Hi! As discussed on irc, if notmuch stores header values in utf8, its safe to decode them to unicode instances here. best, /p On Mon, Jul 11, 2011 at 08:03:38AM -0700, Carl Worth wrote: > On Mon, 11 Jul 2011 16:04:17 +0200, Sebastian Spaeth SSpaeth.de> wrote: > > The answer is that things are very implicit. notmuch.h speaks of > > strings but never mentions encodings > > Much of this was intentional on my part. > > For example, I intentionally avoided restrictions on what could be > stored as a tag in the database, (other than the terminating character > implied by "string" of course). > > > So, can be document what encoding we are expected to pass in the various > > APIs > > Yes, let's clarify documentation wherever we need to. > > > For some of the stuff we read directly from the files, eg > > arbitrary headers, we can probably be least sure > > The headers should be decoded to utf-8, (via > g_mime_utils_header_decode_text), before being stored in the database. > > > but are e.g. the returned tags always utf-8? > > No. The tag data is returned exactly as the user presented it. > > > I would love to make the python bindings use unicode() instances in > > cases where we can be sure to actually receive utf-8 encoded strings. > > > > Encodings make my brain hurt. Unfortunately one cannot simply ignore > > them. > > I think a lot of the pain here is due to some bad design decisions in > python itself. Of course, my saying that doesn't make things any easier > for you. > > But do tell me what more we can do to clarify behavior or documentation. > > -Carl > > -- > carl.d.worth at intel.com > ___ > notmuch mailing list > notmuch at notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch -- next part -- A non-text attachment was scrubbed... Name: 0001-unicode-return-value-for-Message.get_header.patch Type: text/x-diff Size: 1728 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110712/62f17e9c/attachment.patch> -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110712/62f17e9c/attachment.pgp>
Slowness (search opens every email file?)
On Tue, Jul 12, 2011 at 4:24 PM, Jason Woofenden wrote: > On 2011-07-11 06:13PM, Austin Clements wrote: >> On Mon, Jul 11, 2011 at 3:07 PM, Jason Woofenden >> wrote: >> > notmuch search tag:foo is slow! >> > >> > (when my e-mail files are not already in the disk cache) >> > >> > I saw on my activity monitor applet that it was using mostly i/o, >> > and started to wonder if it was opening every e-mail. I little work >> > with strace and searching revealed that this command was opening >> > many many e-mail files from my maildir(s). I spent a little while >> > digging around in the notmuch source, and didn't see where it was >> > opening the email files. >> >> It is opening every file to get a few headers to display in the search >> output. ?Istvan Marko sent an experimental patch to store these >> headers in the database a while ago, though as far as I know there >> hasn't been any progress cleaning it up for inclusion: >> id:"m3sjsv2kw2.fsf at zsu.kismala.com" . > > Cool. I suspected it was reading for header files. > > I googled the id and found this patchwork link: > > http://patchwork.notmuchmail.org/patch/947/ > > (I didn't see any way to ask mailman for a message id.) Here's the full thread from Nabble: http://notmuch.198994.n3.nabble.com/storing-From-and-Subject-in-xapian-td2901262.html > Does someone want to work at this soon or should I try my hand at > it? Istvan, did you make any progress on this patch since the last version? I seem to recall it just needed general cleanup (code style and such) and a better answer for backwards compatibility (the unfortunate " " thing).
Slowness (search opens every email file?)
Austin Clements writes: > I'd say this patch looks good other than coding style > - Tab indentation > - /* */ comments, starting with a capital letter > - Space between function name and open paren > - Space after comma in argument lists > - Spaces around assignment operator Thanks, fixed the ones I see: -- next part -- A non-text attachment was scrubbed... Name: notmuch-value3.patch Type: text/x-patch Size: 3035 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110712/b08ef92b/attachment.bin> -- next part -- -- Istvan
Slowness (search opens every email file?)
On 2011-07-11 06:13PM, Austin Clements wrote: > On Mon, Jul 11, 2011 at 3:07 PM, Jason Woofenden > wrote: > > notmuch search tag:foo is slow! > > > > (when my e-mail files are not already in the disk cache) > > > > I saw on my activity monitor applet that it was using mostly i/o, > > and started to wonder if it was opening every e-mail. I little work > > with strace and searching revealed that this command was opening > > many many e-mail files from my maildir(s). I spent a little while > > digging around in the notmuch source, and didn't see where it was > > opening the email files. > > It is opening every file to get a few headers to display in the search > output. Istvan Marko sent an experimental patch to store these > headers in the database a while ago, though as far as I know there > hasn't been any progress cleaning it up for inclusion: > id:"m3sjsv2kw2.fsf at zsu.kismala.com" . Cool. I suspected it was reading for header files. I googled the id and found this patchwork link: http://patchwork.notmuchmail.org/patch/947/ (I didn't see any way to ask mailman for a message id.) Does someone want to work at this soon or should I try my hand at it? Thanks, - Jason
Slowness (search opens every email file?)
On 2011-07-11 10:58PM, Patrick Totzke wrote: > Hi Jason, > On Mon, Jul 11, 2011 at 03:07:21PM -0400, Jason Woofenden wrote: > > notmuch search tag:foo is slow! > > > yes, i've just used the vim ui for the first time and i agreee, its sluggish, > searching for * takes a while. It's not the vim ui that's the bottleneck. The underlying notmuch search command is slow. I gave examples in my last e-mail: >> notmuch search tag:foo took 0.5 seconds for 32 threads >> >> notmuch search foo took 6.4 seconds for 130 threads > > (when my e-mail files are not already in the disk cache) > > > > I saw on my activity monitor applet that it was using mostly i/o, > > and started to wonder if it was opening every e-mail. I little work > > with strace and searching revealed that this command was opening > > many many e-mail files from my maildir(s). I spent a little while > > digging around in the notmuch source, and didn't see where it was > > opening the email files. > > I cannot reproduce this. I'm no expert, but at least the output of > strace vim -c ":NotMuch" 2>log :NotMuch just shows the mailboxes. that's fast. It's showing the contents that is slow. And it's not vim, it's because the notmuch command is slow. Try this in a terminal: strace notmuch search tag:flagged 2>&1 | grep 'open(.*/cur/' Of course change the tag if you don't have flagged messages. > A quick browse through notmuch.vim tells me that > 1) it doesn't use notmuchs json output, i think it should, as iirc > this api is considered 'more stable' and is easier to parse than the > default output. More importantly, I like this idea. I did some work earlier on improving the message parsing in the vim ui. Might be better to use the json. I'll look into json parsing in vim. > 2) the output of notmuch is copied into a list. This will of course > be slow if your query matches a lot of messages. Could this be done > by asyncronously writing to the buffer somehow? I'm pretty sure vim doesn't do asynchronous anything. That came up in a vim vs emacs article I read. Take care, - Jason
[PATCH 2/3] tags_to_maildir_flags: Cleanup double assignement
The for loop right after already does the job.
Signed-off-by: Louis Rilling
---
lib/message.cc |2 --
1 files changed, 0 insertions(+), 2 deletions(-)
diff --git a/lib/message.cc b/lib/message.cc
index cf651e5..b1b2942 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -1172,8 +1172,6 @@ _new_maildir_filename (void *ctx,
if (info == NULL) {
info = filename + strlen(filename);
} else {
- flags = info + 3;
-
/* Loop through existing flags in filename. */
for (flags = info + 3, last_flag = 0;
*flags;
--
1.7.2.5
[PATCH 2/2] tags_to_maildir_flags: Don't rename if no flags change
On 11/07/11 20:03 -0400, Austin Clements wrote:
> > > The convention in notmuch is to use notmuch_bool_t, TRUE, and FALSE
> > > (though, admittedly, I don't know why; avoiding C99-isms?)
> >
> > And bool is already used at another place in message.cc:
> >
> > struct maildir_flag_tag {
> > char flag;
> > const char *tag;
> > bool inverse;
> > };
> >
> > IIUC it should be changed to notmuch_bool_t too.
>
> Yes, I suppose it should (something slipped by cworth's eagle-eyed
> reviews!). Though that appears to be the sole use of bool in all of
> libnotmuch.
I wonder if this is due to incompatible definitions of type bool in C99 and
C++. In that case this is probably harmless since struct maildir_flag_tag is
only visible from message.cc. Anyway, I'm sending a conversion patch together
with the updated series to make it clearer for Carl.
Thanks,
Louis
[David Bremner] problems with symbol hiding on Debian Squeeze i386
Hello David, [ only cc'ing the list as I'm not subscribed and am not sure my message will be accepted - please CC me in all your answers ] Le lundi 11 juil. 2011 ? 21:43:53 (+0200 CEST), David Bremner a ?crit?: > Julien Valroth reported on IRC that he had problems with the initial > call of 'notmuch new' crashing with > > terminate called after throwing an instance of 'Xapian::InvalidArgumentError' > > I confirmed that the following quick and dirty patch seems to fix the problem. I have tested your patch and built a local backport of the 0.6 Debian package for Squeeze, and confirm everything seems to be OK now. I have been able to build and update the index, and tested a few searches. Thank you very much for working on this issue. Cheers, Julien -- .''`. Julien Valroff ~ ~ : :' : Debian Developer & Free software contributor `. `'` http://www.kirya.net/ `- 4096R/ E1D8 5796 8214 4687 E416 948C 859F EF67 258E 26B1 -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110712/b0f36745/attachment.pgp>
[PATCH v2] tags_to_maildir_flags: Don't rename if no flags change
notmuch_message_tags_to_maildir_flags() unconditionally moves messages from
maildir directory "new/" to maildir directory "cur/", which makes messages lose
their "new" status in the MUA. However some users want to keep this "new"
status after, for instance, an auto-tagging of new messages.
However, as Austin mentioned and according to the maildir specification,
messages living in "new/" are not allowed to have flags, even if mutt allows it
to happen. For this reason, this patch prevents moving messages from "new/" to
"cur/", only if no flags have to be changed. It's hopefully enough to satisfy
mutt (and maybe other MUAs showing the "new" status) users checking the "new"
status.
Changelog:
* v2: Fix bool type as well as NULL returned despite having no errors (Austin
Clements)
Signed-off-by: Louis Rilling
---
lib/message.cc | 12 +++-
1 files changed, 11 insertions(+), 1 deletions(-)
diff --git a/lib/message.cc b/lib/message.cc
index 64b6cf8..3f8c4ba 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -1139,7 +1139,7 @@ _get_maildir_flag_actions (notmuch_message_t *message,
* compute the new maildir filename.
*
* If the existing filename is in the directory "new", the new
- * filename will be in the directory "cur".
+ * filename will be in the directory "cur", unless no flags are changed.
*
* After a sequence of ":2," in the filename, any subsequent
* single-character flags will be added or removed according to the
@@ -1162,6 +1162,7 @@ _new_maildir_filename (void *ctx,
char *filename_new, *dir;
char flag_map[128];
int flags_in_map = 0;
+notmuch_bool_t flags_changed = FALSE;
unsigned int i;
char *s;
@@ -1202,6 +1203,7 @@ _new_maildir_filename (void *ctx,
if (flag_map[flag] == 0) {
flag_map[flag] = 1;
flags_in_map++;
+ flags_changed = TRUE;
}
}
@@ -1210,9 +1212,17 @@ _new_maildir_filename (void *ctx,
if (flag_map[flag]) {
flag_map[flag] = 0;
flags_in_map--;
+ flags_changed = TRUE;
}
}
+/* No need to rename. Messages in new/ can be kept in new/.
+ * Note: We don't even try to fix buggy messages having flags and living in
+ * new/. It's not our business.
+ */
+if (!flags_changed)
+ return talloc_strdup (ctx, filename);
+
filename_new = (char *) talloc_size (ctx,
info - filename +
strlen (":2,") + flags_in_map + 1);
--
1.7.2.5
[PATCH 2/2] tags_to_maildir_flags: Don't rename if no flags change
On 11/07/11 16:07 -0400, Austin Clements wrote:
> I worry that this may compound the confusion caused by mutt's handling
> of the new flag, but I suppose people aren't likely to manipulate any
> of the other maildir-synchronized flags without also marking the
> message as seen.
Even if they don't mark the message as seen, any flag changed would move the
message to cur/. The only buggy behavior would be from mutt, with the bug you
mentioned about mutt putting messages with flags back to new/.
> At any rate, the change is certainly correct
> technically. A few nits below.
They should be addressed by the follow-up patch. Just a comment below.
>
> Quoth Louis Rilling on Jul 11 at 4:36 pm:
> > notmuch_message_tags_to_maildir_flags() unconditionally moves messages from
> > maildir directory "new/" to maildir directory "cur/", which makes messages
> > lose
> > their "new" status in the MUA. However some users want to keep this "new"
> > status after, for instance, an auto-tagging of new messages.
> >
> > However, as Austin mentioned and according to the maildir specification,
> > messages living in "new/" are not allowed to have flags, even if mutt
> > allows it
> > to happen. For this reason, this patch prevents moving messages from "new/"
> > to
> > "cur/", only if no flags have to be changed. It's hopefully enough to
> > satisfy
> > mutt (and maybe other MUAs showing the "new" status) users checking the
> > "new"
> > status.
> >
> > Signed-off-by: Louis Rilling
> > ---
> > lib/message.cc | 12 +++-
> > 1 files changed, 11 insertions(+), 1 deletions(-)
> >
> > diff --git a/lib/message.cc b/lib/message.cc
> > index 64b6cf8..131d99b 100644
> > --- a/lib/message.cc
> > +++ b/lib/message.cc
> > @@ -1139,7 +1139,7 @@ _get_maildir_flag_actions (notmuch_message_t *message,
> > * compute the new maildir filename.
> > *
> > * If the existing filename is in the directory "new", the new
> > - * filename will be in the directory "cur".
> > + * filename will be in the directory "cur", unless no flags are changed.
> > *
> > * After a sequence of ":2," in the filename, any subsequent
> > * single-character flags will be added or removed according to the
> > @@ -1162,6 +1162,7 @@ _new_maildir_filename (void *ctx,
> > char *filename_new, *dir;
> > char flag_map[128];
> > int flags_in_map = 0;
> > +bool flags_changed = false;
>
> The convention in notmuch is to use notmuch_bool_t, TRUE, and FALSE
> (though, admittedly, I don't know why; avoiding C99-isms?)
And bool is already used at another place in message.cc:
struct maildir_flag_tag {
char flag;
const char *tag;
bool inverse;
};
IIUC it should be changed to notmuch_bool_t too.
>
> > unsigned int i;
> > char *s;
> >
> > @@ -1202,6 +1203,7 @@ _new_maildir_filename (void *ctx,
> > if (flag_map[flag] == 0) {
> > flag_map[flag] = 1;
> > flags_in_map++;
> > + flags_changed = true;
> > }
> > }
> >
> > @@ -1210,9 +1212,17 @@ _new_maildir_filename (void *ctx,
> > if (flag_map[flag]) {
> > flag_map[flag] = 0;
> > flags_in_map--;
> > + flags_changed = true;
> > }
> > }
> >
> > +/* No need to rename. Messages in new/ can be kept in new/.
> > + * Note: We don't even try to fix buggy messages having flags and
> > living in
> > + * new/. It's not our business.
> > + */
> > +if (!flags_changed)
> > + return NULL;
> > +
>
> NULL generally indicates an error in notmuch and is currently used
> that way in _new_maildir_filename, so even though the caller currently
> doesn't really care, I'd lean against overloading it to indicate that
> the filename doesn't need to change. Despite the slight inefficiency,
> I would recommend returning talloc_strdup (ctx, filename).
Ok.
Thanks,
Louis
>
> > filename_new = (char *) talloc_size (ctx,
> > info - filename +
> > strlen (":2,") + flags_in_map + 1);
Re: [PATCH 2/2] tags_to_maildir_flags: Don't rename if no flags change
On 11/07/11 20:03 -0400, Austin Clements wrote:
The convention in notmuch is to use notmuch_bool_t, TRUE, and FALSE
(though, admittedly, I don't know why; avoiding C99-isms?)
And bool is already used at another place in message.cc:
struct maildir_flag_tag {
char flag;
const char *tag;
bool inverse;
};
IIUC it should be changed to notmuch_bool_t too.
Yes, I suppose it should (something slipped by cworth's eagle-eyed
reviews!). Though that appears to be the sole use of bool in all of
libnotmuch.
I wonder if this is due to incompatible definitions of type bool in C99 and
C++. In that case this is probably harmless since struct maildir_flag_tag is
only visible from message.cc. Anyway, I'm sending a conversion patch together
with the updated series to make it clearer for Carl.
Thanks,
Louis
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 0/3 v3] lib: Maildir flags synchronization fixes
Hello Carl, Here is the updated series of fixes I have around maildir flags synchronization. The first two patches are just cleanups that can be applied independently. The intent for the third patch (detailed in the commit log) is to allow mutt users to keep using the new status, as long as notmuch can respect the maildir specification. Changelog: * v3: Added patch to kill the last usage of C++ type bool * v2: Fix bool type as well as NULL returned despite having no errors (Austin Clements) Thanks, Louis Louis Rilling (3): lib: Kill last usage of C++ type bool tags_to_maildir_flags: Cleanup double assignement tags_to_maildir_flags: Don't rename if no flags change lib/message.cc | 26 +- 1 files changed, 17 insertions(+), 9 deletions(-) -- 1.7.2.5 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 1/3] lib: Kill last usage of C++ type bool
Signed-off-by: Louis Rilling l.rill...@av7.net
---
lib/message.cc | 12 ++--
1 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/lib/message.cc b/lib/message.cc
index d993cde..cf651e5 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -49,16 +49,16 @@ struct visible _notmuch_message {
struct maildir_flag_tag {
char flag;
const char *tag;
-bool inverse;
+notmuch_bool_t inverse;
};
/* ASCII ordered table of Maildir flags and associated tags */
static struct maildir_flag_tag flag2tag[] = {
-{ 'D', draft, false},
-{ 'F', flagged, false},
-{ 'P', passed, false},
-{ 'R', replied, false},
-{ 'S', unread, true }
+{ 'D', draft, FALSE},
+{ 'F', flagged, FALSE},
+{ 'P', passed, FALSE},
+{ 'R', replied, FALSE},
+{ 'S', unread, TRUE }
};
/* We end up having to call the destructor explicitly because we had
--
1.7.2.5
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 2/3] tags_to_maildir_flags: Cleanup double assignement
The for loop right after already does the job.
Signed-off-by: Louis Rilling l.rill...@av7.net
---
lib/message.cc |2 --
1 files changed, 0 insertions(+), 2 deletions(-)
diff --git a/lib/message.cc b/lib/message.cc
index cf651e5..b1b2942 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -1172,8 +1172,6 @@ _new_maildir_filename (void *ctx,
if (info == NULL) {
info = filename + strlen(filename);
} else {
- flags = info + 3;
-
/* Loop through existing flags in filename. */
for (flags = info + 3, last_flag = 0;
*flags;
--
1.7.2.5
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: Slowness (search opens every email file?)
On 2011-07-11 10:58PM, Patrick Totzke wrote: Hi Jason, On Mon, Jul 11, 2011 at 03:07:21PM -0400, Jason Woofenden wrote: notmuch search tag:foo is slow! yes, i've just used the vim ui for the first time and i agreee, its sluggish, searching for * takes a while. It's not the vim ui that's the bottleneck. The underlying notmuch search command is slow. I gave examples in my last e-mail: notmuch search tag:foo took 0.5 seconds for 32 threads notmuch search foo took 6.4 seconds for 130 threads (when my e-mail files are not already in the disk cache) I saw on my activity monitor applet that it was using mostly i/o, and started to wonder if it was opening every e-mail. I little work with strace and searching revealed that this command was opening many many e-mail files from my maildir(s). I spent a little while digging around in the notmuch source, and didn't see where it was opening the email files. I cannot reproduce this. I'm no expert, but at least the output of strace vim -c :NotMuch 2log :NotMuch just shows the mailboxes. that's fast. It's showing the contents that is slow. And it's not vim, it's because the notmuch command is slow. Try this in a terminal: strace notmuch search tag:flagged 21 | grep 'open(.*/cur/' Of course change the tag if you don't have flagged messages. A quick browse through notmuch.vim tells me that 1) it doesn't use notmuchs json output, i think it should, as iirc this api is considered 'more stable' and is easier to parse than the default output. More importantly, I like this idea. I did some work earlier on improving the message parsing in the vim ui. Might be better to use the json. I'll look into json parsing in vim. 2) the output of notmuch is copied into a list. This will of course be slow if your query matches a lot of messages. Could this be done by asyncronously writing to the buffer somehow? I'm pretty sure vim doesn't do asynchronous anything. That came up in a vim vs emacs article I read. Take care, - Jason ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Slowness (search opens every email file?)
On 2011-07-11 06:13PM, Austin Clements wrote: On Mon, Jul 11, 2011 at 3:07 PM, Jason Woofenden ja...@jasonwoof.com wrote: notmuch search tag:foo is slow! (when my e-mail files are not already in the disk cache) I saw on my activity monitor applet that it was using mostly i/o, and started to wonder if it was opening every e-mail. I little work with strace and searching revealed that this command was opening many many e-mail files from my maildir(s). I spent a little while digging around in the notmuch source, and didn't see where it was opening the email files. It is opening every file to get a few headers to display in the search output. Istvan Marko sent an experimental patch to store these headers in the database a while ago, though as far as I know there hasn't been any progress cleaning it up for inclusion: id:m3sjsv2kw2@zsu.kismala.com . Cool. I suspected it was reading for header files. I googled the id and found this patchwork link: http://patchwork.notmuchmail.org/patch/947/ (I didn't see any way to ask mailman for a message id.) Does someone want to work at this soon or should I try my hand at it? Thanks, - Jason ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Encodings
Hi!
As discussed on irc, if notmuch stores header values in utf8,
its safe to decode them to unicode instances here.
best,
/p
On Mon, Jul 11, 2011 at 08:03:38AM -0700, Carl Worth wrote:
On Mon, 11 Jul 2011 16:04:17 +0200, Sebastian Spaeth sebast...@sspaeth.de
wrote:
The answer is that things are very implicit. notmuch.h speaks of
strings but never mentions encodings
Much of this was intentional on my part.
For example, I intentionally avoided restrictions on what could be
stored as a tag in the database, (other than the terminating character
implied by string of course).
So, can be document what encoding we are expected to pass in the various
APIs
Yes, let's clarify documentation wherever we need to.
For some of the stuff we read directly from the files, eg
arbitrary headers, we can probably be least sure
The headers should be decoded to utf-8, (via
g_mime_utils_header_decode_text), before being stored in the database.
but are e.g. the returned tags always utf-8?
No. The tag data is returned exactly as the user presented it.
I would love to make the python bindings use unicode() instances in
cases where we can be sure to actually receive utf-8 encoded strings.
Encodings make my brain hurt. Unfortunately one cannot simply ignore
them.
I think a lot of the pain here is due to some bad design decisions in
python itself. Of course, my saying that doesn't make things any easier
for you.
But do tell me what more we can do to clarify behavior or documentation.
-Carl
--
carl.d.wo...@intel.com
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
From 988a9832d714dfa0f91b2b1185a50acb4a6ca4b5 Mon Sep 17 00:00:00 2001
From: pazz patricktot...@gmail.com
Date: Tue, 12 Jul 2011 19:47:39 +0100
Subject: [PATCH 1/8] unicode return value for Message.get_header()
As discussed in IRC, notmuch recodes mailheaders to
utf-8, so we can safely decode them into unicode instances.
---
bindings/python/notmuch/message.py |8 +---
1 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/bindings/python/notmuch/message.py b/bindings/python/notmuch/message.py
index 763d2c6..4a43a88 100644
--- a/bindings/python/notmuch/message.py
+++ b/bindings/python/notmuch/message.py
@@ -379,14 +379,16 @@ class Message(object):
:param header: The name of the header to be retrieved.
It is not case-sensitive (TODO: confirm).
-:type header: str
-:returns: The header value as string
+:type header: str or unicode instance
+:returns: The header value as a unicode string
:exception: :exc:`NotmuchError`
* STATUS.NOT_INITIALIZED if the message
is not initialized.
* STATUS.NULL_POINTER, if no header was found
+if isinstance(header, unicode):
+header = header.encode('utf-8')
if self._msg is None:
raise NotmuchError(STATUS.NOT_INITIALIZED)
@@ -394,7 +396,7 @@ class Message(object):
header = Message._get_header (self._msg, header)
if header == None:
raise NotmuchError(STATUS.NULL_POINTER)
-return header
+return header.decode('utf-8')
def get_filename(self):
Returns the file path of the message file
--
1.7.4.1
signature.asc
Description: Digital signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Get mboxes from mailman [Was: Slowness (search opens every email file?)]
Hello, On Tue, Jul 12, 2011 at 04:24:59PM -0400, Jason Woofenden wrote: (I didn't see any way to ask mailman for a message id.) It's possible to let mailman offer mbox downloads parallel to the Gzip'd Text files. You need to set PUBLIC_MBOX = Yes; in your mailman config and restart mailman (i.e. mailmanctl restart). Best regards Uwe ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Encodings
Hiya,
I noticed that commit 687366b920caa5de6ea0b66b70cf2a11e5399f7b
breaks things with Database.get_all_tags:
--%-
AttributeErrorTraceback (most recent call last)
/home/pazz/projects/alot/ipython console in module()
/usr/local/lib/python2.7/dist-packages/notmuch/tag.pyc in next(self)
86 # No need to call nmlib.notmuch_tags_valid(self._tags);
87 # Tags._get safely returns None, if there is no more valid tag.
--- 88 tag = Tags._get(self._tags).decode('utf-8')
89 if tag is None:
90 self._tags = None
AttributeError: 'NoneType' object has no attribute 'decode'
%---
The reason is that the Tags.next() tries to decode before it tests if tag is
None.
Now, we _could_ apply a patch like this one here:
--%-
diff --git a/bindings/python/notmuch/tag.py b/bindings/python/notmuch/tag.py
index 65a9118..2ae670d 100644
--- a/bindings/python/notmuch/tag.py
+++ b/bindings/python/notmuch/tag.py
@@ -85,12 +85,12 @@ class Tags(object):
raise NotmuchError(STATUS.NOT_INITIALIZED)
# No need to call nmlib.notmuch_tags_valid(self._tags);
# Tags._get safely returns None, if there is no more valid tag.
-tag = Tags._get(self._tags).decode('utf-8')
+tag = Tags._get(self._tags)
if tag is None:
self._tags = None
raise StopIteration
nmlib.notmuch_tags_move_to_next(self._tags)
-return tag
+return tag.decode('utf-8')
def __nonzero__(self):
Implement bool(Tags) check that can be repeatedly used
---%-
But as Carl sais, we cannot guarantee that a tag is utf8 encoded anyway.
So i'd suggest we just revore the commit in question.
best,
/p
signature.asc
Description: Digital signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Notmuch mail notifier applet for Gnome?
Hi all I've been using a simple notmuch count-based script together with xmobar in xmonad to notify me of new mail. However, I'm thinking of switching to a Gnome/xmonad-based combo, thus giving up xmobar. I've been looking for a replacement to my mail checker that would work with gnome-panel, but so far I haven't found any. I'm sure there has to be at least one Gnome user on the notmuch mailing list, so I thought I'd ask for help. How are you solving this? Basically anything that would display a notification icon if a given shell command returned true would suffice, though I'd prefer something that would also display a given text (read: a short fom of the specified query and the mail count for it, something like: »personal: 10« – possibly during mouse over). Thanks in advance, Albin ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Notmuch mail notifier applet for Gnome?
Hi Albin, This is not an answer to your inquiry, but because I looked into something similar recently for my setup I'd thought I'd share: I used xmonad before but switched to the awesome [0] tiling WM. I hacked my solution into their wiki, including a screensot [1]. best, /p [0] http://awesome.naquadah.org/ [1] https://awesome.naquadah.org/wiki/Notmuch_mail_integration On Wed, Jul 13, 2011 at 12:02:06AM +0200, Albin Stjerna wrote: Hi all I've been using a simple notmuch count-based script together with xmobar in xmonad to notify me of new mail. However, I'm thinking of switching to a Gnome/xmonad-based combo, thus giving up xmobar. I've been looking for a replacement to my mail checker that would work with gnome-panel, but so far I haven't found any. I'm sure there has to be at least one Gnome user on the notmuch mailing list, so I thought I'd ask for help. How are you solving this? Basically anything that would display a notification icon if a given shell command returned true would suffice, though I'd prefer something that would also display a given text (read: a short fom of the specified query and the mail count for it, something like: »personal: 10« – possibly during mouse over). Thanks in advance, Albin ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch signature.asc Description: Digital signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Slowness (search opens every email file?)
On Tue, Jul 12, 2011 at 4:24 PM, Jason Woofenden ja...@jasonwoof.com wrote: On 2011-07-11 06:13PM, Austin Clements wrote: On Mon, Jul 11, 2011 at 3:07 PM, Jason Woofenden ja...@jasonwoof.com wrote: notmuch search tag:foo is slow! (when my e-mail files are not already in the disk cache) I saw on my activity monitor applet that it was using mostly i/o, and started to wonder if it was opening every e-mail. I little work with strace and searching revealed that this command was opening many many e-mail files from my maildir(s). I spent a little while digging around in the notmuch source, and didn't see where it was opening the email files. It is opening every file to get a few headers to display in the search output. Istvan Marko sent an experimental patch to store these headers in the database a while ago, though as far as I know there hasn't been any progress cleaning it up for inclusion: id:m3sjsv2kw2@zsu.kismala.com . Cool. I suspected it was reading for header files. I googled the id and found this patchwork link: http://patchwork.notmuchmail.org/patch/947/ (I didn't see any way to ask mailman for a message id.) Here's the full thread from Nabble: http://notmuch.198994.n3.nabble.com/storing-From-and-Subject-in-xapian-td2901262.html Does someone want to work at this soon or should I try my hand at it? Istvan, did you make any progress on this patch since the last version? I seem to recall it just needed general cleanup (code style and such) and a better answer for backwards compatibility (the unfortunate thing). ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Slowness (search opens every email file?)
Austin Clements amdragon-3s7wtutd...@public.gmane.org writes:
Istvan, did you make any progress on this patch since the last
version? I seem to recall it just needed general cleanup (code style
and such) and a better answer for backwards compatibility (the
unfortunate thing).
I have been using the version that encodes empty headers to but
another way to handle this is to simply not set a VALUE for empty
headers and then fall back to the original parsing method for these
messages. Emails without from/subject/message-id headers are not very
common so perhaps this is a good compromise.
Below is the patch without the hack.
diff --git a/lib/database.cc b/lib/database.cc
index 9c2f4ec..63a15bb 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1654,7 +1654,7 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
goto DONE;
date = notmuch_message_file_get_header (message_file, date);
- _notmuch_message_set_date (message, date);
+ _notmuch_message_set_header_values (message, date, from, subject);
_notmuch_message_index_file (message, filename);
} else {
diff --git a/lib/message.cc b/lib/message.cc
index d993cde..48a31f5 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -410,6 +410,20 @@ _notmuch_message_ensure_message_file (notmuch_message_t *message)
const char *
notmuch_message_get_header (notmuch_message_t *message, const char *header)
{
+std::string value;
+
+// fetch header from the appropriate xapian value field if available
+if (strcmp(header,from) == 0)
+value=message-doc.get_value(NOTMUCH_VALUE_FROM);
+else if (strcmp(header,subject) == 0)
+value=message-doc.get_value (NOTMUCH_VALUE_SUBJECT);
+else if (strcmp(header,message-id) == 0)
+ value=message-doc.get_value (NOTMUCH_VALUE_MESSAGE_ID);
+
+if (!value.empty())
+ return talloc_strdup (message, value.c_str ());
+
+// otherwise fall back to parsing the file
_notmuch_message_ensure_message_file (message);
if (message-message_file == NULL)
return NULL;
@@ -785,8 +799,10 @@ notmuch_message_set_author (notmuch_message_t *message,
}
void
-_notmuch_message_set_date (notmuch_message_t *message,
- const char *date)
+_notmuch_message_set_header_values (notmuch_message_t *message,
+const char *date,
+const char *from,
+const char *subject)
{
time_t time_value;
@@ -799,6 +815,8 @@ _notmuch_message_set_date (notmuch_message_t *message,
message-doc.add_value (NOTMUCH_VALUE_TIMESTAMP,
Xapian::sortable_serialise (time_value));
+message-doc.add_value (NOTMUCH_VALUE_FROM, from);
+message-doc.add_value (NOTMUCH_VALUE_SUBJECT, subject);
}
/* Synchronize changes made to message-doc out into the database. */
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 02e24ee..2e91afd 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -111,7 +111,9 @@ _internal_error (const char *format, ...) PRINTF_ATTRIBUTE (1, 2);
typedef enum {
NOTMUCH_VALUE_TIMESTAMP = 0,
-NOTMUCH_VALUE_MESSAGE_ID
+NOTMUCH_VALUE_MESSAGE_ID,
+NOTMUCH_VALUE_FROM,
+NOTMUCH_VALUE_SUBJECT
} notmuch_value_t;
/* Xapian (with flint backend) complains if we provide a term longer
@@ -287,9 +289,10 @@ void
_notmuch_message_ensure_thread_id (notmuch_message_t *message);
void
-_notmuch_message_set_date (notmuch_message_t *message,
- const char *date);
-
+_notmuch_message_set_header_values (notmuch_message_t *message,
+const char *date,
+const char *from,
+const char *subject);
void
_notmuch_message_sync (notmuch_message_t *message);
--
Istvan
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: Slowness (search opens every email file?)
Quoth Istvan Marko on Jul 12 at 6:55 pm: Austin Clements amdragon-3s7wtutd...@public.gmane.org writes: Istvan, did you make any progress on this patch since the last version? I seem to recall it just needed general cleanup (code style and such) and a better answer for backwards compatibility (the unfortunate thing). I have been using the version that encodes empty headers to but another way to handle this is to simply not set a VALUE for empty headers and then fall back to the original parsing method for these messages. Emails without from/subject/message-id headers are not very common so perhaps this is a good compromise. Below is the patch without the hack. Ah, clever. I was going to suggest adding another value to indicate the presence or absence of these Xapian values, but I like your solution better. The only downside I can think of is that it might not extend to other headers if we store more header values in the database in the future. I'd say this patch looks good other than coding style - Tab indentation - /* */ comments, starting with a capital letter - Space between function name and open paren - Space after comma in argument lists - Spaces around assignment operator ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Slowness (search opens every email file?)
Austin Clements amdra...@mit.edu writes:
I'd say this patch looks good other than coding style
- Tab indentation
- /* */ comments, starting with a capital letter
- Space between function name and open paren
- Space after comma in argument lists
- Spaces around assignment operator
Thanks, fixed the ones I see:
diff --git a/lib/database.cc b/lib/database.cc
index 9c2f4ec..63a15bb 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1654,7 +1654,7 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
goto DONE;
date = notmuch_message_file_get_header (message_file, date);
- _notmuch_message_set_date (message, date);
+ _notmuch_message_set_header_values (message, date, from, subject);
_notmuch_message_index_file (message, filename);
} else {
diff --git a/lib/message.cc b/lib/message.cc
index d993cde..55070c6 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -410,6 +410,21 @@ _notmuch_message_ensure_message_file (notmuch_message_t *message)
const char *
notmuch_message_get_header (notmuch_message_t *message, const char *header)
{
+std::string value;
+
+/* Fetch header from the appropriate xapian value field if
+ * available */
+if (strcmp(header, from) == 0)
+ value = message-doc.get_value(NOTMUCH_VALUE_FROM);
+else if (strcmp(header, subject) == 0)
+ value = message-doc.get_value (NOTMUCH_VALUE_SUBJECT);
+else if (strcmp(header, message-id) == 0)
+ value = message-doc.get_value (NOTMUCH_VALUE_MESSAGE_ID);
+
+if (!value.empty())
+ return talloc_strdup (message, value.c_str ());
+
+/* Otherwise fall back to parsing the file */
_notmuch_message_ensure_message_file (message);
if (message-message_file == NULL)
return NULL;
@@ -785,8 +800,10 @@ notmuch_message_set_author (notmuch_message_t *message,
}
void
-_notmuch_message_set_date (notmuch_message_t *message,
- const char *date)
+_notmuch_message_set_header_values (notmuch_message_t *message,
+const char *date,
+const char *from,
+const char *subject)
{
time_t time_value;
@@ -799,6 +816,8 @@ _notmuch_message_set_date (notmuch_message_t *message,
message-doc.add_value (NOTMUCH_VALUE_TIMESTAMP,
Xapian::sortable_serialise (time_value));
+message-doc.add_value (NOTMUCH_VALUE_FROM, from);
+message-doc.add_value (NOTMUCH_VALUE_SUBJECT, subject);
}
/* Synchronize changes made to message-doc out into the database. */
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 02e24ee..2e91afd 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -111,7 +111,9 @@ _internal_error (const char *format, ...) PRINTF_ATTRIBUTE (1, 2);
typedef enum {
NOTMUCH_VALUE_TIMESTAMP = 0,
-NOTMUCH_VALUE_MESSAGE_ID
+NOTMUCH_VALUE_MESSAGE_ID,
+NOTMUCH_VALUE_FROM,
+NOTMUCH_VALUE_SUBJECT
} notmuch_value_t;
/* Xapian (with flint backend) complains if we provide a term longer
@@ -287,9 +289,10 @@ void
_notmuch_message_ensure_thread_id (notmuch_message_t *message);
void
-_notmuch_message_set_date (notmuch_message_t *message,
- const char *date);
-
+_notmuch_message_set_header_values (notmuch_message_t *message,
+const char *date,
+const char *from,
+const char *subject);
void
_notmuch_message_sync (notmuch_message_t *message);
--
Istvan
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: Slowness (search opens every email file?)
Quoth Istvan Marko on Jul 12 at 8:07 pm: Austin Clements amdra...@mit.edu writes: I'd say this patch looks good other than coding style - Tab indentation - /* */ comments, starting with a capital letter - Space between function name and open paren - Space after comma in argument lists - Spaces around assignment operator Thanks, fixed the ones I see: +/* Fetch header from the appropriate xapian value field if + * available */ +if (strcmp(header, from) == 0) + value = message-doc.get_value(NOTMUCH_VALUE_FROM); +else if (strcmp(header, subject) == 0) + value = message-doc.get_value (NOTMUCH_VALUE_SUBJECT); +else if (strcmp(header, message-id) == 0) + value = message-doc.get_value (NOTMUCH_VALUE_MESSAGE_ID); The strcmp's should have a space before the paren, as should the first get_value. (Yeah, it's weird. Blame glib.) Also, it occurred to me that these should be strcasecmp's, since headers are case-insensitive. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
