[PATCH RESEND 0/2] emacs: sort saved searches

2011-11-22 Thread David Bremner 
On Fri, 23 Sep 2011 21:57:36 +0300, Jani Nikula  wrote:
> Hi, resending as this seems forgotten. Last time [1] there was no objections 
> to
> this improved approach.
> 
> BR,
> Jani.

Pushed to master.

d

[PATCH] Output unmodified Content-Type header value for JSON format.

2011-11-22 Thread Austin Clements 
Quoth Jameson Graef Rollins on Nov 20 at 12:10 pm:
> The open question seems to be how we handle the content encoding
> parameters.  My argument is that those should either be used by notmuch
> to properly encode the content for the consumer.  If that's not
> possible, then just those parameters needed by the consumer to decode
> the content should be output.

If notmuch is going to include part content in the JSON output (which
perhaps it shouldn't, as per recent IRC discussions), then it must
handle content encodings because JSON must be Unicode and therefore
the content strings in the JSON must be Unicode.

Notmuch indexing 21 million emails

2011-11-22 Thread Austin Clements 
Quoth Tom Bulli on Nov 21 at  7:02 pm:
> I have a project where I need to search about 21 emails - and
> decided to use "notmuch" for it.? The system is a Debian Squeeze,
> the notmuch version is "0.8-1~bpo60+1" from "kyria's" private
> repository.
> 
> I am running the "notmuch new" for approx. 4 days now - and
> according to "not,uch count" it has indexed about 4.5 million
> emails.
> 
> Is this expected performance?? Is there any way to speed that up?

Currently, notmuch is much more optimized for search than it is for
indexing.  This is unfortunate for the initial indexing process and
seems to be becoming increasingly unfortunate.

There are some things you can try.  One is to use an SSD if you aren't
already, since constructing the index requires a lot of random IO.
You can also try libeatmydata to disable fsync's, which may improve
your IO performance, with the obvious crash-safety caveats.  However,
unless you have a lot of RAM, I suspect your index has long outgrown
your buffer cache, so this may have limited impact.

Since you're going to the trouble of indexing 21 million emails, you
might want to try 0.10 (under freeze right now, to be released very,
very soon).  It won't improve your indexing time, but if you're doing
searches with non-trivial numbers of results, emails indexed with 0.10
will search much faster.

Sorry I don't have better news, but I hope this helps.

Incorrect parsing of email addresses (MIME with quotes)

2011-11-22 Thread Austin Clements 
Quoth David Bremner on Nov 22 at  7:30 pm:
> On Mon, 21 Nov 2011 13:38:57 +0100, Petter Reinholdtsen  
> wrote:
> > I just hope the notmuch developers can choose to follow the robustness
> > principle[1] in this case, and be liberal in what notmuch accept, and
> > conservative in what notmuch send.  After all, gnus, mutt and pine
> > handle these from fields.
> 
> It's a nice stick to beat people with, but I'm not sure it's directly
> applicable here. Notmuch is accepting the input, it just isn't
> displaying it the way you want. I guess if somebody has some patches for
> "sloppy header parsing", we can weigh the pros and cons then.
> 
> If someone wants to follow up on this, we are currently using
> 
> http://spruce.sourceforge.net/gmime/doc/gmime-gmime-utils.html#g-mime-utils-header-decode-text
> 
> There is some discussion there about dealing with bad inputs; I don't
> know if it is directly relevant.

Based on
  http://comments.gmane.org/gmane.mail.mime.gmime.devel/65
I thought gmime was supposed to be dealing with broken RFC 2047
encoding now.  Am I misinterpreting it?  Maybe a bug needs to be filed
against gmime?

'notmuch new' leaking memory and getting slower over time?

2011-11-22 Thread Austin Clements 
Quoth Petter Reinholdtsen on Nov 21 at 11:35 pm:
> The indexing took 36 hours.  At the start it claimed it would take 10
> hours, and it continued to underestimate the amount of time left until
> the very end.  It claimed to have 1 hour left when I checked before I
> went to bed, and claimed to have 15 minutes left when I woke up 6-7
> hours later.

notmuch new does a simple linear extrapolation based on how many files
it's examined and how many there are total.  This is doomed to
undershoot at least because indexing becomes slower as the database
grows (B-tree insertion is O(log N), fragmentation will increase over
time, posting lists will get longer...).

I'm not sure much can be done about the estimate at the beginning,
short of throwing in some fudge factor, but the estimates later in the
process would be much more accurate if it used a sliding window,
rather than measuring from the beginning.

> Shortly before the indexing finished, the notmuch process was using 1.2
> GiB of resident memory according to top.  Is the process leaking memory?

It's possible this is just memory fragmentation, but it definitely
sounds like a leak.  talloc has some tools for tracking down leaks and
it would be good to heap profile notmuch new, but to my knowledge
nobody's applied these tools.

Incorrect parsing of email addresses (MIME with quotes)

2011-11-22 Thread David Bremner 
On Mon, 21 Nov 2011 13:38:57 +0100, Petter Reinholdtsen  
wrote:

> I just hope the notmuch developers can choose to follow the robustness
> principle[1] in this case, and be liberal in what notmuch accept, and
> conservative in what notmuch send.  After all, gnus, mutt and pine
> handle these from fields.

It's a nice stick to beat people with, but I'm not sure it's directly
applicable here. Notmuch is accepting the input, it just isn't
displaying it the way you want. I guess if somebody has some patches for
"sloppy header parsing", we can weigh the pros and cons then.

If someone wants to follow up on this, we are currently using

http://spruce.sourceforge.net/gmime/doc/gmime-gmime-utils.html#g-mime-utils-header-decode-text

There is some discussion there about dealing with bad inputs; I don't
know if it is directly relevant.

d

'notmuch new' leaking memory and getting slower over time?

2011-11-22 Thread David Bremner 
On Mon, 21 Nov 2011 23:35:34 +0100, Petter Reinholdtsen  
wrote:
> 
>   NO*TELEMAX**NORWAYII  M0018001012307699038 (unread usit year-2002)
> 
> The message in question is a bounce from some X400 mail system, and its
> message id look like this:
> 
>   Message-Id: 

Maybe try the patch 

  id:"1319884807-7206-1-git-send-email-thomas at schwinge.name"

d

[PATCH] debian/control: require dtach version = 0.8 in Build-Depends

2011-11-22 Thread Tomi Ollila 
Since version 0.8 of dtach -n does no longer require controlling
tty to be present when executed. Currently controlling tty is not
always (if ever) present when tests are executed.
---
 debian/control |2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/debian/control b/debian/control
index 3252f32..f6f415e 100644
--- a/debian/control
+++ b/debian/control
@@ -16,7 +16,7 @@ Build-Depends:
  python-all (= 2.6.6-3~),
  emacs23-nox | emacs23 (=23~) | emacs23-lucid (=23~),
  gdb,
- dtach
+ dtach (= 0.8)
 Standards-Version: 3.9.2
 Homepage: http://notmuchmail.org/
 Vcs-Git: git://notmuchmail.org/git/notmuch
-- 
1.7.7.3
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] make release: verify-version-*: change comparison logic

2011-11-22 Thread Tomi Ollila 
verfy-version-debian, verify-version-python and verify-version-components
checked noneqality of the comparison strings and if got positive
answer then made that goal fail. But in case of the test ([ ])
execution failed it never got to the 'then' part of the line (and
the 'if [ ... ] then ... fi ' construct doesn't make the script line
fail in case of problems inside [ ].
This commit inverses the logic, so that only if the comparison for
equality succeeds the script line will exit with 0 and execution
can continue past the failure case to the next line (executed by another
shell) with '@echo done'
---
 Makefile.local |   15 ++-
 1 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/Makefile.local b/Makefile.local
index 775f393..b6e216a 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -207,28 +207,25 @@ endif
 .PHONY: verify-version-debian
 verify-version-debian: verify-version-components
@echo -n Checking that Debian package version is $(VERSION)-1...
-   @if [ $(VERSION)-1 != $$(dpkg-parsechangelog | grep ^Version | awk 
'{print $$2}') ] ; then \
+   @[ $(VERSION)-1 = $$(dpkg-parsechangelog | grep ^Version | awk 
'{print $$2}') ] || \
(echo No.  \
-echo Please edit version and debian/changelog to have 
consistent versions.  false) \
-fi
+echo Please edit version and debian/changelog to have 
consistent versions.  false)
@echo Good.
 
 .PHONY: verify-version-python
 verify-version-python: verify-version-components
@echo -n Checking that python bindings version is $(VERSION)...
-   @if [ $(VERSION) != $$(python -c execfile('$(PV_FILE)'); print 
__VERSION__) ] ; then \
+   @[ $(VERSION) = $$(python -c execfile('$(PV_FILE)'); print 
__VERSION__) ] || \
(echo No.  \
-echo Please edit version and $(PV_FILE) to have consistent 
versions.  false) \
-fi
+echo Please edit version and $(PV_FILE) to have consistent 
versions.  false)
@echo Good.
 
 .PHONY: verify-version-components
 verify-version-components:
@echo -n Checking that $(VERSION) consists only of digits and 
periods...
-   @if echo $(VERSION) | grep -q -v -x '[0-9.]*'; then \
+   @echo $(VERSION) | grep -q -x '^[0-9.]*$$' || \
(echo No.  \
-echo Please follow the instructions in RELEASING to choose a 
version  false) \
-else :; fi
+echo Please follow the instructions in RELEASING to choose a 
version  false)
@echo Good.
 
 .PHONY: verify-newer
-- 
1.7.7.3

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] make release: added goal verify-version-manpage

2011-11-22 Thread Tomi Ollila 
Check that the version mentioned in notmuch manual page
is consistent with the version file.
---
 Makefile.local |   10 +-
 1 files changed, 9 insertions(+), 1 deletions(-)

diff --git a/Makefile.local b/Makefile.local
index b6e216a..02afdd0 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -187,7 +187,7 @@ release-message:
 verify-source-tree-and-version: verify-no-dirty-code
 
 .PHONY: verify-no-dirty-code
-verify-no-dirty-code: verify-version-debian verify-version-python
+verify-no-dirty-code: verify-version-debian verify-version-python 
verify-version-manpage
 ifeq ($(IS_GIT),yes)
@printf Checking that source tree is clean...
 ifneq ($(shell git ls-files -m),)
@@ -220,6 +220,14 @@ verify-version-python: verify-version-components
 echo Please edit version and $(PV_FILE) to have consistent 
versions.  false)
@echo Good.
 
+.PHONY: verify-version-manpage
+verify-version-manpage: verify-version-components
+   @echo -n Checking that manual page version is $(VERSION)...
+   @[ $(VERSION) = $$(sed -n '/^[.]TH NOTMUCH 1/{s/.*Notmuch 
//;s/.*//p;}' notmuch.1) ] || \
+   (echo No.  \
+echo Please edit version and notmuch.1 to have consistent 
versions.  false)
+   @echo Good.
+
 .PHONY: verify-version-components
 verify-version-components:
@echo -n Checking that $(VERSION) consists only of digits and 
periods...
-- 
1.7.7.3


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 2/4] tags_to_maildir_flags: Cleanup double assignement

2011-11-22 Thread David Bremner 
On Thu, 15 Sep 2011 00:23:19 +0200, Louis Rilling l.rill...@av7.net wrote:
 The for loop right after already does the job.
 

I pushed the first two patches in this series; the second two need to be
updated for the new broken test framework, and reviewed.

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Installation problem on OS X

2011-11-22 Thread Michael Williams 
Hi,

there seems to be a problem with notmuch on OS X when installed to a 
non-standard location: it's install name is libnotmuch.*.dylib rather than 
something like $(LIBDIR)/libnotmuch.*.dylib, which prevents the binary from 
locating the library when things are installed in a nonstandard location. make 
works, but attempts to run it result in

dyld: Library not loaded: libnotmuch.2.dylib
  Referenced from: /Users/mike/.homebrew/bin/notmuch
  Reason: image not found
Trace/BPT trap

I ran into this when installing it via the homebrew packaing system into 
somewhere other than /usr/local. I've fixed it for the time being by doing 

  install_name_tool -change libnotmuch.2.dylib 
/Users/mike/.homebrew/lib/libnotmuch.2.dylib /Users/mike/.homebrew/bin/notmuch

and in principle I could add a post-install hook to the homebrew formula to do 
this for other users of homebrew/OS X, but it seems like this is something that 
would be better fixed upstream. Any thoughts?

A related issue has come up before: 
http://notmuchmail.org/pipermail/notmuch/2011/004094.html. Here's the 
discussion of this issue at homebrew: 
https://github.com/mxcl/homebrew/issues/8284. I am not on the mailing list so 
please CC me on replies.

Mike
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Notmuch indexing 21 million emails

2011-11-22 Thread Austin Clements 
Quoth Tom Bulli on Nov 21 at  7:02 pm:
 I have a project where I need to search about 21 emails - and
 decided to use notmuch for it.  The system is a Debian Squeeze,
 the notmuch version is 0.8-1~bpo60+1 from kyria's private
 repository.
 
 I am running the notmuch new for approx. 4 days now - and
 according to not,uch count it has indexed about 4.5 million
 emails.
 
 Is this expected performance?  Is there any way to speed that up?

Currently, notmuch is much more optimized for search than it is for
indexing.  This is unfortunate for the initial indexing process and
seems to be becoming increasingly unfortunate.

There are some things you can try.  One is to use an SSD if you aren't
already, since constructing the index requires a lot of random IO.
You can also try libeatmydata to disable fsync's, which may improve
your IO performance, with the obvious crash-safety caveats.  However,
unless you have a lot of RAM, I suspect your index has long outgrown
your buffer cache, so this may have limited impact.

Since you're going to the trouble of indexing 21 million emails, you
might want to try 0.10 (under freeze right now, to be released very,
very soon).  It won't improve your indexing time, but if you're doing
searches with non-trivial numbers of results, emails indexed with 0.10
will search much faster.

Sorry I don't have better news, but I hope this helps.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Output unmodified Content-Type header value for JSON format.

2011-11-22 Thread Austin Clements 
Quoth Jameson Graef Rollins on Nov 20 at 12:10 pm:
 The open question seems to be how we handle the content encoding
 parameters.  My argument is that those should either be used by notmuch
 to properly encode the content for the consumer.  If that's not
 possible, then just those parameters needed by the consumer to decode
 the content should be output.

If notmuch is going to include part content in the JSON output (which
perhaps it shouldn't, as per recent IRC discussions), then it must
handle content encodings because JSON must be Unicode and therefore
the content strings in the JSON must be Unicode.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch