notmuch-hello buffer slow doe to slow query

2014-04-12 Thread David Bremner
Nils Dagsson Moskopp  writes:

>
> mjw1009 can reproduce if the from-query contains an "@" and thinks the
> problem may be "something deeper down in notmuch (actually probably in
> xapian)".

My own experiments contradict that (see attached patch for a test to
run). The exact address is about twice as fast to count on both of my
machines, which matches the amount of I/O. FWIW, my SSD based laptop is
about 10x faster (compared to a similarly specced desktop with HD) for
both of them, which kindof reinforces how I/O bound this is.  

Replacing 'count' with 'search' gives pretty similar relative behaviour,
although much slower since it seems to do about 10x I/O.  

Finally after an initial "notmuch tag +fromlinus
from:torvalds at linux-foundation.org", "notmuch count tag:fromlinus" seems
to be about 5-10x faster than "notmuch count
from:torvalds at linux-foundation.org".  So that might be a workaround for
some people.

To sum up, I think the underlying question is whether the amount of I/O
that count does can be reduced when using non-boolean query terms.

-- next part --
A non-text attachment was scrubbed...
Name: 0001-WIP-new-performance-test-for-notmuch-count.patch
Type: text/x-diff
Size: 1119 bytes
Desc: not available
URL: 



Re: notmuch-hello buffer slow doe to slow query

2014-04-12 Thread David Bremner
Nils Dagsson Moskopp n...@dieweltistgarnichtso.net writes:


 mjw1009 can reproduce if the from-query contains an @ and thinks the
 problem may be something deeper down in notmuch (actually probably in
 xapian).

My own experiments contradict that (see attached patch for a test to
run). The exact address is about twice as fast to count on both of my
machines, which matches the amount of I/O. FWIW, my SSD based laptop is
about 10x faster (compared to a similarly specced desktop with HD) for
both of them, which kindof reinforces how I/O bound this is.  

Replacing 'count' with 'search' gives pretty similar relative behaviour,
although much slower since it seems to do about 10x I/O.  

Finally after an initial notmuch tag +fromlinus
from:torva...@linux-foundation.org, notmuch count tag:fromlinus seems
to be about 5-10x faster than notmuch count
from:torva...@linux-foundation.org.  So that might be a workaround for
some people.

To sum up, I think the underlying question is whether the amount of I/O
that count does can be reduced when using non-boolean query terms.

From d3fbbe7ac0a1fb5b5691674226a3a3d000e15a1c Mon Sep 17 00:00:00 2001
From: David Bremner da...@tethera.net
Date: Sat, 12 Apr 2014 11:50:22 -0300
Subject: [PATCH] WIP: new performance test for notmuch count

Probably not suitable for production because of sudo
---
 performance-test/T03-count.sh | 23 +++
 1 file changed, 23 insertions(+)
 create mode 100755 performance-test/T03-count.sh

diff --git a/performance-test/T03-count.sh b/performance-test/T03-count.sh
new file mode 100755
index 000..e74437e
--- /dev/null
+++ b/performance-test/T03-count.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+test_description='counting'
+
+. ./perf-test-lib.sh
+
+time_start
+
+for i in $(seq 1 10); do
+echo 3 | sudo tee /proc/sys/vm/drop_caches
+notmuch --version  /dev/null
+time_run from:torva...@linux-foundation.org   \
+	notmuch count from:torva...@linux-foundation.org
+
+echo 3 | sudo tee /proc/sys/vm/drop_caches
+notmuch --version  /dev/null
+
+time_run from:linux-foundation.org\
+	notmuch count from:linux-foundation.org
+done
+
+
+time_done
-- 
1.9.1

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


notmuch-hello buffer slow doe to slow query

2014-04-11 Thread Nils Dagsson Moskopp
Hello,

If notmuch-hello includes a saved search with a slow query, switching to
a notmuch-hello buffer is very slow due to notmuch-mode updating counts
for search results. mjw1009 suggested "(setq notmuch-hello-auto-refresh
nil)", which stops the counting and works around the problem.

Fundamentally, the problem is a slow query. On my laptop (Thinkpad T60),
many things are pretty much instant, even though I have a HDD, no SSD:

> ; time notmuch count 'tag:inbox and tag:list'
> 25452
> 0.02user 0.00system 0:00.03elapsed 72%CPU (0avgtext+0avgdata 3852maxresident)k
> 0inputs+0outputs (0major+1135minor)pagefaults 0swaps

However, from-queries take their time:

> ; time notmuch count 'not tag:replied and to:nils at dieweltistgarnichtso.net'
> 5328
> 0.10user 0.15system 0:14.14elapsed 1%CPU (0avgtext+0avgdata 3472maxresident)k
> 157544inputs+0outputs (0major+1039minor)pagefaults 0swaps

mjw1009 can reproduce if the from-query contains an "@" and thinks the
problem may be "something deeper down in notmuch (actually probably in
xapian)".


Greetings,
-- 
Nils Dagsson Moskopp // erlehmann

-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 212 bytes
Desc: not available
URL: 



notmuch-hello buffer slow doe to slow query

2014-04-11 Thread Nils Dagsson Moskopp
Hello,

If notmuch-hello includes a saved search with a slow query, switching to
a notmuch-hello buffer is very slow due to notmuch-mode updating counts
for search results. mjw1009 suggested (setq notmuch-hello-auto-refresh
nil), which stops the counting and works around the problem.

Fundamentally, the problem is a slow query. On my laptop (Thinkpad T60),
many things are pretty much instant, even though I have a HDD, no SSD:

 ; time notmuch count 'tag:inbox and tag:list'
 25452
 0.02user 0.00system 0:00.03elapsed 72%CPU (0avgtext+0avgdata 3852maxresident)k
 0inputs+0outputs (0major+1135minor)pagefaults 0swaps

However, from-queries take their time:

 ; time notmuch count 'not tag:replied and to:n...@dieweltistgarnichtso.net'
 5328
 0.10user 0.15system 0:14.14elapsed 1%CPU (0avgtext+0avgdata 3472maxresident)k
 157544inputs+0outputs (0major+1039minor)pagefaults 0swaps

mjw1009 can reproduce if the from-query contains an @ and thinks the
problem may be something deeper down in notmuch (actually probably in
xapian).


Greetings,
-- 
Nils Dagsson Moskopp // erlehmann
http://dieweltistgarnichtso.net


pgpxCpPcJjxH7.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch