Re: performance problems with notmuch new
Franz Fellner writes: > > On Thu Apr 23 00:21:30 2020, Olly Betts wrote: >> Then I'd try compacting the database (I think there's a "notmuch >> compact" subcommand to do this). > And there we go. Cured the issues. Dropped the very first indexing > from several minutes to 1.5 seconds on the desktop. ?!?! This is a > really new setup and I suffered from bad performance from the very > first notmuch new after the initial indexing. Is it really needed to > run notmch compact directly after the initial notmuch new? Desktop > currently has 38502 messages indexed, in case that matters. That's a fairly small number of messages by the usual notmuch standards. It's possible there is something pathological about the interaction of notmuch and your particular set of mail. Running the performance test suite on your machine would be one way of telling. Also, the size of your notmuch database on disk; really huge databases are a sign that something is going wrong. I think most people find the database is 1/3 to 1/2 the size of the mail it is indexing, but we have had some odd cases where the database is 2 or 3 times the mail, and that was an indexing bug. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: performance problems with notmuch new
Don Zickus writes: > > Of course I am using this in a strange context. I am deleting emails > locally and then running 'notmuch new' to clean up the database. > [snip] > So maybe my use case is unusual and the slowness is expected. I'd say it's a known performance bug in notmuch that it doesn't deal well with large numbers of deletions. At some point it is faster to reindex the whole database from scratch. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: performance problems with notmuch new
On Fri, Apr 24, 2020 at 07:36:23AM -0300, David Bremner wrote: > Don Zickus writes: > > > > > The only thing I can think of is my fstrim service runs 1x / week on Monday > > at midnight and maybe that helped clean things up?? Perhaps I should > > increase that frequency or run it manually when things go bad. > > > > Is notmuch new still slow on your mail store, or did that mysteriously > improve as well? It improved slightly. Perhaps my expectations are too high. After about a 1000, the speed slows down significantly. Outside of the performance testing, after about 1000 emails, the processing slows down to about 10/second. Of course I am using this in a strange context. I am deleting emails locally and then running 'notmuch new' to clean up the database. notmuch search --output=files |xargs -l rm notmuch new and sometimes I delete a couple thousand emails from my mailing list folder in neomutt and while exiting the folder it takes a long while to sync up with notmuch. So maybe my use case is unusual and the slowness is expected. Cheers, Don ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: performance problems with notmuch new
On Thu Apr 23 00:21:30 2020, Olly Betts wrote: > First question: what version of Xapian are you using? On my laptop it's 1.4.15 (arch linux) and the desktop runs 1.4.14 (Gentoo linux) > And second thing to check, are you committing each message separately? No, I sync with mbsync which dosnloads a bunch of mails, then I run notmuch new which indexes all in one go. > After reboot the disk cache won't have any of the database in, so the > first operation will typically be slower, especially with a spinning > drive where seeks are relatively slow. Yes, I know that, I just wanted to mention the number, which IMO is insane. I want to setup notmuch for my dad on the desktop PC. 5 minutes to wait for his mail in the morning would have made notmuch a no-go. > It sounds like you're seek-limited in this "cold cache" phase. That is > not necessarily related to the slow indexing, but it could be. > > I'd check the SMART diagnostics for the drive first (e.g. with > smartctl). It's not the most likely cause, but it's quick to check and > if the drive is starting to fail it's better to find out sooner rather > than later. HDDs are healthy. I actually checked quite recently when converting the laptop from Gentoo to arch. > > Then I'd try compacting the database (I think there's a "notmuch > compact" subcommand to do this). And there we go. Cured the issues. Dropped the very first indexing from several minutes to 1.5 seconds on the desktop. ?!?! This is a really new setup and I suffered from bad performance from the very first notmuch new after the initial indexing. Is it really needed to run notmch compact directly after the initial notmuch new? Desktop currently has 38502 messages indexed, in case that matters. Regards Franz ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Inconsistencies in handling command flags: `--flag=value` different than `--flag value`
[Again sorry for double reporting. BTW, where should I search for previous bugs? I've currently tried the mailing list archive.] Trying to play with `notmuch` from a wrapper, I've stumbled upon the following command line flags handling bug: notmuch show --format json --entire-thread true --body false -- 'cipr...@volution.ro' notmuch show --format json --entire-thread true --body=false -- 'cipr...@volution.ro' #=> yields nothing notmuch show --format json --entire-thread=true --body false -- 'cipr...@volution.ro' #=> yields some emails notmuch show --format json --entire-thread=true --body=false -- 'cipr...@volution.ro' #=> yields lots of emails I would expect that `--flag value` and `--flag=value` are equivalent, at least for the options that the manual states `--flag=(true|false)`. However based on the previous experiments it seems that using anything except `--flag=value` yields inconsistent results. Hope it helps, Ciprian. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Strip spaces in `tags` in `~/.notmuch-config` (and other fields)
Ciprian Dorin Craciun writes: > > I've tried to manually edit my `~/.notmuch-config`, and I've seen that > the field `tags` was written as `tags=unread;inbox;`. In order to > increase readability I've decided to update my configuration file by > adding spaces around `=` and `;` as in `tags = unread ; inbox ;`. > Everything worked without a warning, until it didn't... :) > > What happened: all my emails are now tagged with `unread ` and ` > inbox `; (i.e. whitespace in tags). > Digging into the code a bit, the culprit seems to be _config_get_list. It is a bit surprising that the glib function g_key_file_get_string_list isn't a bit friendlier here (or at least better documented), but I guess the output of it needs to be postprocessed. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Strip spaces in `tags` in `~/.notmuch-config` (and other fields)
[Sorry if I'm double reporting this. I've tried my best to search for previous discussions.] I've tried to manually edit my `~/.notmuch-config`, and I've seen that the field `tags` was written as `tags=unread;inbox;`. In order to increase readability I've decided to update my configuration file by adding spaces around `=` and `;` as in `tags = unread ; inbox ;`. Everything worked without a warning, until it didn't... :) What happened: all my emails are now tagged with `unread ` and ` inbox `; (i.e. whitespace in tags). Given that the `~/.notmuch-config` resembles an INI file, and given how lax the actual syntax is in general, I would suggest the following: * allow white-spaces around `[ section ]`, and `field = value`; * strip white-spaces (left and right) from values like `tags = unread ; inbox ;`; (but not infix like `tag = some tag ; some other tag;`;) * allow skipping the last `;` separator from `tags` and similar; Failing that, perhaps add a warning when parsing the configuration file. Hope it helps, Ciprian. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: performance problems with notmuch new
Don Zickus writes: > > The only thing I can think of is my fstrim service runs 1x / week on Monday > at midnight and maybe that helped clean things up?? Perhaps I should > increase that frequency or run it manually when things go bad. > Is notmuch new still slow on your mail store, or did that mysteriously improve as well? d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: test_emacs_expect_t does ignore Emacs as prerequisite
Milton Vandersloot writes:
>
> [PATCH] Let test_emacs_expect_t respect missing external prerequisites
>
> test_emacs_expect_t did not test for missing prerequisites (even though
> it called test_emacs which does it). Fix that by testing for missing
> prerequisites.
>
I agree there's a bug here, but I'm not sure this is the best/cleanest
fix. Maybe Tomi (in Cc) can comment. The logic for prerequisite checking
is already opaque. For example test_skip is already calling
test_check_missing_external_prereqs_ as a side effect. For starters I
wonder if test_emacs should use a return value to indicate failure,
along the lines of the patch at the end of the message.
BTW, it will make our life easier if you follow
https://notmuchmail.org/contributing/#index5h2; in particular using
git-send-email and keeping the discussion/notes after ---.
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 7f8a3a4d..eecc52f7 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -543,9 +543,8 @@ test_emacs_expect_t () {
# Run the test.
if ! test_skip "$test_subtest_name"
then
- test_emacs "(notmuch-test-run $1)" >/dev/null
-
- # Restore state after the test.
+ test_emacs "(notmuch-test-run $1)" || return
+# Restore state after the test.
exec 1>&6 2>&7 # Restore stdout and stderr
inside_subtest=
@@ -997,7 +996,7 @@ test_emacs () {
test_require_external_prereq dtach || missing_dependencies=1
test_require_external_prereq emacs || missing_dependencies=1
test_require_external_prereq ${TEST_EMACSCLIENT} ||
missing_dependencies=1
- test -z "$missing_dependencies" || return
+ test -z "$missing_dependencies" || return 1
if [ -z "$EMACS_SERVER" ]; then
emacs_tests="$NOTMUCH_SRCDIR/test/${this_test_bare}.el"
@@ -1034,6 +1033,7 @@ test_emacs () {
touch OUTPUT
${TEST_EMACSCLIENT} --socket-name="$EMACS_SERVER" --eval
"(notmuch-test-progn $*)"
+return 0
}
test_python() {
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
