Re: out of memory on idle machine

2021-02-11 Thread Olly Betts
On Thu, Feb 11, 2021 at 06:53:27AM -0400, David Bremner wrote: > At this point I don't really have any good ideas, so I'm waiting for > results from the 1.4.18 trial. I've uploaded a backport, but it's the first backport of xapian-core to buster so it'll need manual approval. Hopefully that'll

Re: out of memory on idle machine

2021-02-08 Thread Olly Betts
the file that means the lock is held. It's not at all frequent, but perhaps we should adjust this message to better reflect that.) Have you tried xapian-check on this database? > Olly Betts mentioned in a different thread that he will build a version > of xapian 1.4.18 for buster backport

Re: performance problems with notmuch new

2020-04-22 Thread Olly Betts
On Mon, Apr 20, 2020 at 11:36:36AM -0300, David Bremner wrote: > Franz Fellner writes: > > > I also suffer from bad performance of notmuch new. I used notmuch > > some years ago and notmuch new always felt instantanious. Had to stop > > using it because internet was too slow to sync my mails

Re: crash after running notmuch new

2020-04-07 Thread Olly Betts
On Tue, Apr 07, 2020 at 05:21:47PM -0300, David Bremner wrote: > Matt writes: [...] > > termlist: > > blocksize=8K items=186136 firstunused=62058 revision=421 levels=2 root=12260 > > B-tree checked okay > > termlist table structure checked OK > > > > postlist: > > blocksize=8K items=2598971

Re: Transitioning notmuch/Xapian from 32-bit to 64-bit system

2019-07-09 Thread Olly Betts
On Tue, Jul 09, 2019 at 01:28:51PM -0300, David Bremner wrote: > Thomas Schwinge writes: > > > sizes?). Doing some light (read-only!) testing, it seems to work fine to > > access the old 32-bit built Xapian "chert" database with 64-bit > > notmuch/Xapian. Is that (a) generally safe and

Re: Feature request: Limit output of notmuch show

2019-03-25 Thread Olly Betts
On Fri, Mar 22, 2019 at 12:41:12PM -0300, David Bremner wrote: > jo...@joergvolbers.de (Jörg Volbers) writes: > > Is it a difficulty to implement that? Would anyone do that? I am > > not able to write anything in C, so I'm out of it. > > I don't have any good ideas how to impliment that in a

Re: xapian parser bug?

2018-09-30 Thread Olly Betts
On Sun, Sep 30, 2018 at 09:05:25AM -0300, David Bremner wrote: > if (str.find (' ') != std::string::npos) > query_str = '"' + str + '"'; > else > query_str = str; > > return parser.parse_query (query_str, NOTMUCH_QUERY_PARSER_FLAGS, >

Re: xapian parser bug?

2018-09-30 Thread Olly Betts
On Sun, Sep 30, 2018 at 09:50:30AM +0100, James Aylett wrote: > Note that I'm using 1.4.7, and from your output I believe you're not > (the * in the query description I believe doesn't happen in those > situations any more). 1.4.4 and later eliminate redundant 0 scaling factors, but this one

Re: Notmuch DB Problems

2018-09-10 Thread Olly Betts
On Mon, Sep 10, 2018 at 08:01:06AM -0300, David Bremner wrote: > Mueen Nawaz writes: > > Now killing all those jobs did not fix the database. It was still > > broken. And as we saw the second time round, it was /really/ broken - it > > would not even open in read-only mode. > > That seems like

Re: Database corruption after clean rebuild

2018-04-07 Thread Olly Betts
On Sat, Apr 07, 2018 at 12:17:39PM -0300, David Bremner wrote: > Javier Garcia writes: > > > The following is a solid workaround I've stumbled upon. Afew no longer > > complains and database corruption is gone. > > > > $ notmuch compact > > $ xapian-check

Re: bug: "no top level messages" crash on Zen email loops

2018-03-29 Thread Olly Betts
On Thu, Mar 29, 2018 at 08:50:22AM -0400, Antoine Beaupré wrote: > On 2018-03-29 04:17:21, Olly Betts wrote: > > If changes to a new database which didn't modify the termlist table were > > committed, then a disk block which had been allocated to be the root > > block

Re: bug: "no top level messages" crash on Zen email loops

2018-03-28 Thread Olly Betts
On Mon, Mar 19, 2018 at 05:03:21PM -0300, David Bremner wrote: > I can confirm this reproduces both the xapian-check and the notmuch-show > error. Olly agrees that whatever notmuch is doing wrong, it shouldn't > lead to a corrupted database There was a Xapian bug here, which I fixed on master

Re: emacs-notmuch: A Xapian exception occurred parsing query

2018-02-15 Thread Olly Betts
On 2018-02-07, David Bremner wrote: > The underlying issue is that * is parsed (simplistically) by notmuch > before passing to Xapian, so only works if it is the entire query. > > For cases like you report, where the user has not entered '*', but > rather it is contained in some generated query

Re: Inconsistent query results

2017-03-09 Thread Olly Betts
On Wed, Mar 08, 2017 at 10:32:56PM -0400, David Bremner wrote: > "Kirill A. Shutemov" writes: > > I found that on particular queries notmuch return different results if run > > the query few times. Re-initialing the query or db doesn't help. > > Thanks for the report. I

Re: [PATCH v4 16/16] add "notmuch reindex" subcommand

2016-08-14 Thread Olly Betts
On Mon, Aug 15, 2016 at 07:42:39AM +0900, David Bremner wrote: > Daniel Kahn Gillmor writes: > > +Supported options for **reindex** include > > + > > +``--try-decrypt`` > > + > > +For each message, if it is encrypted, try to decrypt it while > > +

Re: slowdown in notmuch perf suite with xapian 1.3.5

2016-04-07 Thread Olly Betts
On Thu, Apr 07, 2016 at 09:40:59PM -0300, David Bremner wrote: > Olly Betts <o...@survex.com> writes: > > > > > So the T00-new.sh numbers make sense - there's more work to do, and > > we need to read existing positional data more to insert the new stuff, > > so

Re: slowdown in notmuch perf suite with xapian 1.3.5

2016-04-07 Thread Olly Betts
On Thu, Apr 07, 2016 at 08:56:46AM -0300, David Bremner wrote: > I hadn't noticed any interactive slowdown, but when I got around to > running the notmuch performance suite, there seems to be some noticable > slowdown with the glass backend (default in Xapian 1.3.5) compared to > chert (using

segfault with xapian 1.3.1

2013-09-06 Thread Olly Betts
Olly wrote on IRC: > bremner: ok, 1.2 explicitly no-oped skip_to() on an iterator at_end on > trunk that code has been rewritten without that explicit check, and > the iterator internals are NULL then i think restoring the check is > reasonable, though I'm not sure if we actually promise that's

Re: segfault with xapian 1.3.1

2013-09-06 Thread Olly Betts
Olly wrote on IRC: bremner: ok, 1.2 explicitly no-oped skip_to() on an iterator at_end on trunk that code has been rewritten without that explicit check, and the iterator internals are NULL then i think restoring the check is reasonable, though I'm not sure if we actually promise that's

[PATCH] Actually close the xapian database in notmuch_database_close

2012-03-01 Thread Olly Betts
On Thu, Mar 01, 2012 at 07:59:30AM +0100, Justus Winter wrote: > Olly wrote: > >It is hard to say if calling close() is actually useful here from just > >seeing the patch. > > Huh? I provided a test case... I only saw the part of the patch Austin quoted in the mail he cc-ed to me. > Quoting

Re: [PATCH] Actually close the xapian database in notmuch_database_close

2012-03-01 Thread Olly Betts
On Thu, Mar 01, 2012 at 07:59:30AM +0100, Justus Winter wrote: Olly wrote: It is hard to say if calling close() is actually useful here from just seeing the patch. Huh? I provided a test case... I only saw the part of the patch Austin quoted in the mail he cc-ed to me. Quoting Austin

[PATCH] Actually close the xapian database in notmuch_database_close

2012-02-29 Thread Olly Betts
On Wed, Feb 29, 2012 at 10:48:33AM -0500, Austin Clements wrote: > Quoth Justus Winter on Feb 29 at 10:19 am: > > Formerly the xapian database object was deleted and closed in its > > destructor once the object was garbage collected. Explicitly call > > close() so that the database and the

Re: [PATCH] Actually close the xapian database in notmuch_database_close

2012-02-29 Thread Olly Betts
On Wed, Feb 29, 2012 at 10:48:33AM -0500, Austin Clements wrote: Quoth Justus Winter on Feb 29 at 10:19 am: Formerly the xapian database object was deleted and closed in its destructor once the object was garbage collected. Explicitly call close() so that the database and the associated

doc: notmuch help search-terms, boolean operators

2011-04-27 Thread Olly Betts
On Tue, Apr 26, 2011 at 04:01:17PM -0700, Carl Worth wrote: > On Wed, 27 Apr 2011 00:24:38 +0200, Florian Friesdorf > wrote: > > Through playing with `notmuch tag` and `notmuch search > > --output=messages` I found: > > > > Complete list of boolean operators in order of precedence: > > - NOT >

Some Xapian tips and thoughts on rebuilding

2010-07-28 Thread Olly Betts
Kan-Ru Chen writes: > Seems this still does not work as expected. > > With the new `notmuch count' command: > > % notmuch count > 131720 > > After xapian-compact-1.1 > > % notmuch count > 1001 > > And a subsequent `notmuch new' > > % notmuch new > Processed 6

Re: Some Xapian tips and thoughts on rebuilding

2010-07-28 Thread Olly Betts
Kan-Ru Chen writes: Seems this still does not work as expected. With the new `notmuch count' command: % notmuch count 131720 After xapian-compact-1.1 % notmuch count 1001 And a subsequent `notmuch new' % notmuch new Processed 6 total files in

Re: [PATCH 2/5] Add quotes around id:message-id queries.

2010-07-05 Thread Olly Betts
On Fri, Jul 02, 2010 at 05:04:46PM +0400, Dmitry Kurochkin wrote: On Fri, 2 Jul 2010 04:41:43 + (UTC), Olly Betts o...@survex.com wrote: On 2010-07-01, Dmitry Kurochkin wrote: - (concat id: (notmuch-show-get-prop :id props))) + (concat id:\ (notmuch-show-get-prop :id props

[PATCH 2/5] Add quotes around id:"message-id" queries.

2010-07-02 Thread Olly Betts
On 2010-07-01, Dmitry Kurochkin wrote: > - (concat "id:" (notmuch-show-get-prop :id props))) > + (concat "id:\"" (notmuch-show-get-prop :id props) "\"")) This is probably a good idea (the ".." example is arguably a Xapian bug so that should be fixed soon, but you find all sorts of junk in

Re: [PATCH 2/5] Add quotes around id:message-id queries.

2010-07-01 Thread Olly Betts
On 2010-07-01, Dmitry Kurochkin wrote: - (concat id: (notmuch-show-get-prop :id props))) + (concat id:\ (notmuch-show-get-prop :id props) \)) This is probably a good idea (the .. example is arguably a Xapian bug so that should be fixed soon, but you find all sorts of junk in message-ids.

Failing test cases

2010-04-30 Thread Olly Betts
On 2010-04-28, Jason White wrote: > It seems to be repeatable here, even after running git clean -d -f -x, > followed by make && make test > > Is anyone else seeing this? Yes, I got the same failure trying to rebuild the notmuch 0.3.1 Debian package with Xapian 1.2.0 (building in a clean Debian

Re: Failing test cases

2010-04-29 Thread Olly Betts
On 2010-04-28, Jason White wrote: It seems to be repeatable here, even after running git clean -d -f -x, followed by make make test Is anyone else seeing this? Yes, I got the same failure trying to rebuild the notmuch 0.3.1 Debian package with Xapian 1.2.0 (building in a clean Debian sid

[PATCH] allow to not sort the search results

2010-04-16 Thread Olly Betts
On Fri, Apr 16, 2010 at 08:37:04AM +0200, Sebastian Spaeth wrote: > On 2010-04-15, Olly Betts wrote: > > Also, sorting by relevance requires more calculations and may require > > fetching additional data (document length for example). > > > > So I think it would make

Re: [PATCH] allow to not sort the search results

2010-04-16 Thread Olly Betts
On Fri, Apr 16, 2010 at 08:37:04AM +0200, Sebastian Spaeth wrote: On 2010-04-15, Olly Betts wrote: Also, sorting by relevance requires more calculations and may require fetching additional data (document length for example). So I think it would make sense for --sort=relevance and --sort

[PATCH] allow to not sort the search results

2010-04-15 Thread Olly Betts
Sebastian Spaeth writes: > On 2010-04-14, Jason White wrote: > > > Also add a --sort=unsorted command line option to notmuch search to test > > > this. > > > > Does this provide relevance-ranked search results? I think relevance ranking > > is the Xapian default if a sort order isn't specified.

Re: [PATCH] allow to not sort the search results

2010-04-15 Thread Olly Betts
Sebastian Spaeth writes: On 2010-04-14, Jason White wrote: Also add a --sort=unsorted command line option to notmuch search to test this. Does this provide relevance-ranked search results? I think relevance ranking is the Xapian default if a sort order isn't specified. Yes, by

[notmuch] Notmuch performance (literally, in my case)

2010-03-16 Thread Olly Betts
On Mon, Mar 15, 2010 at 10:29:36AM -0700, Ben Gamari wrote: > On Mon, 15 Mar 2010 09:29:35 + (UTC), Olly Betts > wrote: > > http://oligarchy.co.uk/xapian/patches/xapian-1.0.18-flint-group-fsyncs.patch > > > > What this does it to at least pair up the calls to fdatasy

Re: [notmuch] Notmuch performance (literally, in my case)

2010-03-16 Thread Olly Betts
On Mon, Mar 15, 2010 at 10:29:36AM -0700, Ben Gamari wrote: On Mon, 15 Mar 2010 09:29:35 + (UTC), Olly Betts o...@survex.com wrote: http://oligarchy.co.uk/xapian/patches/xapian-1.0.18-flint-group-fsyncs.patch What this does it to at least pair up the calls to fdatasync(). It's

[notmuch] Notmuch performance (literally, in my case)

2010-03-15 Thread Olly Betts
On 2010-03-15, Hans Dieter Pearcey wrote: > On Sun, 14 Mar 2010 22:59:28 -0700 (PDT), Ben Gamari wrote: >> Notmuch is using xapian 1.08-1.99karmic from the Xapian backports PPA, which >> I believe includes the recent database update optimizations. > > As far as I know, it doesn't. 1.0.18 is the

Re: [notmuch] Notmuch performance (literally, in my case)

2010-03-15 Thread Olly Betts
On 2010-03-15, Hans Dieter Pearcey wrote: On Sun, 14 Mar 2010 22:59:28 -0700 (PDT), Ben Gamari wrote: Notmuch is using xapian 1.08-1.99karmic from the Xapian backports PPA, which I believe includes the recent database update optimizations. As far as I know, it doesn't. 1.0.18 is the stable

[notmuch] Backport of Xapian term update optimisation

2010-02-15 Thread Olly Betts
Thanks for all the testing feedback folks. I've made a new Xapian release (1.0.18) and packaged it for Debian. If you are using unstable, and not on mips or hurd, it's now built: https://buildd.debian.org/status/package.php?p=xapian-core I haven't updated the xapian.org download page, etc yet

Re: [notmuch] Backport of Xapian term update optimisation

2010-02-14 Thread Olly Betts
Thanks for all the testing feedback folks. I've made a new Xapian release (1.0.18) and packaged it for Debian. If you are using unstable, and not on mips or hurd, it's now built: https://buildd.debian.org/status/package.php?p=xapian-core I haven't updated the xapian.org download page, etc yet

[notmuch] Notmuch performance problems on OSX

2010-02-09 Thread Olly Betts
On 2010-02-09, Oliver Charles wrote: > I just upgraded to xapian-core HEAD and notmuch master tip today, in > desparation to get away from GMail. Sadly it's still taking at least > 0.7s to tag a single thread (with one message). I'm really eager to > solve this, could anyone give me any pointers

[notmuch] strange behavior of indexing of and searching for strings containing '[]'

2010-02-05 Thread Olly Betts
On 2010-02-05, Jameson Rollins wrote: > Hey, folks. I've been noticing some strange behavior of notmuch search > results for strings containing '[]'. Here are some searches for some > exact strings in messages subjects: The '[]' is a red herring. Xapian's TermGenerator and QueryParser classes

[notmuch] Backport of Xapian term update optimisation

2010-02-04 Thread Olly Betts
On Thu, Feb 04, 2010 at 11:55:44AM -0500, micah anderson wrote: > Once this is available in unstable, the notmuch package should be > re-uploaded with a build-dependency on this version so that the package > users see the speed improvement. As it is now, the debian package is > pretty slow.

[notmuch] Backport of Xapian term update optimisation

2010-02-04 Thread Olly Betts
On Wed, Feb 03, 2010 at 02:35:14PM -0500, Jameson Rollins wrote: > On Thu, 28 Jan 2010 00:06:59 + (UTC), Olly Betts > wrote: > > I've backported the term update optimisation patches > > <http://trac.xapian.org/ticket/250> to Xapian's 1.0 branch, and you can &g

Re: [notmuch] Backport of Xapian term update optimisation

2010-02-03 Thread Olly Betts
On Wed, Feb 03, 2010 at 02:35:14PM -0500, Jameson Rollins wrote: On Thu, 28 Jan 2010 00:06:59 + (UTC), Olly Betts o...@survex.com wrote: I've backported the term update optimisation patches http://trac.xapian.org/ticket/250 to Xapian's 1.0 branch, and you can find snapshot tarballs

[notmuch] Internal error no thread ID

2010-01-28 Thread Olly Betts
On 2010-01-27, Matthias Teege wrote: > Internal error: Message with document ID of 62004 has no thread ID. > (lib/message.cc:353). > > Is it possible to get the filename for the document 62004? You can get it from the database using Xapian's delve utility: delve -d -r62004

[notmuch] Backport of Xapian term update optimisation

2010-01-28 Thread Olly Betts
I've backported the term update optimisation patches to Xapian's 1.0 branch, and you can find snapshot tarballs including these changes here: http://oligarchy.co.uk/xapian/branches/1.0/ Xapian's testsuite passes (including the additional test coverage which I

[notmuch] Backport of Xapian term update optimisation

2010-01-27 Thread Olly Betts
I've backported the term update optimisation patches http://trac.xapian.org/ticket/250 to Xapian's 1.0 branch, and you can find snapshot tarballs including these changes here: http://oligarchy.co.uk/xapian/branches/1.0/ Xapian's testsuite passes (including the additional test coverage which I

[notmuch] indexing mail?

2010-01-15 Thread Olly Betts
On 2010-01-15, Dirk-Jan C Binnema wrote: >Olly> Other than Linux, the d_type field is available mainly only on BSD >Olly> systems. > > Yes, my patch could me generalized a bit more, just like your patch could not > hardcode the '/'-separator :) Well, '/' works as a directory

[notmuch] indexing mail?

2010-01-15 Thread Olly Betts
On 2010-01-15, Dirk-Jan C Binnema wrote: >>>>>> "Olly" == Olly Betts writes: >Olly> Not a full patch, but I already posted what this code should look > like >Olly> to handle both systems without d_type, and those which return > DT_

[notmuch] Notmuch performance problems on OSX

2010-01-15 Thread Olly Betts
On 2010-01-14, Oliver Charles wrote: > I've installed the latest notmuch from Git at this time of writing, > along with Xapian from SVN head. However, just tagging a single thread > with only one message seems to take too long: One difference between OS X and other systems is that OS X supports

[notmuch] indexing mail?

2010-01-15 Thread Olly Betts
On 2010-01-14, Carl Worth wrote: > On Thu, 14 Jan 2010 18:38:54 +0100, Adrian Perez de Castro igalia.com> wrote: >> I am using XFS, which always returns DT_UNKNOWN. Taking into account that >> there is a good deal of people using filesystems other than the ones you >> mention, and that other

[notmuch] indexing encrypted messages (was: OpenPGP support)

2010-01-14 Thread Olly Betts
On 2010-01-08, James Westby wrote: > That would leave an open question over whether future notmuch show > invocations would return the plaintext or ciphertext. If it is the > latter then it requires decrypting every time you want to view it, but > it does mean that there is less information

Re: [notmuch] indexing encrypted messages (was: OpenPGP support)

2010-01-14 Thread Olly Betts
On 2010-01-08, James Westby wrote: That would leave an open question over whether future notmuch show invocations would return the plaintext or ciphertext. If it is the latter then it requires decrypting every time you want to view it, but it does mean that there is less information leakage

Re: [notmuch] indexing mail?

2010-01-14 Thread Olly Betts
On 2010-01-14, Carl Worth wrote: On Thu, 14 Jan 2010 18:38:54 +0100, Adrian Perez de Castro ape...@igalia.com wrote: I am using XFS, which always returns DT_UNKNOWN. Taking into account that there is a good deal of people using filesystems other than the ones you mention, and that other

Re: [notmuch] Notmuch performance problems on OSX

2010-01-14 Thread Olly Betts
On 2010-01-14, Oliver Charles wrote: I've installed the latest notmuch from Git at this time of writing, along with Xapian from SVN head. However, just tagging a single thread with only one message seems to take too long: One difference between OS X and other systems is that OS X supports the

Re: [notmuch] indexing mail?

2010-01-14 Thread Olly Betts
On 2010-01-15, Dirk-Jan C Binnema wrote: Olly == Olly Betts o...@survex.com writes: Olly Not a full patch, but I already posted what this code should look like Olly to handle both systems without d_type, and those which return DT_UNKNOWN: Olly http://article.gmane.org

Re: [notmuch] [PATCH] Add post-add and post-tag hooks

2009-12-25 Thread Olly Betts
[Sorry, I seemed to manage to attach my reply to the wrong thread...] On Wed, Dec 23, 2009 at 07:57:21AM +0100, Tomas Carnecky wrote: On 12/23/09 12:02 AM, Olly Betts wrote: Rather than a platform-specific check, it would be better to check if DT_DIR is defined. Beware that even on Linux

[notmuch] Rather simple optimization for notmuch tag

2009-12-24 Thread Olly Betts
Mark Anderson writes: > On Wed, 23 Dec 2009 03:45:14 +0000, Olly Betts wrote: > > Handling a combination of removals and additions is trickier, but probably > > possible, although the more tags you are dealing with, the less profitable > > the filtering is likely to be (as

[notmuch] [PATCH] Add post-add and post-tag hooks

2009-12-23 Thread Olly Betts
[Sorry, I seemed to manage to attach my reply to the wrong thread...] On Wed, Dec 23, 2009 at 07:57:21AM +0100, Tomas Carnecky wrote: > On 12/23/09 12:02 AM, Olly Betts wrote: >> Rather than a platform-specific check, it would be better to check if DT_DIR >> is defined. >>

[notmuch] Rather simple optimization for notmuch tag

2009-12-23 Thread Olly Betts
Carl Worth writes: > On Fri, 18 Dec 2009 00:49:00 -0700, Mark Anderson wrote: > > I was updating my poll script that tags messages, and a common idiom is > > to put > > tag +mytag and not tag:mytag > > > > I don't know anything about efficiency, but for the simple single-tag > > case, couldn't

[notmuch] [PATCH] Add post-add and post-tag hooks

2009-12-22 Thread Olly Betts
Tomas Carnecky writes: > #if defined(__sun__) > ... sprintf, stat etc > #else > (void) path; > return dirent->d_type == DT_DIR; > #endif Rather than a platform-specific check, it would be better to check if DT_DIR is defined. Beware that even on Linux (where the d_type field is

[notmuch] Missing messages breaking threads

2009-12-22 Thread Olly Betts
Carl Worth writes: > We don't have any concept of versioning yet, but it would obviously be > easy to have a new version document with an increasing integer. Adding a magic document for this isn't ideal as you have to make sure it can't appear in search results, etc. This is just the sort of

Re: [notmuch] [PATCH] Add post-add and post-tag hooks

2009-12-22 Thread Olly Betts
Tomas Carnecky writes: #if defined(__sun__) ... sprintf, stat etc #else (void) path; return dirent-d_type == DT_DIR; #endif Rather than a platform-specific check, it would be better to check if DT_DIR is defined. Beware that even on Linux (where the d_type field is

[notmuch] Notmuch's search view sucks

2009-12-04 Thread Olly Betts
Karl Wiberg writes: > On Fri, Dec 4, 2009 at 1:29 AM, Carl Worth wrote: > > And a step beyond that would support different languages for > > different emails, but that sounds like something "hard" to identify. > > But probably not as hard as identifying spam. It could probably be > done with a

Re: [notmuch] Notmuch's search view sucks

2009-12-04 Thread Olly Betts
Karl Wiberg writes: On Fri, Dec 4, 2009 at 1:29 AM, Carl Worth wrote: And a step beyond that would support different languages for different emails, but that sounds like something hard to identify. But probably not as hard as identifying spam. It could probably be done with a simple