[PATCH] Clarify usage of `additional_headers' in test/test-lib.sh:generate_message.
On Fri, 28 Jan 2011 08:45:35 +0100, Thomas Schwinge
wrote:
> Matches my thoughts :-) -- and as it occurs to me right now, doing it in
> one here document should be possible like this, if additional_headers is
> changed to have the newline *at the beginning* of the string:
...
> Date: ${template[date]}${additional_headers}
>
> ${template[body]}
That looks quite reasonable. I hadn't thought of that one myself.
> If you'd like me to prepare (and test) any of these, please tell.
Please feel free! I would appreciate it.
> PS: Didn't know you'd be doing a presentation of notmuch at LCA2011 -- I
> saw your announcement on the IRC channel (re live stream) what it was too
> late already. But then, it would have been a rather inconvenient time /
> timezone anyways, being based in Germany. So, how has it been?
I probably could have done a better job of letting people know that I
would be giving a talk. But then again, there wasn't that much in the
talk that people who read this list regularly don't already know[*].
For anyone interested, a video of the talk should be appearing on
http://linux.conf.au at some point.
The talk went fairly well, I think. But I spent a bit too much at the
beginning just explaining why we need notmuch, so I didn't have enough
time at the end to really show off notmuch well. And I didn't have the
demonstration portions of the talk as rehearsed as I should have, so
some parts of it were not smooth.
Ah well, I'll get better for the next talk.
-Carl
[*] Then again, Eric works right next to me two days a week and uses
exclusively notmuch for email; as we were attending the conference
together this week I mentioned two features in conversation with others
and he said, "What!? Notmuch can do that?! That's awesome". So that was
fun.
I think the two features were tab completion of tag names (which is a
really old feature by now) and the display of an error message if a pipe
fails (which is a fairly new feature).
--
carl.d.worth at intel.com
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL:
<http://notmuchmail.org/pipermail/notmuch/attachments/20110128/0d36c49b/attachment.pgp>
[PATCH 0/4] Versatile date/time parser
On 2011-01-23, Michal Sojka wrote: > Hi all, > > the following patch series brings into notmuch date/time parser stolen > from GNU coreutils. It can be applied on top of custom query parser > patches from Austin Clements. > > This is RFC and it not meant for merging. Another source for date parsing is perhaps date.c from git, which (probably) has much smaller (none?) dependencies. Tom
A tool for printing from notmuch
On Fri, 28 Jan 2011 16:11:00 -0500, Jesse Rosenthal wrote: > > Hmm... looks like a problem with ConfigParser. Maybe the default values > aren't working well? > > Try setting up a ~/.notmuchprintrc: > > notmuch_command: "/usr/local/bin/notmuch" > browser_command: "/usr/bin/firefox" > plain_text_font: "sans-serif" > always_prefer_plaintext: no > view_in_browser: yes It works! Very nice and thanks a lot! > > If that works, I need to futz with its defaults a bit. (Also, I just > noticed that I named a config boolean with the same name as a function: > view_in_browser. That shouldn't be the problem here, but I should fix it > all the same.) I was confused by that coincidence as well ... Best, Andreas
Xapian locking errors with custom query parser
Looks like I managed to remove the line that deletes (and thus closes and unlocks) the Xapian::Database object in a bout of overzealous code removal. The consequence was exactly what you suspected; there was a brief window after notmuch had exited before the database actually got unlocked. Here's the fix. cworth, what's the most convenient way for me to slip this in to the patch series? diff --git a/lib/database.cc b/lib/database.cc index 7253bdf..3c488a9 100644 --- a/lib/database.cc +++ b/lib/database.cc @@ -826,6 +826,7 @@ notmuch_database_close (notmuch_database_t *notmuch) } delete notmuch->term_gen; +delete notmuch->xapian_db; talloc_free (notmuch); } On Fri, Jan 28, 2011 at 11:35 AM, Austin Clements wrote: > This should most definitely not happen. ?I'll look in to it. > > On Fri, Jan 28, 2011 at 5:18 AM, Michal Sojka wrote: >> >> Hi Austin, >> >> when I switched to using your custom query parser I started experiencing >> "Unable to get write lock" errors when I run my initial tagging script. >> I thought that this was because I run the script while processing the >> mail in Emacs, but today I realized that this happens even without Emacs >> and that it can be reproduced by a simple command below. >> >> i=0; while notmuch tag +xxx tag:whatever; do i=$((i+1)); echo $i; done >> >> It seems that sometimes the unlocking of the Xapian database happens >> somewhat lazily and when the subsequent command starts I get: >> >> A Xapian exception occurred opening database: Unable to get write lock on >> /home/wsh/mail/.notmuch/xapian: already locked >> >> This happens almost regularly after a few iterations of the loop even if >> the query doesn't match anything. >> >> Do you have an idea what may cause this? >> >> Can you reproduce this on your system? >> >> The version I use can be obtained from >> >> ?git://rtime.felk.cvut.cz/notmuch.git custom-query-pasrser >> >> Thanks >> -Michal >> ___ >> notmuch mailing list >> notmuch at notmuchmail.org >> http://notmuchmail.org/mailman/listinfo/notmuch > >
A tool for printing from notmuch
Dear Jesse,
> Printing from notmuch is a bit of a pain. Muttprint doesn't really help
> much, because it can't handle multiparts well, doesn't know what to do
> with html, and will print out pages of base64 if you have
> attachments. And more often than not, what I need to print is an HTML
> email (a bus ticket or something).
>
> A solution I've been working on for my own use is here:
>
> http://commonmeasure.org/~jkr/notmuchprint
Thanks, that sounds great! I have been waiting for something like this
...
Unfortunately I could not get it to work with python2.7:
Traceback (most recent call last):
File "/home/amann/local/bin/notmuchprint", line 284, in
main(config)
File "/home/amann/local/bin/notmuchprint", line 238, in main
VIEW_IN_BROWSER = config.getboolean('DEFAULT', 'view_in_browser')
File "/usr/lib/python2.7/ConfigParser.py", line 360, in getboolean
v = self.get(section, option)
File "/usr/lib/python2.7/ConfigParser.py", line 581, in get
return self._interpolate(section, option, value, d)
File "/usr/lib/python2.7/ConfigParser.py", line 621, in _interpolate
if value and "%(" in value:
TypeError: argument of type 'bool' is not iterable
is this a known problem, or am I missing something obvious?
Next I commented out the offending line via
VIEW_IN_BROWSER = False
however still no luck:
Traceback (most recent call last):
File "/home/amann/local/bin/notmuchprint", line 284, in
main(config)
File "/home/amann/local/bin/notmuchprint", line 263, in main
nm_msg = NotmuchMsg(msgid, config)
File "/home/amann/local/bin/notmuchprint", line 58, in __init__
self.json_thread = json.loads(json_string)
File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 360, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 378, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Any hints?
Andreas
[PATCH] Have to configure and build inside the source directory.
Carl Worth writes: > What do other build systems generally do when running configure from > some other directory? Copy/link the Makefiles and then construct them > carefully such that they can find all the source files? I think the use of GNU make VPATH is fairly common. For example, I believe the autotools build a tree that matches the srcdir structure and add Makefiles that contain something like this: VPATH = ../../wherever/notmuch/thisdir ... Of course, given that, the build tree doesn't include any source files. FWIW -- Rob Browning rlb @defaultvalue.org and @debian.org GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4
A tool for printing from notmuch
On Fri, 28 Jan 2011 16:11:00 -0500, Jesse Rosenthal
wrote:
> Hmm... looks like a problem with ConfigParser. Maybe the default values
> aren't working well?
Yep, that was the problem. It turns out it doesn't take True and False
as defaults -- needs strings ("yes", "no", "true", "false) or 1 or
0. Anyway, I fixed that. Thanks, Andreas!
Best,
Jesse
A tool for printing from notmuch
Hi Andreas,
On Fri, 28 Jan 2011 20:59:06 +, Andreas Amann wrote:
> Unfortunately I could not get it to work with python2.7:
>
> Traceback (most recent call last):
> File "/home/amann/local/bin/notmuchprint", line 284, in
> main(config)
> File "/home/amann/local/bin/notmuchprint", line 238, in main
> VIEW_IN_BROWSER = config.getboolean('DEFAULT', 'view_in_browser')
> File "/usr/lib/python2.7/ConfigParser.py", line 360, in getboolean
> v = self.get(section, option)
> File "/usr/lib/python2.7/ConfigParser.py", line 581, in get
> return self._interpolate(section, option, value, d)
> File "/usr/lib/python2.7/ConfigParser.py", line 621, in _interpolate
> if value and "%(" in value:
> TypeError: argument of type 'bool' is not iterable
...
Hmm... looks like a problem with ConfigParser. Maybe the default values
aren't working well?
Try setting up a ~/.notmuchprintrc:
notmuch_command: "/usr/local/bin/notmuch"
browser_command: "/usr/bin/firefox"
plain_text_font: "sans-serif"
always_prefer_plaintext: no
view_in_browser: yes
If that works, I need to futz with its defaults a bit. (Also, I just
noticed that I named a config boolean with the same name as a function:
view_in_browser. That shouldn't be the problem here, but I should fix it
all the same.)
Best,
Jesse
[PATCH] Clarify usage of `additional_headers' in test/test-lib.sh:generate_message.
On Thu, 27 Jan 2011 02:17:21 -0700, Thomas Schwinge
wrote:
> +# Note that in the way we're setting it above and using it below,
> +# `additional_headers' will also serve as the header / body separator
> +# (empty line in between).
Thanks, Thomas.
I've merged this now.
I'd even prefer to have the newline explicitly in the HERE document, but
it's awkward to avoid having the "extra" newline at the end of
${additional_headers} the way I'm constructing it incrementally. So just
documenting the current approach is probably best for now.
-Carl
--
carl.d.worth at intel.com
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL:
<http://notmuchmail.org/pipermail/notmuch/attachments/20110128/fa1eb172/attachment.pgp>
notmuch's idea of concurrency / failing an invocation
On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge wrote: > > It would definitely be nice to avoid the complexity inherent in having a > > daemon, but how do you imagine "queue on a lock" to work? We don't have > > anything like that in place now. > > I suppose what he means is trying to get the lock, and if that fails wait > a bit / wait until it is available again. > > Actually, as a next step, wouldn't it also be possible to add some > heuristic to avoid ``notmuch new'' (being a low-priority task) blocking > some interactive user (UI; high-priority task)? But we can pursue such > schemes as soon as the basic infrastructure is in place. Couldn't we pretty much get the desired behavior by using flock(2)? Basically, take out a LOCK_EX when we need to write, and a LOCK_SH when we only need to read. Using the blocking form, things should pretty much just queue up and take their turn, right? I'm not familiar with Xapian, but if it doesn't give us something we could use this sort of locking on, couldn't we just add some /path/to/mail/.notmuch.lock file that we open to hold a lock on? We already have to specify if we want a read-only or read-write database handle in notmuch_database_open, so it seems like it'd be easy enough to hook in there. -- Mike Kelly
A tool for printing from notmuch
Dear all, Printing from notmuch is a bit of a pain. Muttprint doesn't really help much, because it can't handle multiparts well, doesn't know what to do with html, and will print out pages of base64 if you have attachments. And more often than not, what I need to print is an HTML email (a bus ticket or something). A solution I've been working on for my own use is here: http://commonmeasure.org/~jkr/notmuchprint It's a python script that takes a message-id and sends printable html to your browser or stdout. Plain text is put in tags, html mails are, as much as possible, left alone, except for the headers at the top. You can configure (in ~/.notmuchprintrc) the font for plain text emails, what browser to use, whether to view automatically or output html, and whether or not to prefer plain text if both are available. (Since it's mainly html emails I need to print or view, I set the last one to no.) It also has a few command line options. Type "--help" to get the scoop. The only requirement outside of the stdlib is BeautifulSoup, to deal with email html. It's been working well for me. I have the following in my .emacs so I can use it from show mode by pressing "z": (define-key notmuch-show-mode-map "z" '(lambda () (interactive) (shell-command (concat "notmuchprint '" (notmuch-show-get-message-id) "'" The single-quotes are necessary for msg-ids with dollar-signs and whatnot in them. A couple of points: 1. The HTML generation is intentionally pretty primitive. First, I didn't want to spend too much time on what is just a header box. Second, I didn't want to introduce any CSS that might screw with whatever godawful styles the email has. I'd welcome someone fixing this part up, but I still think a goal should be to make sure html emails remain unchanged. 2. It uses json instead of the python bindings for two reasons. First, for my own personal use-case (remote) the bindings wouldn't have helped. Second, I'm guessing that the slowdown from getting json output is more than compensated for by having gmime do the parsing instead of python. Anyway, if people find this useful, or want to make changes, I'll make a git repo. For now, though, I just wanted to share it, since it's been making my life a bit easier. Best, Jesse
[PATCH] Add a few tests for searching LWN emails.
Hi Carl and Thomas, On Sat, 29 Jan 2011 05:59:38 +1000 Carl Worth wrote: > Yes, I believe this is related to the dot in the name. From my > recollection a name with an address requires quoting. So the header > that is currently formatted as: > > From: LWN.net Weekly Notification > > should instead be: > > From: "LWN.net Weekly Notification" I am by no means an expert, but http://www.ietf.org/rfc/rfc2821.txt would seem to indicate that names with a '.' in them don't need to be quoted as there are several lines in the Scenarios section that look like: C: From: John Q. Public unless the problem is XXX.yyy (i.e. no spaces on either side of the '.'), but that seems like a pretty arbitrary differentiator (i.e. 'Q. ' is fine, but 'LWN.net' isn't) aren't the '<' and '>' the real delimiters here? If we do need to fix something, though, we'd be more than happy to do so I suspect ... jake -- Jake Edge - LWN - jake at lwn.net - http://lwn.net
notmuch's idea of concurrency / failing an invocation
On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements wrote: > I'm looking into breaking notmuch new up into small transactions. It > wouldn't be much a leap from there to simply close and reopen the database > between transactions if another task wants to use it, which would release > the lock and let the queued notmuch task have the database for a bit. That sounds like something very useful to pursue. Please continue! > It seems silly to have a daemon when all of notmuch's state is already on disk > and queue on a lock is as good as a queue in a daemon, but without the > accompanying architectural shenanigans. It would definitely be nice to avoid the complexity inherent in having a daemon, but how do you imagine "queue on a lock" to work? We don't have anything like that in place now. Another advantage that can happen with queueing (wherever it occurs) is to allow a client to be very responsive without waiting for an operation to complete. Though that can of course be band if the operation isn't reliably committed. -Carl -- carl.d.worth at intel.com -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110128/5d7df2b8/attachment-0001.pgp>
notmuch's idea of concurrency / failing an invocation
On Thu, 27 Jan 2011 19:20:00 +0100, Thomas Schwinge wrote: > Which is the original idea here? Is it that... There's no original idea yet. It's essentially an unsolved problem right now. > * each and every client should catch these kinds of errors, and retry, > or eventually give up at some point, and report the status to the > user; or is it that... > > * notmuch internally should catch these concurrency cases, and retry, > or eventually give up at some point (``notmuch --maximum-wait=30s tag > [...]''), and fail as seen above? Some people have actually already done work solutions in one way or another. Here are a few of the messages I found in my "outstanding notmuch mail to read"[*] queue: James Vasile patched the emacs interface to call notmuch asynchronously and to repeatedly call it if it fails (he also wonders if it should have some sort of timeout): id:"87vddnlxos.wl%james at hackervisions.org" James also wrote a shell script that repeatedly calls the notmuch binary as necessary (and he wonders if this retrying should happen inside notmuch itself): id:"87pr3sw43a.fsf at hackervisions.org" "servilio" wrote a new "notmuch repl" command which can accept notmuch operations expressed in text form on stdin, and then interpret and execute them. That's a good start on a notmuch daemon: id:"AANLkTi=7eCt0=NqUiJFrGDcaZ17LOd3qNNqN1-ASwYzr at mail.gmail.com" I'm not sure yet which approach (or approaches) we want. But I would love to see some of the limitations described in the messages above addressed. That would definitely make some of the patches more acceptable. -Carl [*] And yes, my queue really does span a year(!) or so. That's embarrassing. I'm committed to making progress on this queue and staying up-to-date with new patches, so I've made a couple of recent changes: 1. I'm now processing the queue largely in reverse-chronological order. The idea here is that I can stay on top of new posts, while also making progress on previously-sent items. This does mean that you can hack my workflow by replying to an old thread, (and thereby bringing it back to my attention). Please feel free to do that---ideally by mentioning any new information such as "these patches are now rebased " or "I've tested these patches in daily use for X months and they still apply fine to master" or similar. 2. I've date-limited my saved search for my notmuch queue to show a small number of messages. This is a cheap psychological hack. If the number on the queue is too large it makes me hesitant to even look at it. But with a small number, it's easier to make progress since the end is apparently in sight. Of course, once I reduce my date-limited queue to 0, I'll extend the date back into the past and try to keep working through things. -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110128/6bf21b58/attachment-0001.pgp>
notmuch's idea of concurrency / failing an invocation
Actually, this is trivial to play with. Here's a stop-gap wrapper script for people having trouble with Xapian locking, #!/bin/bash NOTMUCH_BIN="/path/to/notmuch" MAIL_DIR="/path/to/mailroot" ( case "$1" in setup|help) ;; search|show|count|reply|dump|search-tags|part) flock -s 200;; *) flock -x 200;; esac "$NOTMUCH_BIN" "$@" 200>&- ) 200>"$MAIL_DIR"/.notmuch/lock On Fri, Jan 28, 2011 at 11:57 AM, Austin Clements wrote: > > On Fri, Jan 28, 2011 at 10:36 AM, Mike Kelly wrote: >> >> On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge > schwinge.name> wrote: >> > > It would definitely be nice to avoid the complexity inherent in having a >> > > daemon, but how do you imagine "queue on a lock" to work? We don't have >> > > anything like that in place now. >> > >> > I suppose what he means is trying to get the lock, and if that fails wait >> > a bit / wait until it is available again. >> > >> > Actually, as a next step, wouldn't it also be possible to add some >> > heuristic to avoid ``notmuch new'' (being a low-priority task) blocking >> > some interactive user (UI; high-priority task)? ?But we can pursue such >> > schemes as soon as the basic infrastructure is in place. >> >> Couldn't we pretty much get the desired behavior by using flock(2)? >> Basically, take out a LOCK_EX when we need to write, and a LOCK_SH when >> we only need to read. Using the blocking form, things should pretty much >> just queue up and take their turn, right? >> >> I'm not familiar with Xapian, but if it doesn't give us something we >> could use this sort of locking on, couldn't we just add some >> /path/to/mail/.notmuch.lock file that we open to hold a lock on? > > Yes, exactly. ?All of this. ?Unfortunately, Xapian doesn't expose the ability > to block on the lock (see the fcntl call in backends/flint_lock.cc, which is > hard-coded to the non-blocking F_SETLK instead of F_SETLKW), so we'd either > need a new Xapian option, or we would just have to wrap our own flock/fcntl > lock around things as you suggest.
Remote usage script updated
On Thu, 27 Jan 2011 10:51:42 -0500, Jesse Rosenthal wrote: > Just a note to say that I finally got around to updating the remote > usage script on the wiki to what I'm using now. With "--format=raw" in, > it's all pretty straightforward. The only things the script does now > are: Thanks for passing on this note, Jesse! > I've actually switched over to keeping my messages on my IMAP server and > using this remote script on all of my computers. It avoids any need for > syncing. It's been working very well for me so far. It sounds entirely useful for a large class of users. Is there any reason we shouldn't just host the script in the notmuch repository itself? Let me know if there's anything I can do. -Carl -- carl.d.worth at intel.com -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110128/51898d9d/attachment.pgp>
notmuch's idea of concurrency / failing an invocation
On Fri, Jan 28, 2011 at 10:36 AM, Mike Kelly wrote: > On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge > wrote: > > > It would definitely be nice to avoid the complexity inherent in having > a > > > daemon, but how do you imagine "queue on a lock" to work? We don't have > > > anything like that in place now. > > > > I suppose what he means is trying to get the lock, and if that fails wait > > a bit / wait until it is available again. > > > > Actually, as a next step, wouldn't it also be possible to add some > > heuristic to avoid ``notmuch new'' (being a low-priority task) blocking > > some interactive user (UI; high-priority task)? But we can pursue such > > schemes as soon as the basic infrastructure is in place. > > Couldn't we pretty much get the desired behavior by using flock(2)? > Basically, take out a LOCK_EX when we need to write, and a LOCK_SH when > we only need to read. Using the blocking form, things should pretty much > just queue up and take their turn, right? > > I'm not familiar with Xapian, but if it doesn't give us something we > could use this sort of locking on, couldn't we just add some > /path/to/mail/.notmuch.lock file that we open to hold a lock on? > Yes, exactly. All of this. Unfortunately, Xapian doesn't expose the ability to block on the lock (see the fcntl call in backends/flint_lock.cc, which is hard-coded to the non-blocking F_SETLK instead of F_SETLKW), so we'd either need a new Xapian option, or we would just have to wrap our own flock/fcntl lock around things as you suggest. ------ next part -- An HTML attachment was scrubbed... URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110128/3f25e30c/attachment.html>
notmuch's idea of concurrency / failing an invocation
On Fri, Jan 28, 2011 at 4:45 AM, Thomas Schwinge wrote: > On Fri, 28 Jan 2011 15:10:01 +1000, Carl Worth wrote: > > On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements > wrote: > > > I'm looking into breaking notmuch new up into small transactions. It > > > wouldn't be much a leap from there to simply close and reopen the > database > > > between transactions if another task wants to use it, which would > release > > > the lock and let the queued notmuch task have the database for a bit. > > > > That sounds like something very useful to pursue. Please continue! > > Ack! And actually -- I just wondered about that: what happens if > ``notmuch new'' has executed notmuch_database_add_message for a new > message M, but then is killed before adding any tags and finishing up > (and supposing that the DB isn't in an invalid state now). This process > of adding M to the DB and applying any tags isn't one single transaction > currently, right? (And this is exactly what you're working on > chainging?) Am I right that what currently happens is that upon the next > ``notmuch new'' run, notmuch will not reconsider M (given that it already > is present in the DB), but continue with the next messages -- thus > leaving M without any tags? This isn't a very likely scenario, but still > a possible one. There are quite a few bugs like this. In fact, last night I added a test that interrupts notmuch new (for real, not SIGINT) after every database write, and on each interrupted database snapshot, re-runs notmuch new to completion, then checks that the database winds up in the correct state. There are dozens of interruption points where it doesn't, many of which are permanent, even if you force notmuch new to rescan the maildir. > Another advantage that can happen with queueing (wherever it occurs) is > > to allow a client to be very responsive without waiting for an operation > > to complete. Though that can of course be band if the operation isn't > > reliably committed. > > (Obviously this can only work as long as we don't immediatelly need the > operation's result; think ``notmuch show''.) > > So, if the DB has the functionality to internally queue and immediatelly > acknowledge transactions, and only later (reliably) commit them, wouldn't > that be fine indeed? For example, ``notmuch tag'' then wouldn't have to > wait for the DB to be writable. (And note that I have no idea whether > Xapian supports such things.) But on the other hand we would like to > immediatelly display the requested change in the UI, right? > This would be fantastic, if the client could indicate the difference between a "pending" change and a "committed" change as you suggest below. I don't think having the database lie about its commit state is the right way to do this, though (nor should the client lie about this, thus the "pending" display). A better way would be for the client to update the display to "pending", start the notmuch operation asynchronously, have the notmuch operation block and queue up on the database lock, then have the client update the display to "committed" when the asynchronous operation returns. No weird database operations or transactional semantics and the client side is fairly straightforward. -- next part -- An HTML attachment was scrubbed... URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110128/c62788b6/attachment.html>
Xapian locking errors with custom query parser
This should most definitely not happen. I'll look in to it. On Fri, Jan 28, 2011 at 5:18 AM, Michal Sojka wrote: > Hi Austin, > > when I switched to using your custom query parser I started experiencing > "Unable to get write lock" errors when I run my initial tagging script. > I thought that this was because I run the script while processing the > mail in Emacs, but today I realized that this happens even without Emacs > and that it can be reproduced by a simple command below. > > i=0; while notmuch tag +xxx tag:whatever; do i=$((i+1)); echo $i; done > > It seems that sometimes the unlocking of the Xapian database happens > somewhat lazily and when the subsequent command starts I get: > > A Xapian exception occurred opening database: Unable to get write lock on > /home/wsh/mail/.notmuch/xapian: already locked > > This happens almost regularly after a few iterations of the loop even if > the query doesn't match anything. > > Do you have an idea what may cause this? > > Can you reproduce this on your system? > > The version I use can be obtained from > > git://rtime.felk.cvut.cz/notmuch.git custom-query-pasrser > > Thanks > -Michal > ___ > notmuch mailing list > notmuch at notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch > -- next part -- An HTML attachment was scrubbed... URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110128/97bda4b5/attachment.html>
Xapian locking errors with custom query parser
Hi Austin, when I switched to using your custom query parser I started experiencing "Unable to get write lock" errors when I run my initial tagging script. I thought that this was because I run the script while processing the mail in Emacs, but today I realized that this happens even without Emacs and that it can be reproduced by a simple command below. i=0; while notmuch tag +xxx tag:whatever; do i=$((i+1)); echo $i; done It seems that sometimes the unlocking of the Xapian database happens somewhat lazily and when the subsequent command starts I get: A Xapian exception occurred opening database: Unable to get write lock on /home/wsh/mail/.notmuch/xapian: already locked This happens almost regularly after a few iterations of the loop even if the query doesn't match anything. Do you have an idea what may cause this? Can you reproduce this on your system? The version I use can be obtained from git://rtime.felk.cvut.cz/notmuch.git custom-query-pasrser Thanks -Michal
notmuch's idea of concurrency / failing an invocation
Hallo! On Fri, 28 Jan 2011 15:10:01 +1000, Carl Worth wrote: > On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements > wrote: > > I'm looking into breaking notmuch new up into small transactions. It > > wouldn't be much a leap from there to simply close and reopen the database > > between transactions if another task wants to use it, which would release > > the lock and let the queued notmuch task have the database for a bit. > > That sounds like something very useful to pursue. Please continue! Ack! And actually -- I just wondered about that: what happens if ``notmuch new'' has executed notmuch_database_add_message for a new message M, but then is killed before adding any tags and finishing up (and supposing that the DB isn't in an invalid state now). This process of adding M to the DB and applying any tags isn't one single transaction currently, right? (And this is exactly what you're working on chainging?) Am I right that what currently happens is that upon the next ``notmuch new'' run, notmuch will not reconsider M (given that it already is present in the DB), but continue with the next messages -- thus leaving M without any tags? This isn't a very likely scenario, but still a possible one. > > It seems silly to have a daemon when all of notmuch's state is already on > > disk > > and queue on a lock is as good as a queue in a daemon, but without the > > accompanying architectural shenanigans. Ack, too. A daemon seems one abstraction layer too much. (But I'm not actively opposed either, if someone has a valid use for such a scheme.) > It would definitely be nice to avoid the complexity inherent in having a > daemon, but how do you imagine "queue on a lock" to work? We don't have > anything like that in place now. I suppose what he means is trying to get the lock, and if that fails wait a bit / wait until it is available again. Actually, as a next step, wouldn't it also be possible to add some heuristic to avoid ``notmuch new'' (being a low-priority task) blocking some interactive user (UI; high-priority task)? But we can pursue such schemes as soon as the basic infrastructure is in place. > Another advantage that can happen with queueing (wherever it occurs) is > to allow a client to be very responsive without waiting for an operation > to complete. Though that can of course be band if the operation isn't > reliably committed. (Obviously this can only work as long as we don't immediatelly need the operation's result; think ``notmuch show''.) So, if the DB has the functionality to internally queue and immediatelly acknowledge transactions, and only later (reliably) commit them, wouldn't that be fine indeed? For example, ``notmuch tag'' then wouldn't have to wait for the DB to be writable. (And note that I have no idea whether Xapian supports such things.) But on the other hand we would like to immediatelly display the requested change in the UI, right? What notmuch-show.el:notmuch-show-remove-tag currently does is *not* re-asking the DB for a message's current tags after having removed a specific one, but instead it interprets the tag removal command itself -- which is easy enough in this case, and rather unlikely to ever yield different results, at least unless there's another process operating on the DB concurrently. Otherwise, the other way round, the client could maintain a list of to-do items, to which actions are added if the DB is currently busy, and this list is periodically worked on in order to get it empty. For example, tag changes that are in this list, but not yet committed in the DB could be displayed in another color in the UI. But doing so would shift the responsibility to the UI, which should be in the DB, in my humble opinion. (Actually, this issue feels similar to the one who should be doing the re-trying in case the DB is busy: the UI, or the notmuch process itself, which we're discussing in another thread.) As you can guess I'm not very much into DBs, and neither too much into concurrent systems, so if my ideas don't make sense, please feel free to refer me to literature. Gr??e, Thomas -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110128/a3d67691/attachment.pgp>
[PATCH 1/3] new: Do not defer maildir flag synchronization during the first run
On Thu, 27 Jan 2011 19:08:38 -0500, Austin Clements wrote: > I added a "notmuch_database_remove_message_get" to the public API that's > just like "notmuch_database_remove_message" except that it also returns a > notmuch_message_t if other instances of the message still exist. It feels > clunky to have two almost identical variants of this function. Is this the > preferred way to change the public API? Or should I just add the argument > to the existing function and fix the other three calls to it? Just adding an argument to the public API without library version bump would break my python bindings. So if we modify the public API, I would prefer if we rename the function and remove the old one completely. That I could detect at least. Sebastian -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110128/2378ba4f/attachment.pgp>
[PATCH] Clarify usage of `additional_headers' in test/test-lib.sh:generate_message.
Hallo!
On Fri, 28 Jan 2011 15:36:25 +1000, Carl Worth wrote:
> On Thu, 27 Jan 2011 02:17:21 -0700, Thomas Schwinge
> wrote:
> > +# Note that in the way we're setting it above and using it below,
> > +# `additional_headers' will also serve as the header / body separator
> > +# (empty line in between).
> I'd even prefer to have the newline explicitly in the HERE document, but
> it's awkward to avoid having the "extra" newline at the end of
> ${additional_headers} the way I'm constructing it incrementally. So just
> documenting the current approach is probably best for now.
Matches my thoughts :-) -- and as it occurs to me right now, doing it in
one here document should be possible like this, if additional_headers is
changed to have the newline *at the beginning* of the string:
cat <"$gen_msg_filename"
From: ${template[from]}
To: ${template[to]}
Message-Id: <${gen_msg_id}>
Subject: ${template[subject]}
Date: ${template[date]}${additional_headers}
${template[body]}
EOF
Or, of course, we could split the here document: base header,
conditionally (if set at all) additional_headers, new line, body.
If you'd like me to prepare (and test) any of these, please tell.
Gr??e,
Thomas
PS: Didn't know you'd be doing a presentation of notmuch at LCA2011 -- I
saw your announcement on the IRC channel (re live stream) what it was too
late already. But then, it would have been a rather inconvenient time /
timezone anyways, being based in Germany. So, how has it been?
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL:
<http://notmuchmail.org/pipermail/notmuch/attachments/20110128/a7d49d4e/attachment.pgp>
Re: Xapian locking errors with custom query parser
This should most definitely not happen. I'll look in to it. On Fri, Jan 28, 2011 at 5:18 AM, Michal Sojka sojk...@fel.cvut.cz wrote: Hi Austin, when I switched to using your custom query parser I started experiencing Unable to get write lock errors when I run my initial tagging script. I thought that this was because I run the script while processing the mail in Emacs, but today I realized that this happens even without Emacs and that it can be reproduced by a simple command below. i=0; while notmuch tag +xxx tag:whatever; do i=$((i+1)); echo $i; done It seems that sometimes the unlocking of the Xapian database happens somewhat lazily and when the subsequent command starts I get: A Xapian exception occurred opening database: Unable to get write lock on /home/wsh/mail/.notmuch/xapian: already locked This happens almost regularly after a few iterations of the loop even if the query doesn't match anything. Do you have an idea what may cause this? Can you reproduce this on your system? The version I use can be obtained from git://rtime.felk.cvut.cz/notmuch.git custom-query-pasrser Thanks -Michal ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: notmuch's idea of concurrency / failing an invocation
On Fri, Jan 28, 2011 at 10:36 AM, Mike Kelly pi...@pioto.org wrote: On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge tho...@schwinge.name wrote: It would definitely be nice to avoid the complexity inherent in having a daemon, but how do you imagine queue on a lock to work? We don't have anything like that in place now. I suppose what he means is trying to get the lock, and if that fails wait a bit / wait until it is available again. Actually, as a next step, wouldn't it also be possible to add some heuristic to avoid ``notmuch new'' (being a low-priority task) blocking some interactive user (UI; high-priority task)? But we can pursue such schemes as soon as the basic infrastructure is in place. Couldn't we pretty much get the desired behavior by using flock(2)? Basically, take out a LOCK_EX when we need to write, and a LOCK_SH when we only need to read. Using the blocking form, things should pretty much just queue up and take their turn, right? I'm not familiar with Xapian, but if it doesn't give us something we could use this sort of locking on, couldn't we just add some /path/to/mail/.notmuch.lock file that we open to hold a lock on? Yes, exactly. All of this. Unfortunately, Xapian doesn't expose the ability to block on the lock (see the fcntl call in backends/flint_lock.cc, which is hard-coded to the non-blocking F_SETLK instead of F_SETLKW), so we'd either need a new Xapian option, or we would just have to wrap our own flock/fcntl lock around things as you suggest. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: notmuch's idea of concurrency / failing an invocation
Actually, this is trivial to play with. Here's a stop-gap wrapper script for people having trouble with Xapian locking, #!/bin/bash NOTMUCH_BIN=/path/to/notmuch MAIL_DIR=/path/to/mailroot ( case $1 in setup|help) ;; search|show|count|reply|dump|search-tags|part) flock -s 200;; *) flock -x 200;; esac $NOTMUCH_BIN $@ 200- ) 200$MAIL_DIR/.notmuch/lock On Fri, Jan 28, 2011 at 11:57 AM, Austin Clements amdra...@mit.edu wrote: On Fri, Jan 28, 2011 at 10:36 AM, Mike Kelly pi...@pioto.org wrote: On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge tho...@schwinge.name wrote: It would definitely be nice to avoid the complexity inherent in having a daemon, but how do you imagine queue on a lock to work? We don't have anything like that in place now. I suppose what he means is trying to get the lock, and if that fails wait a bit / wait until it is available again. Actually, as a next step, wouldn't it also be possible to add some heuristic to avoid ``notmuch new'' (being a low-priority task) blocking some interactive user (UI; high-priority task)? But we can pursue such schemes as soon as the basic infrastructure is in place. Couldn't we pretty much get the desired behavior by using flock(2)? Basically, take out a LOCK_EX when we need to write, and a LOCK_SH when we only need to read. Using the blocking form, things should pretty much just queue up and take their turn, right? I'm not familiar with Xapian, but if it doesn't give us something we could use this sort of locking on, couldn't we just add some /path/to/mail/.notmuch.lock file that we open to hold a lock on? Yes, exactly. All of this. Unfortunately, Xapian doesn't expose the ability to block on the lock (see the fcntl call in backends/flint_lock.cc, which is hard-coded to the non-blocking F_SETLK instead of F_SETLKW), so we'd either need a new Xapian option, or we would just have to wrap our own flock/fcntl lock around things as you suggest. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Remote usage script updated
On Fri, 28 Jan 2011 12:54:17 +1000, Carl Worth cwo...@cworth.org wrote: On Thu, 27 Jan 2011 10:51:42 -0500, Jesse Rosenthal jrosent...@jhu.edu wrote It sounds entirely useful for a large class of users. Is there any reason we shouldn't just host the script in the notmuch repository itself? Let me know if there's anything I can do. -Carl I just realized I single-replied to Carl (after all the time I spent making sure I didn't group-reply by accident). Just to repeat briefly what I said there -- there was a slight problem with yesterday's script (wild-cards were eaten by the shell) that should now be fixed. And I'd be happy to house it in the repository if that seems like it would be useful to people. But I'm curious about whether we should have a separate tools/utilities folder. I know this came up before with notmuch-deliver and other similar tools. What was the outcome of that discussion? Best, Jesse ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: A tool for printing from notmuch
Dear Jesse,
Printing from notmuch is a bit of a pain. Muttprint doesn't really help
much, because it can't handle multiparts well, doesn't know what to do
with html, and will print out pages of base64 if you have
attachments. And more often than not, what I need to print is an HTML
email (a bus ticket or something).
A solution I've been working on for my own use is here:
http://commonmeasure.org/~jkr/notmuchprint
Thanks, that sounds great! I have been waiting for something like this
...
Unfortunately I could not get it to work with python2.7:
Traceback (most recent call last):
File /home/amann/local/bin/notmuchprint, line 284, in module
main(config)
File /home/amann/local/bin/notmuchprint, line 238, in main
VIEW_IN_BROWSER = config.getboolean('DEFAULT', 'view_in_browser')
File /usr/lib/python2.7/ConfigParser.py, line 360, in getboolean
v = self.get(section, option)
File /usr/lib/python2.7/ConfigParser.py, line 581, in get
return self._interpolate(section, option, value, d)
File /usr/lib/python2.7/ConfigParser.py, line 621, in _interpolate
if value and %( in value:
TypeError: argument of type 'bool' is not iterable
is this a known problem, or am I missing something obvious?
Next I commented out the offending line via
VIEW_IN_BROWSER = False
however still no luck:
Traceback (most recent call last):
File /home/amann/local/bin/notmuchprint, line 284, in module
main(config)
File /home/amann/local/bin/notmuchprint, line 263, in main
nm_msg = NotmuchMsg(msgid, config)
File /home/amann/local/bin/notmuchprint, line 58, in __init__
self.json_thread = json.loads(json_string)
File /usr/lib/python2.7/json/__init__.py, line 326, in loads
return _default_decoder.decode(s)
File /usr/lib/python2.7/json/decoder.py, line 360, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File /usr/lib/python2.7/json/decoder.py, line 378, in raw_decode
raise ValueError(No JSON object could be decoded)
ValueError: No JSON object could be decoded
Any hints?
Andreas
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: A tool for printing from notmuch
Hi Andreas,
On Fri, 28 Jan 2011 20:59:06 +, Andreas Amann a.am...@ucc.ie wrote:
Unfortunately I could not get it to work with python2.7:
Traceback (most recent call last):
File /home/amann/local/bin/notmuchprint, line 284, in module
main(config)
File /home/amann/local/bin/notmuchprint, line 238, in main
VIEW_IN_BROWSER = config.getboolean('DEFAULT', 'view_in_browser')
File /usr/lib/python2.7/ConfigParser.py, line 360, in getboolean
v = self.get(section, option)
File /usr/lib/python2.7/ConfigParser.py, line 581, in get
return self._interpolate(section, option, value, d)
File /usr/lib/python2.7/ConfigParser.py, line 621, in _interpolate
if value and %( in value:
TypeError: argument of type 'bool' is not iterable
...
Hmm... looks like a problem with ConfigParser. Maybe the default values
aren't working well?
Try setting up a ~/.notmuchprintrc:
notmuch_command: /usr/local/bin/notmuch
browser_command: /usr/bin/firefox
plain_text_font: sans-serif
always_prefer_plaintext: no
view_in_browser: yes
If that works, I need to futz with its defaults a bit. (Also, I just
noticed that I named a config boolean with the same name as a function:
view_in_browser. That shouldn't be the problem here, but I should fix it
all the same.)
Best,
Jesse
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: A tool for printing from notmuch
On Fri, 28 Jan 2011 16:11:00 -0500, Jesse Rosenthal jrosent...@jhu.edu wrote: Hmm... looks like a problem with ConfigParser. Maybe the default values aren't working well? Yep, that was the problem. It turns out it doesn't take True and False as defaults -- needs strings (yes, no, true, false) or 1 or 0. Anyway, I fixed that. Thanks, Andreas! Best, Jesse ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [ANN] notmuch-deliver
On Tue, 11 Jan 2011 12:46:38 +0100, Thomas Schwinge tho...@schwinge.name wrote: What's the best way to advertise this to potential users? I recently put a description and link onto the notmuch web pages. Great. Thanks for doing this. Should we include a separate utils directory in the notmuch repository with auxiliary programs like this? I wouldn't do so. But that is not a very strong opinion of mine. Well, your opinion matters quite a bit in a case like this where it's your utility. I guess I was simply offering the hosting of the repository if you thought it would be useful. I understand the desire to keep things cleanly packaged separately. I'll also take the liberty to put stuff from the mailing list or IRC discussions into web pages, for we have to document this notmuch beast ;-), and it's better to have a generic place to refer people to, instead of discussing the same things more than once. Yes! Please continue to improve the web pages, and everyone, please feel free to grab useful stuff from the lists or IRC and stuff them into the web pages. Also, if things should get shoved into the notmuch man page, then we should do that too. -Carl -- carl.d.wo...@intel.com pgphaZbfohfIw.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] Various small clean-ups to doc ID set code.
On Wed, 8 Dec 2010 17:01:53 -0500, Austin Clements amdra...@mit.edu wrote: Remove the repeated sizeof (doc_ids-bitmap[0]) that bothered cworth by instead defining macros to compute the word and bit offset of a given bit in the bitmap. Don't require the caller of _notmuch_doc_id_set_init to pass in a correct bound; instead compute it from the array. This simplifies the caller and makes this interface easier to use correctly. ... +#define BITMAP_WORD(bit) ((bit) / sizeof (unsigned int)) +#define BITMAP_BIT(bit) ((bit) % sizeof (unsigned int)) These macros look great, they definitely simplify the code. _notmuch_doc_id_set_init (void *ctx, notmuch_doc_id_set_t *doc_ids, - GArray *arr, unsigned int bound) + GArray *arr) ... +for (unsigned int i = 0; i arr-len; i++) + max = MAX(max, g_array_index (arr, unsigned int, i)); And computing an argument automatically definitely makes the interface easier to use. So that's good too. But those two changes are independent so really need to be in separate commits. -if (doc_id = doc_ids-bound) +if (doc_id doc_ids-max) And this looks really like a *third* independent change to me. A code change like the above has the chance to introduce (or fix) an off-by-one bug---or even leave the code effectively unchanged as the intent is here. In order to distinguish all of those cases, I'd like to see a change like this as a minimal change, and described in the commit message. (Rather than hidden amongst various cleanups that are mostly about replacing some common code with a macro.) So I'd be happy to see this patch broken up and sent again. -Carl -- carl.d.wo...@intel.com pgpBv80dp1bEp.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 3/4] Optimize thread search using matched docid sets.
On Wed, 8 Dec 2010 16:58:44 -0500, Austin Clements amdra...@mit.edu wrote: Now that this is in (and I have a temporary respite from TA duties), I'm going to finish up and send out my other ~1.7X improvement, just to get it out of my queue. Then I'll look at making a performance regression suite. Were you thinking of some standard set of timed operations wrapped in a little script that can tell you if you've made things worse, or something more elaborate? I recently started making a perf/notmuch-perf script for notmuch (see below). I was doing this in preparation for my linux.conf.au talk on notmuch, (though I ended up not talking about performance in concrete terms). I don't know how much further I'll run with this now, but if this is a useful starting place for anyone, let me know and I can obviously add this to the repository. So the idea with this script is that the timed operations actually depend on local data, (your current mail collection as indicated by NOTMUCH_CONFIG). So the operations aren't standardized to enable comparison between different people, (unless they also agree on some common mail collection). My script as attached runs only notmuch new to time the original indexing. Beyond that I'd like to time some common operations, (adding a new message, searching for a single message, searching for many messages, searching for all messages, etc.). And then on top of this, I'd like to have a little utility that could compare several different runs captured previously. That would let me do the regression testing I'd like to ensure we never make performance worse. Please feel free to run with this or with your own approach as you see fit. -Carl pgpxBMsIPtpem.pgp Description: PGP signature notmuch-perf Description: Binary data ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [ANN] notmuch-deliver
On Wed, 12 Jan 2011 15:50:24 -0500, Austin Clements amdra...@mit.edu wrote: Out of curiosity, has anyone considered using inotify to monitor maildirs for new mail to hand to notmuch? For systems supporting inotify (or equivalents), this would have the advantage of being compatible with any delivery mechanism, be it a mail server, procmail, or emacs fcc'ing a maildir. The idea has definitely been floated before. If I search for: notmuch search to:notmuch@notmuchmail.org inotify in my collection I see 5 messages sent to the list, (other than yours). But none of those include any code that I have seen. It sounds like playing with inotify and notmuch might make a fun weekend project for somebody. Is anyone looking for something like that? -Carl -- carl.d.wo...@intel.com pgpyyv7GQDdu5.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] Have to configure and build inside the source directory.
+if ! { : configure; } 2 /dev/null; then
+cat EOF
+*** Error: You have to configure and build in the source directory.
+
+EOF
+exit 1
+fi
Rather than documenting a limitation here, why don't we do what people
actually want.
What do other build systems generally do when running configure from
some other directory? Copy/link the Makefiles and then construct them
carefully such that they can find all the source files?
That doesn't sound like it would be that hard.
-Carl
--
carl.d.wo...@intel.com
pgpIVO1lbvFpk.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: About the json output and the number of results shown.
On Wed, 12 Jan 2011 19:37:21 +0100, Christophe-Marie Duquesne chm.duque...@gmail.com wrote: So I am wondering: what is the point of having a tool that is able to output json and ending in not using it? Is there a solution to make the json output more useable? One solution I've been thinking about would be to add an option: the range of results to show (something like --range=25:50). Is it doable easily? This is fairly easy to do, yes. We even had functionality like this once, and I'll probably even add it back soon, (since a client like the vim interface isn't able to do the kind of asynchronous processing that you would really want). One problem with the ranged output (for notmuch search at least) is that small ranges with large initial offsets will take longer than expected. This is because in this case notmuch can't directly use Xapian's range offset support. The user is asking for an offset as a number of threads, but within Xapian we only have messages stored. So notmuch will have to search for messages from the beginning, construct a bunch of useless threads, and then throw those threads away after doing no more than counting them. This inefficiency in this API was one of the reasons I dropped this functionality before. It's pretty ugly. But I don't see a really good answer for that. feature request. In any case, do you have any proposal for making sense of this json output without modifications in the notmuch CLI? We've run into basically the same issue with the emacs interface. We've been avoiding using the json output precisely because the emacs JSON parsing would need to see all the output before it could start parsing. And that wouldn't give us the responsive user interface that we want. One idea I've had for this is to change the output (perhaps with a command-line option) to avoid emitting the outer array. That is, the results would instead be a series of independent JSON objects rather than a single JSON object. That should let the application treat things quickly by simply calling the JSON parser for each complete object. (Though, here, the application would likely want a cheap way to know when the input represented a complete object.) If anyone wants to help improve our JSON output here, then that would be great. For any change to the structure of the JSON output, I'd also like to see some documentation added to specify that structure clearly. -Carl -- carl.d.wo...@intel.com pgpG6sRtRfMiH.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: About the json output and the number of results shown.
On Wed, 12 Jan 2011 22:39:45 +, Mike Kelly pi...@pioto.org wrote: For starters, if I'm simply trying to retrieve a single message, the interface is rather awkard. I seem to need to do something like: my $json = `notmuch show --format=json id:$message_id`; my $parsed_json = decode_json($json); my $message = $parsed_json-[0][0][0]; That does seem fairly awkward, yes. Do you have a suggestion for how you'd like the output to be structured instead? And, when I'm doing my search earlier to even find those message ids, I need to do a check to `notmuch count` first to see if I'll even get any results, because the 0 result case is not valid JSON. Yikes! That's a bug in notmuch that we should get fixed rather than you just working around it. I just started adding a test for this case. Currently: notmuch search --format=json string that matches nothing returns nothing. Presumably, this should return just an empty json array instead, (that is, [])? -Carl -- carl.d.wo...@intel.com pgpfgp0DWCsaf.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [ANN] notmuch-deliver
On Tue, 11 Jan 2011 16:01:00 +0200, Felipe Contreras felipe.contre...@gmail.com wrote: I think this should be part of notmuch itself, I'll be happy to see any proposed additions for this. (And to the extent that some of this functionality exists in patches already proposed and just waiting for me, then I'm already happy about that too!) and there should be a configuration to use this as Fcc, instead of relying on the mail composer. This way both emacs and vim interfaces would share the same configuration regarding the Fcc/Bcc preference. Sharing the configuration as much as possible is definitely good. But Fcc is going to have to rely quite a bit on the mail composer. Currently, notmuch isn't involved at all in the sending of a mail, and it's not until a mail is actually sent that it's time to deliver the message to the Fcc location. So even getting notmuch to become informed about the message at Fcc time will require modification of the mail composer. -Carl -- carl.d.wo...@intel.com pgpskZKD0gE7f.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: A tool for printing from notmuch
On Fri, 28 Jan 2011 16:11:00 -0500, Jesse Rosenthal jrosent...@jhu.edu wrote: Hmm... looks like a problem with ConfigParser. Maybe the default values aren't working well? Try setting up a ~/.notmuchprintrc: notmuch_command: /usr/local/bin/notmuch browser_command: /usr/bin/firefox plain_text_font: sans-serif always_prefer_plaintext: no view_in_browser: yes It works! Very nice and thanks a lot! If that works, I need to futz with its defaults a bit. (Also, I just noticed that I named a config boolean with the same name as a function: view_in_browser. That shouldn't be the problem here, but I should fix it all the same.) I was confused by that coincidence as well ... Best, Andreas ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] Have to configure and build inside the source directory.
Carl Worth cwo...@cworth.org writes: What do other build systems generally do when running configure from some other directory? Copy/link the Makefiles and then construct them carefully such that they can find all the source files? I think the use of GNU make VPATH is fairly common. For example, I believe the autotools build a tree that matches the srcdir structure and add Makefiles that contain something like this: VPATH = ../../wherever/notmuch/thisdir ... Of course, given that, the build tree doesn't include any source files. FWIW -- Rob Browning rlb @defaultvalue.org and @debian.org GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Xapian locking errors with custom query parser
Looks like I managed to remove the line that deletes (and thus closes and unlocks) the Xapian::Database object in a bout of overzealous code removal. The consequence was exactly what you suspected; there was a brief window after notmuch had exited before the database actually got unlocked. Here's the fix. cworth, what's the most convenient way for me to slip this in to the patch series? diff --git a/lib/database.cc b/lib/database.cc index 7253bdf..3c488a9 100644 --- a/lib/database.cc +++ b/lib/database.cc @@ -826,6 +826,7 @@ notmuch_database_close (notmuch_database_t *notmuch) } delete notmuch-term_gen; +delete notmuch-xapian_db; talloc_free (notmuch); } On Fri, Jan 28, 2011 at 11:35 AM, Austin Clements amdra...@mit.edu wrote: This should most definitely not happen. I'll look in to it. On Fri, Jan 28, 2011 at 5:18 AM, Michal Sojka sojk...@fel.cvut.cz wrote: Hi Austin, when I switched to using your custom query parser I started experiencing Unable to get write lock errors when I run my initial tagging script. I thought that this was because I run the script while processing the mail in Emacs, but today I realized that this happens even without Emacs and that it can be reproduced by a simple command below. i=0; while notmuch tag +xxx tag:whatever; do i=$((i+1)); echo $i; done It seems that sometimes the unlocking of the Xapian database happens somewhat lazily and when the subsequent command starts I get: A Xapian exception occurred opening database: Unable to get write lock on /home/wsh/mail/.notmuch/xapian: already locked This happens almost regularly after a few iterations of the loop even if the query doesn't match anything. Do you have an idea what may cause this? Can you reproduce this on your system? The version I use can be obtained from git://rtime.felk.cvut.cz/notmuch.git custom-query-pasrser Thanks -Michal ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 0/4] Versatile date/time parser
On 2011-01-23, Michal Sojka wrote: Hi all, the following patch series brings into notmuch date/time parser stolen from GNU coreutils. It can be applied on top of custom query parser patches from Austin Clements. This is RFC and it not meant for merging. Another source for date parsing is perhaps date.c from git, which (probably) has much smaller (none?) dependencies. Tom ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
