[PATCH 00/10] CLI output versioning
Quoth Mark Walters on Dec 08 at 9:29 am: > > Hi > > Overall this series looks good. As we discussed on irc I think i would > prefer global NOTMUCH_SCHEMA_MIN as I am a little worried about these > proliferating (eg if someone decides text output also needs versioning > etc) In addition, if we do find the distinction useful it would be easy > to add at a later date. Will do. > One tiny comment on the manpage updates: now that you mention two return > values explicitly should the other possibilities be mentioned too or are > they so obvious it is not needed? I thought about this, but couldn't figure out what to say for exit status 1 other than "Unspecified error". Hence I carefully worded the man page to say *special* exit status codes. > Would it be worth having some emacs test for the error handling? (eg set > notmuch-command to something giving some stderr and an error) Inherently > these code paths won't be tested much so I think tests could be > particularly useful. Good idea. > Best wishes > > Mark > > > > > > > On Mon, 03 Dec 2012, Austin Clements wrote: > > (Sorry; I forgot to include a cover letter.) > > > > This series is intended to help with our long-standing output format > > versioning issue. While the JSON format is amenable to extension, > > there's still a high barrier to extensions because of the need to > > support them going forward, and an even higher barrier to modifications > > that break backwards compatibility. Versioning will make the format > > more dynamic, enabling us to easily improve and iterate on it. It will > > also address the slew of confusing bugs that people encounter when they > > use a mismatched CLI and front-end. > > > > On IRC we've talking about adding version information to the output > > format itself. This series takes a different and, I think, better > > approach: callers request a specific output format version on the > > command line. This allows notmuch to remain backwards compatible with > > older format versions when it's easy or necessary. This also doesn't > > require shoehorning a version number into the output, which would be > > awkward for both the CLI and the consumer. > > > > I called the argument --use-schema, but I'm open to other suggestions. > > --use-schema is technically accurate, but perhaps not as self-describing > > as something like --schema-version or --format-version (to parallel > > --format). > > ___ > > notmuch mailing list > > notmuch at notmuchmail.org > > http://notmuchmail.org/mailman/listinfo/notmuch -- Austin Clements MIT/'06/PhD/CSAIL amdragon at mit.edu http://web.mit.edu/amdragon Somewhere in the dream we call reality you will find me, searching for the reality we call dreams.
[PATCH 07/10] emacs: Use --use-schema for search
Quoth Mark Walters on Dec 08 at 8:48 am: > On Sun, 02 Dec 2012, Austin Clements wrote: > > We detect the special exit statuses and use these to produce specific > > diagnostic messages. > > --- > > emacs/notmuch-lib.el | 32 > > emacs/notmuch.el | 17 + > > 2 files changed, 45 insertions(+), 4 deletions(-) > > > > diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el > > index 9402456..49b0da6 100644 > > --- a/emacs/notmuch-lib.el > > +++ b/emacs/notmuch-lib.el > > @@ -325,6 +325,38 @@ string), a property list of face attributes, or a list > > of these." > > (put-text-property pos next 'face (cons face cur)) > > (setq pos next) > > > > +(defun notmuch-pop-up-error (msg) > > + "Pop up an error buffer displaying MSG." > > + > > + (let ((buf (get-buffer-create "*Notmuch errors*"))) > > +(with-current-buffer buf > > + (view-mode) > > + (let ((inhibit-read-only t)) > > + (erase-buffer) > > + (insert msg) > > + (unless (bolp) > > + (insert "\n")) > > + (goto-char (point-min > > +(pop-to-buffer buf))) > > I am not sure about the erase-buffer in the above: do we definitely want > to remove all previous error information? For version mismatch possibly > we do but in patch 9 it is done for all show command-line error returns. Why wouldn't we want to use a cleared buffer in all cases? notmuch-pop-up-error is only ever used when a command terminates, so there's no danger of us clearing errors that the user hasn't seen yet. > Incidentally why does show always pop-up an error but search only for > version-mismatches? Historical reasons, I suppose. I was maintaining the status quo for search (which already had *some* error handling), but there was no error-handling status quo for show. Probably search should be more vocal when the command fails. > Otherwise this looks good to me (but I am not that familiar with lisp > error handling) > > Best wishes > > Mark > > > > > > > > +(defun notmuch-version-mismatch-error (exit-status) > > + "Signal a schema version mismatch error. > > + > > +EXIT-STATUS must be the exit status of the notmuch CLI command, > > +and must have the value 20 or 21. This function will pop up an > > +error buffer with a descriptive message and signal an error." > > + (cond ((= exit-status 20) > > +(notmuch-pop-up-error "Error: Version mismatch. > > +Emacs requested an older output format than supported by the notmuch CLI. > > +You may need to restart Emacs or upgrade your notmuch Emacs package.")) > > + ((= exit-status 21) > > +(notmuch-pop-up-error "Error: Version mismatch. > > +Emacs requested a newer output format than supported by the notmuch CLI. > > +You may need to restart Emacs or upgrade your notmuch package.")) > > + (t > > +(error "Bad exit status %d" exit-status))) > > + (error "notmuch CLI version mismatch")) > > + > > ;; Compatibility functions for versions of emacs before emacs 23. > > ;; > > ;; Both functions here were copied from emacs 23 with the following > > copyright: > > diff --git a/emacs/notmuch.el b/emacs/notmuch.el > > index f9454d8..e1f28ca 100644 > > --- a/emacs/notmuch.el > > +++ b/emacs/notmuch.el > > @@ -644,6 +644,7 @@ of the result." > > (exit-status (process-exit-status proc)) > > (never-found-target-thread nil)) > > (when (memq status '(exit signal)) > > + (catch 'return > > (kill-buffer (process-get proc 'parse-buf)) > > (if (buffer-live-p buffer) > > (with-current-buffer buffer > > @@ -655,8 +656,16 @@ of the result." > > (insert "Incomplete search results (search process was > > killed).\n")) > > (when (eq status 'exit) > > (insert "End of search results.") > > - (unless (= exit-status 0) > > - (insert (format " (process returned %d)" exit-status))) > > + (cond ((or (= exit-status 20) (= exit-status 21)) > > + (kill-buffer) > > + (condition-case err > > + (notmuch-version-mismatch-error exit-status) > > +;; Strange things happen when you signal > > +;; an error from a sentinel. > > +(error (throw 'return nil > > + ((/= exit-status 0) > > + (insert (format " (process returned %d)" > > + exit-status > > (insert "\n") > > (if (and atbob > > (not (string= notmuch-search-target-thread > > "found"))) > > @@ -664,7 +673,7 @@ of the result." > > (when (and never-found-target-thread > >notmuch-search-target-line) > > (goto-char (point-min)) > > - (forward-line (1- notmuch-search-target-line > > + (forward-line (1- notmuch-search-target-line)
notmuch python bindings corrupt db index (was: gmail importer script)
On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke
wrote:
> OK, i assumed i'd not have to download all messages because i already have
> them
> on disk -- synced via offlineimap. but then i remembered that OI stores uses
> its own naming scheme for the files in a maildir. so thats probably why your
> script
> did not realize theyre already there. (maybe it should? it could look up the
> MIDs from notmuch)
Gmail bandwidth restrictions make this unfeasible. Requesting mail
header IDs via IMAP will rate limit the listing to 2 mails per second.
The best gmail will allow us to do at quasi rapid speed (~180kB/s) is
ask for the IMAP sequence number and the unique gmail 64bit number.
So, the maildir scheme winds up being "${gmail-unique-number}.gmail",
and the script doesn't redownload based on the preexistence of a file
with the corresponding name.
> i am kind of reluctant to reproduce this again (on my work-desktop).
It'd really be a huge help if you could backup and try...
> All i did was start your script, writing to the index in another terminal in
> between (using alot)
> possibly my cronjob fired a "notmuch new; afew.." in between also.
> when I killed your script the index war broken.
>
> Can i turn on some global logging for notmuch to debug this?
> Maybe libnotmuch had trouble with so many db-writes at a time?
That is very wrong and bizarre. I open the database in read/write
mode, which is supposed to lock other programs from accessing it in
this way... Very strange.
Re: [PATCH 00/10] CLI output versioning
Quoth Mark Walters on Dec 08 at 9:29 am: > > Hi > > Overall this series looks good. As we discussed on irc I think i would > prefer global NOTMUCH_SCHEMA_MIN as I am a little worried about these > proliferating (eg if someone decides text output also needs versioning > etc) In addition, if we do find the distinction useful it would be easy > to add at a later date. Will do. > One tiny comment on the manpage updates: now that you mention two return > values explicitly should the other possibilities be mentioned too or are > they so obvious it is not needed? I thought about this, but couldn't figure out what to say for exit status 1 other than "Unspecified error". Hence I carefully worded the man page to say *special* exit status codes. > Would it be worth having some emacs test for the error handling? (eg set > notmuch-command to something giving some stderr and an error) Inherently > these code paths won't be tested much so I think tests could be > particularly useful. Good idea. > Best wishes > > Mark > > > > > > > On Mon, 03 Dec 2012, Austin Clements wrote: > > (Sorry; I forgot to include a cover letter.) > > > > This series is intended to help with our long-standing output format > > versioning issue. While the JSON format is amenable to extension, > > there's still a high barrier to extensions because of the need to > > support them going forward, and an even higher barrier to modifications > > that break backwards compatibility. Versioning will make the format > > more dynamic, enabling us to easily improve and iterate on it. It will > > also address the slew of confusing bugs that people encounter when they > > use a mismatched CLI and front-end. > > > > On IRC we've talking about adding version information to the output > > format itself. This series takes a different and, I think, better > > approach: callers request a specific output format version on the > > command line. This allows notmuch to remain backwards compatible with > > older format versions when it's easy or necessary. This also doesn't > > require shoehorning a version number into the output, which would be > > awkward for both the CLI and the consumer. > > > > I called the argument --use-schema, but I'm open to other suggestions. > > --use-schema is technically accurate, but perhaps not as self-describing > > as something like --schema-version or --format-version (to parallel > > --format). > > ___ > > notmuch mailing list > > notmuch@notmuchmail.org > > http://notmuchmail.org/mailman/listinfo/notmuch -- Austin Clements MIT/'06/PhD/CSAIL amdra...@mit.edu http://web.mit.edu/amdragon Somewhere in the dream we call reality you will find me, searching for the reality we call dreams. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 07/10] emacs: Use --use-schema for search
Quoth Mark Walters on Dec 08 at 8:48 am: > On Sun, 02 Dec 2012, Austin Clements wrote: > > We detect the special exit statuses and use these to produce specific > > diagnostic messages. > > --- > > emacs/notmuch-lib.el | 32 > > emacs/notmuch.el | 17 + > > 2 files changed, 45 insertions(+), 4 deletions(-) > > > > diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el > > index 9402456..49b0da6 100644 > > --- a/emacs/notmuch-lib.el > > +++ b/emacs/notmuch-lib.el > > @@ -325,6 +325,38 @@ string), a property list of face attributes, or a list > > of these." > > (put-text-property pos next 'face (cons face cur)) > > (setq pos next) > > > > +(defun notmuch-pop-up-error (msg) > > + "Pop up an error buffer displaying MSG." > > + > > + (let ((buf (get-buffer-create "*Notmuch errors*"))) > > +(with-current-buffer buf > > + (view-mode) > > + (let ((inhibit-read-only t)) > > + (erase-buffer) > > + (insert msg) > > + (unless (bolp) > > + (insert "\n")) > > + (goto-char (point-min > > +(pop-to-buffer buf))) > > I am not sure about the erase-buffer in the above: do we definitely want > to remove all previous error information? For version mismatch possibly > we do but in patch 9 it is done for all show command-line error returns. Why wouldn't we want to use a cleared buffer in all cases? notmuch-pop-up-error is only ever used when a command terminates, so there's no danger of us clearing errors that the user hasn't seen yet. > Incidentally why does show always pop-up an error but search only for > version-mismatches? Historical reasons, I suppose. I was maintaining the status quo for search (which already had *some* error handling), but there was no error-handling status quo for show. Probably search should be more vocal when the command fails. > Otherwise this looks good to me (but I am not that familiar with lisp > error handling) > > Best wishes > > Mark > > > > > > > > +(defun notmuch-version-mismatch-error (exit-status) > > + "Signal a schema version mismatch error. > > + > > +EXIT-STATUS must be the exit status of the notmuch CLI command, > > +and must have the value 20 or 21. This function will pop up an > > +error buffer with a descriptive message and signal an error." > > + (cond ((= exit-status 20) > > +(notmuch-pop-up-error "Error: Version mismatch. > > +Emacs requested an older output format than supported by the notmuch CLI. > > +You may need to restart Emacs or upgrade your notmuch Emacs package.")) > > + ((= exit-status 21) > > +(notmuch-pop-up-error "Error: Version mismatch. > > +Emacs requested a newer output format than supported by the notmuch CLI. > > +You may need to restart Emacs or upgrade your notmuch package.")) > > + (t > > +(error "Bad exit status %d" exit-status))) > > + (error "notmuch CLI version mismatch")) > > + > > ;; Compatibility functions for versions of emacs before emacs 23. > > ;; > > ;; Both functions here were copied from emacs 23 with the following > > copyright: > > diff --git a/emacs/notmuch.el b/emacs/notmuch.el > > index f9454d8..e1f28ca 100644 > > --- a/emacs/notmuch.el > > +++ b/emacs/notmuch.el > > @@ -644,6 +644,7 @@ of the result." > > (exit-status (process-exit-status proc)) > > (never-found-target-thread nil)) > > (when (memq status '(exit signal)) > > + (catch 'return > > (kill-buffer (process-get proc 'parse-buf)) > > (if (buffer-live-p buffer) > > (with-current-buffer buffer > > @@ -655,8 +656,16 @@ of the result." > > (insert "Incomplete search results (search process was > > killed).\n")) > > (when (eq status 'exit) > > (insert "End of search results.") > > - (unless (= exit-status 0) > > - (insert (format " (process returned %d)" exit-status))) > > + (cond ((or (= exit-status 20) (= exit-status 21)) > > + (kill-buffer) > > + (condition-case err > > + (notmuch-version-mismatch-error exit-status) > > +;; Strange things happen when you signal > > +;; an error from a sentinel. > > +(error (throw 'return nil > > + ((/= exit-status 0) > > + (insert (format " (process returned %d)" > > + exit-status > > (insert "\n") > > (if (and atbob > > (not (string= notmuch-search-target-thread > > "found"))) > > @@ -664,7 +673,7 @@ of the result." > > (when (and never-found-target-thread > >notmuch-search-target-line) > > (goto-char (point-min)) > > - (forward-line (1- notmuch-search-target-line > > + (forward-line (1- notmuch-search-target-line)
notmuch python bindings corrupt db index (was: gmail importer script)
Austin Clements writes: > > There should be no way to corrupt the database at this level through > the Xapian API, which means nothing libnotmuch can do (much less users > of libnotmuch) should be able to corrupt the database. If you can > reproduce the problem, it's probably a serious bug in Xapian, but it > could also have been a file system bug or even random file system > corruption. One thing that Olly Betts mentioned is that there is a recently fixed bug in xapian with respect to multiple threads in the same process http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695542 I don't _think_ this is relevant here, but I mention it for completeness. d
notmuch python bindings corrupt db index (was: gmail importer script)
Quoth Jason A. Donenfeld on Dec 12 at 7:35 pm: > On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke > wrote: > > i am kind of reluctant to reproduce this again (on my work-desktop). > > It'd really be a huge help if you could backup and try... > > > All i did was start your script, writing to the index in another terminal > > in between (using alot) > > possibly my cronjob fired a "notmuch new; afew.." in between also. > > when I killed your script the index war broken. > > > > Can i turn on some global logging for notmuch to debug this? > > Maybe libnotmuch had trouble with so many db-writes at a time? > > That is very wrong and bizarre. I open the database in read/write > mode, which is supposed to lock other programs from accessing it in > this way... Very strange. There should be no way to corrupt the database at this level through the Xapian API, which means nothing libnotmuch can do (much less users of libnotmuch) should be able to corrupt the database. If you can reproduce the problem, it's probably a serious bug in Xapian, but it could also have been a file system bug or even random file system corruption.
Re: notmuch python bindings corrupt db index (was: gmail importer script)
Austin Clements writes: > > There should be no way to corrupt the database at this level through > the Xapian API, which means nothing libnotmuch can do (much less users > of libnotmuch) should be able to corrupt the database. If you can > reproduce the problem, it's probably a serious bug in Xapian, but it > could also have been a file system bug or even random file system > corruption. One thing that Olly Betts mentioned is that there is a recently fixed bug in xapian with respect to multiple threads in the same process http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695542 I don't _think_ this is relevant here, but I mention it for completeness. d ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: notmuch python bindings corrupt db index (was: gmail importer script)
Quoth Jason A. Donenfeld on Dec 12 at 7:35 pm: > On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke > wrote: > > i am kind of reluctant to reproduce this again (on my work-desktop). > > It'd really be a huge help if you could backup and try... > > > All i did was start your script, writing to the index in another terminal > > in between (using alot) > > possibly my cronjob fired a "notmuch new; afew.." in between also. > > when I killed your script the index war broken. > > > > Can i turn on some global logging for notmuch to debug this? > > Maybe libnotmuch had trouble with so many db-writes at a time? > > That is very wrong and bizarre. I open the database in read/write > mode, which is supposed to lock other programs from accessing it in > this way... Very strange. There should be no way to corrupt the database at this level through the Xapian API, which means nothing libnotmuch can do (much less users of libnotmuch) should be able to corrupt the database. If you can reproduce the problem, it's probably a serious bug in Xapian, but it could also have been a file system bug or even random file system corruption. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: notmuch python bindings corrupt db index (was: gmail importer script)
On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke wrote:
> OK, i assumed i'd not have to download all messages because i already have
> them
> on disk -- synced via offlineimap. but then i remembered that OI stores uses
> its own naming scheme for the files in a maildir. so thats probably why your
> script
> did not realize theyre already there. (maybe it should? it could look up the
> MIDs from notmuch)
Gmail bandwidth restrictions make this unfeasible. Requesting mail
header IDs via IMAP will rate limit the listing to 2 mails per second.
The best gmail will allow us to do at quasi rapid speed (~180kB/s) is
ask for the IMAP sequence number and the unique gmail 64bit number.
So, the maildir scheme winds up being "${gmail-unique-number}.gmail",
and the script doesn't redownload based on the preexistence of a file
with the corresponding name.
> i am kind of reluctant to reproduce this again (on my work-desktop).
It'd really be a huge help if you could backup and try...
> All i did was start your script, writing to the index in another terminal in
> between (using alot)
> possibly my cronjob fired a "notmuch new; afew.." in between also.
> when I killed your script the index war broken.
>
> Can i turn on some global logging for notmuch to debug this?
> Maybe libnotmuch had trouble with so many db-writes at a time?
That is very wrong and bizarre. I open the database in read/write
mode, which is supposed to lock other programs from accessing it in
this way... Very strange.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
