Re: Notmuch DB Problems

2018-09-11 Thread Mueen Nawaz

Olly Betts  writes:

It doesn't make much sense to me that holding the lock alone 
could be causing any sort of corruption - that's just an fcntl() 
lock. 

I would suggest to make sure you're running Xapian 1.4.7 as that 
fixed a cursor handling bug which affected notmuch.  I didn't 
find a way to make it corrupt on-disk data, but it's hard to be 
completely certain that it couldn't ever do that, so ruling out 
that as a cause would be good. 


I was running 1.4.5 - maybe that's the cause?


--
Don't take life so seriously.  It won't last.


   /\  /\   /\  /
  /  \/  \ u e e n /  \/  a w a z
  >>mu...@nawaz.org<<
  anl


___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: Notmuch DB Problems

2018-09-10 Thread Mueen Nawaz

David Bremner  writes:

Here's why it would freeze: 

I have a post-new hook that runs a Python script. Depending on 
whether the new email it is processing matches a rule I have, 
it will fire off an email to the sender using the SMTP library 
in Python. 

I had recently upgraded my MTA (PostFix), and it had a backward 
incompatible change that broke my config. I don't know why, but 
I could still send emails via Emacs, but when I tried to send 
them via Python, Postfix would log an error and it would not 
send. The Python statement would freeze (I guess Postfix 
doesn't return an appropriate response?  Not sure why).  
 
I have a cron job to run "notmuch new" 3 times an hour. Since 
the hook was frozen, so was the notmuch new command. I had 
quite a lot of "notmuch new" processes. I assume this meant the 
DB was locked all this time for writing. 


notmuch unlocks the database before running the hook, so I don't 
understand how a hung hook results in a locked database. If it 
happens again (or you're motivated to set up a testbed) I'd be 
interested in the output of 


Well, it results in a locked database because I have this in the 
(Python) hook:


DATABASE = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)

Soon after that I freeze the new messages. And at the end I thaw 
them out. The hang occurs in between the two, I think.


Also, is this by chance a network file system? Because those 
often break locking. 


No - regular hard drive.

Now killing all those jobs did not fix the database. It was 
still broken. And as we saw the second time round, it was 
/really/ broken - it would not even open in read-only mode. 


That seems like something the Xapian devs (in copy) might be 
interested in fixing, if you could come up with a simple 
reproducer. 


I can think of two experiments:

1. Write a hook that opens the database as above, and then just 
does nothing (e.g. while True). Let it run, say, for 24 
hours. (Not sure if the "freeze" part is relevant.


2. Same as the above, but have a cron job that fires "notmuch new" 
every 20 minutes. This will freeze on the database line above (all 
except the first invocation which will be stuck at while True).


After a day of this, check if you can open the database in 
READ_WRITE mode. 

notmuch could be cleverer about timing out on trying to acquire 
a lock. I suspect it's a bit delicate to get that right, and 
I've been hoping the underlying primitives would get a bit more 
flexible w.r.t. locking. 


I agree having notmuch handle it is not ideal. I was originally 
thinking there should be a default timeout that one can adjust as 
needed. However, when someone does "notmuch new" to build a new 
database, that can take several minutes. And others may have flows 
very different from mine.


At the very least, we probably should know why the DB be clobbered 
at all.


--
Don't take life so seriously.  It won't last.


   /\  /\   /\  /
  /  \/  \ u e e n /  \/  a w a z
  >>mu...@nawaz.org<<
  anl


___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: Notmuch DB Problems

2018-09-09 Thread Mueen Nawaz
Jani Nikula  writes:

> It might be interesting to see an strace log to possibly get an idea
> where it gets stuck.
>
> Is the filesystem writable and working okay?
>
> If search and show work, I'm guessing it gets stuck in trying to open
> the database writable. One hackish idea is to patch notmuch dump to open
> the database in read-only mode, and dump the tags. See below. The dump
> command opens the database writable to prevent changes while
> dumping. (Arguably this could be a command line option for cases like
> yours.)

Thanks - your patch worked. I dumped all the tags, deleted the database,
rebuilt it and restored the tags. All was well.

Until the following day at noon I noticed the problem was back. By
evening, I could not even do queries - it wouldn't open even in read
only mode. The database was dead.

After a lot of poking around, I figured out the problem, and this may be
of interest to the developers (although not sure if it is a xapian issue
or a notmuch issue).

Here's why it would freeze:

I have a post-new hook that runs a Python script. Depending on whether
the new email it is processing matches a rule I have, it will fire off
an email to the sender using the SMTP library in Python.

I had recently upgraded my MTA (PostFix), and it had a backward
incompatible change that broke my config. I don't know why, but I could
still send emails via Emacs, but when I tried to send them via Python,
Postfix would log an error and it would not send. The Python statement
would freeze (I guess Postfix doesn't return an appropriate response?
Not sure why). 


I have a cron job to run "notmuch new" 3 times an hour. Since the hook
was frozen, so was the notmuch new command. I had quite a lot of
"notmuch new" processes. I assume this meant the DB was locked all this
time for writing.

Now killing all those jobs did not fix the database. It was still
broken. And as we saw the second time round, it was /really/ broken - it
would not even open in read-only mode.

It is scary that if a post-new hook freezes while the database is
locked, it could (eventually) clobber the database. I don't know if
notmuch can do anything to prevent this outcome?

BTW, I think the DB would die only after a while. In my experiments, if
I killed the hook soon (e.g. under 1 minute), the database seemed fine. 

-- 
Don't use a big word where a diminutive one will suffice.


/\  /\   /\  /
   /  \/  \ u e e n /  \/  a w a z
   >>mu...@nawaz.org<<
   anl

 
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Notmuch DB Problems

2018-09-05 Thread mueen

Hi,

A few days ago I noticed notmuch new was no longer working (I have it
as a cron job so it took a while to figure it out).

It just freezes. I do have a Python hook, and it was freezing on the
line that opens the database.

I tried a notmuch dump. Same problem - freezes

Based on some earlier threads, I tried a notmuch compact. Same problem
- freezes.

All these freezes seem to use no memory/CPU. 

Interestingly, queries work fine - from both the command line and the
Emacs interface. So I can read old stuff just fine. But all the
commands above cause a freeze. 

Currently using notmuch-0.24.2. I tried notmuch-0.27 - same problem.

Results of a xapian check:

docdata:
blocksize=8K items=6 firstunused=3 revision=6442 levels=0 root=0
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=178562 firstunused=53441 revision=6442 levels=2
root=46086
/usr/bin/xapian-check: DatabaseError: 1 unused block(s) missing from
the free list, first is 0

What are my options? Unfortunately the last dump I have is many months
old, so I'm a bit wary of deleting the database and rebuilding. Given
that the show and search commands work, I was wondering if I can write
a script to get all the message/thread ID's for all the tags and store
them, and then rebuild the database and use that stored information to
retag all my messages (all without using the dump command)?

Thanks!

Mueen

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 0/4] notmuch reply bugfix & reply to sender only

2012-01-07 Thread Mueen Nawaz
Jameson Graef Rollins
 writes:

> On Thu,  5 Jan 2012 22:25:11 +0200, Jani Nikula  wrote:
>> Bikeshedding topic #1: How about making replying to just the sender the 
>> default
>> in "notmuch reply", and having --reply-all option (instead of 
>> --no-reply-all)?
>> 
>> Bikeshedding topic #2: How about binding 'r' to reply to just the sender by
>> default, and making 'R' reply-all (instead of vice versa)?
>
> I personally like both of these suggestions, and would not be bothered
> by the changed default, so I support both of these changes.

I too feel that 'r' should reply only to sender, and 'R' should reply
all. Having reply-all as default is a pain.




Re: [PATCH 0/4] notmuch reply bugfix reply to sender only

2012-01-07 Thread Mueen Nawaz
Jameson Graef Rollins
jroll...@finestructure.net writes:

 On Thu,  5 Jan 2012 22:25:11 +0200, Jani Nikula j...@nikula.org wrote:
 Bikeshedding topic #1: How about making replying to just the sender the 
 default
 in notmuch reply, and having --reply-all option (instead of 
 --no-reply-all)?
 
 Bikeshedding topic #2: How about binding 'r' to reply to just the sender by
 default, and making 'R' reply-all (instead of vice versa)?

 I personally like both of these suggestions, and would not be bothered
 by the changed default, so I support both of these changes.

I too feel that 'r' should reply only to sender, and 'R' should reply
all. Having reply-all as default is a pain.


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


a python terminal gui?

2011-05-20 Thread Mueen Nawaz
Patrick Totzke 
writes:


> * use libraries wherever possible. Therefore, instead of
>   programming raw curses, I go for the urwid toolkit: It's a bit like gtk,
>   you create a bunch of widgets and put them together in a tree-like structure
>   and the library handles redrawing the screen and redirecting key presses 
> etc.
>   I'm aware of some arguments against urwid (PEP8 compatibility, slow py3k 
> awareness),
>   but consider the advantages:

Not too long ago, I felt likewise and even looked at urwid for precisely
this goal. Unfortunately, real life interfered, and continues to do so,
so it may be a long time before I can fiddle with it. 

This, IMO, is much needed. I personally think I have most of the things
I need (as opposed to want) working in Emacs, and as an Emacs user, I'd
like to stick to Emacs. 

However, I know Python, and don't know elisp - so it'd be great to have
a Python interface. If there's a feature I want, I could one day try
implementing it in that interface, show to everyone here how cool it is,
and hopefully some soul will feel deprived and port it to the Emacs
interface ;-)

I'd encourage you to post this on the Wiki on the notmuch web site. 

On a side note, I see that someone created an entry for notmuch on Open
Hatch (openhatch.org). However, no requests for features has been
added. I really don't know how effective Open Hatch is, but Carl or
someone else may want to consider utilizing it. 

(For those still confused, Open Hatch is a web site that encourages
people to contribute to open source projects - via code, via docs, via
web maintenance, etc. It seems to integrate with existing bug tracking
sites, and has a bunch of other cool features as well. You can see their
Google Tech Talk at: http://www.youtube.com/watch?v=Qt3yPm7mKEc)



Re: a python terminal gui?

2011-05-20 Thread Mueen Nawaz
Patrick Totzke patricktot...@googlemail.com
writes:


 * use libraries wherever possible. Therefore, instead of
   programming raw curses, I go for the urwid toolkit: It's a bit like gtk,
   you create a bunch of widgets and put them together in a tree-like structure
   and the library handles redrawing the screen and redirecting key presses 
 etc.
   I'm aware of some arguments against urwid (PEP8 compatibility, slow py3k 
 awareness),
   but consider the advantages:

Not too long ago, I felt likewise and even looked at urwid for precisely
this goal. Unfortunately, real life interfered, and continues to do so,
so it may be a long time before I can fiddle with it. 

This, IMO, is much needed. I personally think I have most of the things
I need (as opposed to want) working in Emacs, and as an Emacs user, I'd
like to stick to Emacs. 

However, I know Python, and don't know elisp - so it'd be great to have
a Python interface. If there's a feature I want, I could one day try
implementing it in that interface, show to everyone here how cool it is,
and hopefully some soul will feel deprived and port it to the Emacs
interface ;-)

I'd encourage you to post this on the Wiki on the notmuch web site. 

On a side note, I see that someone created an entry for notmuch on Open
Hatch (openhatch.org). However, no requests for features has been
added. I really don't know how effective Open Hatch is, but Carl or
someone else may want to consider utilizing it. 

(For those still confused, Open Hatch is a web site that encourages
people to contribute to open source projects - via code, via docs, via
web maintenance, etc. It seems to integrate with existing bug tracking
sites, and has a bunch of other cool features as well. You can see their
Google Tech Talk at: http://www.youtube.com/watch?v=Qt3yPm7mKEc)

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Multiple sender identities (composing)

2011-05-16 Thread Mueen Nawaz
Jameson Graef Rollins
 writes:

> This is great, guys.  I've been wanting to get something like this
> working for a while now.

Agreed. Would be awesome if someone could post it to the Wiki...




Header and other questions

2011-05-16 Thread Mueen Nawaz
Jameson Graef Rollins
 writes:

> On Mon, 16 May 2011 11:27:52 -0400, Jesse Rosenthal  
> wrote:
>> The decision, if I remember correctly, was that regions are so intuitive
>> for emacs users, and many people reported trying to tag by region
>> without thinking about it, that this approach would make sense for the
>> emacs interface. In other words, keep the emacs interface emacsy. I
>> think I still agree with that, since I feel like having both would be
>> confusing (what do you do when you select a marked-line in a region?)
>> and I like the one I tend toward intuitively. But I certainly see the
>> benefit of the other functionality as well.
>
> I should mention that in my experience it's usually not that hard to
> construct searches (or filters on searches) that pair down to just the
> messages you want to tag, in which case you can just tag the entire
> buffer.

I guess so. I still haven't really switched to notmuch for real mail so
I have little experience. I don't quite remember why I would have found
it handy when I wrote it, but it was while I was busy experimenting with
tagging my existing archive. 



Re: Header and other questions

2011-05-16 Thread Mueen Nawaz
Jameson Graef Rollins
jroll...@finestructure.net writes:

 On Mon, 16 May 2011 11:27:52 -0400, Jesse Rosenthal jrosent...@jhu.edu 
 wrote:
 The decision, if I remember correctly, was that regions are so intuitive
 for emacs users, and many people reported trying to tag by region
 without thinking about it, that this approach would make sense for the
 emacs interface. In other words, keep the emacs interface emacsy. I
 think I still agree with that, since I feel like having both would be
 confusing (what do you do when you select a marked-line in a region?)
 and I like the one I tend toward intuitively. But I certainly see the
 benefit of the other functionality as well.

 I should mention that in my experience it's usually not that hard to
 construct searches (or filters on searches) that pair down to just the
 messages you want to tag, in which case you can just tag the entire
 buffer.

I guess so. I still haven't really switched to notmuch for real mail so
I have little experience. I don't quite remember why I would have found
it handy when I wrote it, but it was while I was busy experimenting with
tagging my existing archive. 

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: Multiple sender identities (composing)

2011-05-16 Thread Mueen Nawaz
Jameson Graef Rollins
jroll...@finestructure.net writes:

 This is great, guys.  I've been wanting to get something like this
 working for a while now.

Agreed. Would be awesome if someone could post it to the Wiki...


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: Header and other questions

2011-05-15 Thread mueen
Daniel Kahn Gillmor
d...@fifthhorseman.net writes:

 On 05/15/2011 01:23 AM, mu...@nawaz.org wrote:
 1. How do I see *all* the headers using the emacs interface? It shows me
 only 4 headers.

 shift-v from within a notmuch-show buffer will show the entire source
 of the message, including all headers.

That did the trick - thanks!

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Not build threads option?

2011-04-01 Thread mueen

Is there a way in the notmuch emacs client not to build the threads for
each query? It can be a bit time consuming. Sometimes (often, actually),
I just want to read all mails from someone, and am not interested in the
whole threads. 

If the above can be implemented, it would then be nice to have a
keybinding that would build the thread, but only for the message being
viewed. 

I don't know if I'm explaining this well, but does this sound
reasonable?


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: Sorting and attachments

2011-03-28 Thread mueen
Jameson Graef Rollins
jroll...@finestructure.net writes:

 On Sat, 26 Mar 2011 21:53:49 -0700, mu...@nawaz.org wrote:
 1. Can notmuch return results sorted by size?

 As far as I know, there is currently no way to do that (but see below).

It would be really convenient. Would this be a challenging task?

 2. Can I tell notmuch to limit the items it returns (e.g. show me
 first ten messages matching ... )

 Are you looking for this in the emacs UI?  The following works for me on
 the command line:

 notmuch search '*' | head -10

Well, yes - I was thinking of the Emacs UI. I suppose I could just hack
the .el files, but my knowledge of eLisp is fairly poor.

 3. Any plans for searching via attachment size or type? Say I want all
 emails with attachments bigger than 1 MB.

 You can currently search through attachments with the 'attachment:'
 search term.  See notmuch help search-terms for more info.  This works
 for searching for mime types.  I don't know of a way to search by size,
 though.

Again, that would be really convenient. I don't know if it would be
challenging. though.

 Essentially, I have a tendency to prune off large attachments so that
 my mail directory doesn't blow up...

 You might just try using find, which is very useful for this kind of
 thing:

 find $MAILDIR -printf '%b %p\n' | sort -n

Well, I would want to exclude certain tags in the query...

In my current mutt workflow, I'm a lot more aggressive. Once I'm done
looking at a (binary) attachment, I actively save or delete it. Then
every few months I just tell mutt to sort by size and delete any
attachments I may have missed. 

I don't expect the exact same workflow with notmuch, but I was trying to
find some way to handle this.


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Sorting and attachments

2011-03-26 Thread mueen
Hi,

1. Can notmuch return results sorted by size?

2. Can I tell notmuch to limit the items it returns (e.g. show me first
ten messages matching  ... )

3. Any plans for searching via attachment size or type? Say I want all
emails with attachments bigger than 1 MB. 

Essentially, I have a tendency to prune off large attachments so that my
mail directory doesn't blow up...

Thanks.



 

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Questions about importing mail (mbox)

2011-03-21 Thread Mueen Nawaz
Jesse Rosenthal  writes:

> I didn't need to convert when I started using notmuch, but for past
> mbox-to-maildir conversions, I always had the most confidence in using
> mutt interactively. Tag all messages (S-t, all), copy or save to a
> maildir, and make sure your mbox_type is set appropriately. There are
> scripts out there to automate it, but if you're worried about missing
> something, doing it by hand might work a bit better for you. (You can
> also do it in chunks by date to make sure everything is moving over.)
> Not the most efficient, but you should only have to do it once.

Thanks - will give it a try. It half solves my problem, in that I can do
a message count using mutt before and after to see the conversion went
well. The second issue is figuring out if notmuch really did index all
of them - challenging because I have plenty of dupes. I may just have to
take it all on faith for now.

As I had mentioned, when using going from MH to notmuch, it complained
for about 20 messages. I was in a hurry so didn't take a detailed look,
but two of them were clearly corrupt in my mbox file. They had a from
and virtually no other headers. So perhaps all the problems I'm having
stem from corrupt messages in my mbox...



Questions about importing mail (mbox)

2011-03-21 Thread Mueen Nawaz
Pieter Praet  writes:
> It would've been a no-brainer if you'd been using Maildir all along
> (mbox is evil incarnate), but...

Sure, but mbox is too convenient.

> I'd suggest keeping your original mbox file safe in git [1], and
> consistently commiting every step of the way, so even if messages were
> to get lost in translation, you still have a way to get them back, with
> negligible storage overhead (just remember to "git gc --aggressive
> --prune=now" when you're finished).

I think you misunderstood me. A part of me suspects this has something
to do with my not explaining myself, but who's to say?

I'm experimenting with notmuch, and if I can translate everything I
currently do in mutt to notmuch, then I'll just dump mutt. The set of
mboxes I have will remain archived, but for all future incoming email,
I'll switch to MH or MailDir. So I don't actually need to put my old
mboxes under revision control - I just need to save them somewhere.

> For the actual conversion to Maildir (and any type of mail fetching in
> general), I'd suggest using FDM [2], you'll never look back.

Thanks - will take a look.

> Regarding the significant discrepancy between processed and added files
> in Notmuch: Could be dupes (e.g. mail to/cc/bcc yourself or mailing
> lists, ending up in both Inbox and Sent), which are automatically
> suppressed by Notmuch.

It definitely was dupes. I didn't realize that notmuch did not keep
track of dupes. 

So I wrote a Python script to go through the mboxes and do a count of
only unique messages. Problem? I have over 1000 emails that don't have a
Message-ID header (case invariant search). I could go over why that is,
but suffice it to say that I hate Microsoft.

Once I remove all dupes, I get to within 300-400 of the count that
notmuch provides. The remaining 1000+ emails do contain some dupes, and
I can't find a convenient way to get an accurate count of unique emails
from them, but at least now I'm in the ballpark, and a lot more
confident.

Incidentally, one reason I didn't realize dupes were the reason is that
I did a search for a word in one email I had and notmuch did not find
it - so I assumed it had not been indexed. Later on, I realized I had
written a partial word and discovered that notmuch does find it if I
type the full word.

What am I doing wrong? Can't notmuch handle partial word matches? Do I
need to specify an option to get that to work?

Anyway, thanks for the help - I'll investigate further.




Re: Questions about importing mail (mbox)

2011-03-21 Thread Mueen Nawaz
Jesse Rosenthal jrosent...@jhu.edu writes:

 I didn't need to convert when I started using notmuch, but for past
 mbox-to-maildir conversions, I always had the most confidence in using
 mutt interactively. Tag all messages (S-t, all), copy or save to a
 maildir, and make sure your mbox_type is set appropriately. There are
 scripts out there to automate it, but if you're worried about missing
 something, doing it by hand might work a bit better for you. (You can
 also do it in chunks by date to make sure everything is moving over.)
 Not the most efficient, but you should only have to do it once.

Thanks - will give it a try. It half solves my problem, in that I can do
a message count using mutt before and after to see the conversion went
well. The second issue is figuring out if notmuch really did index all
of them - challenging because I have plenty of dupes. I may just have to
take it all on faith for now.

As I had mentioned, when using going from MH to notmuch, it complained
for about 20 messages. I was in a hurry so didn't take a detailed look,
but two of them were clearly corrupt in my mbox file. They had a from
and virtually no other headers. So perhaps all the problems I'm having
stem from corrupt messages in my mbox...

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Questions about importing mail (mbox)

2011-03-20 Thread Mueen Nawaz

Hi,

I'm trying to experiment with notmuch. 

As I understand it, notmuch does not handle mbox for input. The problem
is that all my mail is currently in mbox format.

So I first tried converting mbox to maildir using mb2md.

It didn't do a good job. When I subsequently tried importing to notmuch,
notmuch complained about lots of non-mail files - I confirmed that
indeed mb2md had botched converting those emails.

So then I tried to convert to mh format using Sylpheed. This seemed to
go well, but then when importing to notmuch, it complained again for
about 20 emails, and a manual check confirmed that some messages did not
get converted properly to mh (they don't show up in Sylpheed).

And then I noticed another discrepancy. mutt shows that I started with
44473 messages in mbox. When I imported into Sylpheed, it showed 44482
messages (no idea where the extra 9 came from). However, notmuch is
reporting that it processed 44482 files, but that it added 35602
messages.

Why only 35602 (it complained for only about 20 messages)? A search
confirmed that some messages that show up in both mutt (in mbox) and
Sylpheed (in mh format) were not indexed.

So I want to know: When you guys switched to notmuch, how did you ensure
you did not miss any emails. I really, really, really don't want to lose
any emails in this process!

Thanks.