segfault with xapian 1.3.1

[Jason A. Donenfeld]

As discussed on IRC, notmuch segfaults with 1.3.1 but not with 1.2.5 on
notmuch-new:

Program received signal SIGSEGV, Segmentation fault.
0x775a5367 in Xapian::TermIterator::skip_to(std::string const&) ()
from /usr/lib64/libxapian-1.3.so.1
(gdb) bt
#0  0x775a5367 in Xapian::TermIterator::skip_to(std::string const&)
() from /usr/lib64/libxapian-1.3.so.1
#1  0x77decaa5 in _notmuch_message_get_term(_notmuch_message*,
Xapian::TermIterator&, Xapian::TermIterator&, char const*) () from
/usr/lib64/libnotmuch.so.3
#2  0x77ded1fb in
_notmuch_message_ensure_metadata(_notmuch_message*) () from
/usr/lib64/libnotmuch.so.3
#3  0x77ded535 in notmuch_message_get_message_id () from
/usr/lib64/libnotmuch.so.3
#4  0x77de9c11 in notmuch_database_add_message () from
/usr/lib64/libnotmuch.so.3
#5  0x0040c610 in add_files ()
#6  0x0040cfbf in notmuch_new_command ()
#7  0x00407a1b in main ()

This occurs irrespective of the state of the database or the message being
added.
-- next part --
An HTML attachment was scrubbed...
URL: 

gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 15, 2012 at 11:41 AM, Patrick Totzke wrote:
>
> Well, thats not the point.. the script shouldn't die like this.
> I think it's be better if the script caught that exception, deleted the
> file
> and continued..


Probably, but I suspect it's related to whatever mystery error you ran into
before with the corruption.

Does deleting that file and trying again fix it?


Anyway, this is extremely stable for me and a few others at this point. I'm
going to wait for other users to report errors. Alternatively, send me
patches if you want things to happen.
-- next part --
An HTML attachment was scrubbed...
URL: 

Re: gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 15, 2012 at 11:41 AM, Patrick Totzke patricktot...@gmail.comwrote:

 Well, thats not the point.. the script shouldn't die like this.
 I think it's be better if the script caught that exception, deleted the
 file
 and continued..


Probably, but I suspect it's related to whatever mystery error you ran into
before with the corruption.

Does deleting that file and trying again fix it?


Anyway, this is extremely stable for me and a few others at this point. I'm
going to wait for other users to report errors. Alternatively, send me
patches if you want things to happen.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

gmail importer script

[Jason A. Donenfeld]

On Thu, Dec 13, 2012 at 5:48 PM, Patrick Totzke wrote:

>
> notmuch.errors.FileNotEmailError


Delete the file it dies on and try again.
-- next part --
An HTML attachment was scrubbed...
URL: 

Re: gmail importer script

[Jason A. Donenfeld]

On Thu, Dec 13, 2012 at 5:48 PM, Patrick Totzke patricktot...@gmail.comwrote:


 notmuch.errors.FileNotEmailError


Delete the file it dies on and try again.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Sat, Dec 15, 2012 at 7:18 AM, Austin Clements amdra...@mit.edu wrote:

 need to at least os.fsync before the os.link.


Fixed, thanks for the suggestion.
http://git.zx2c4.com/gmail-notmuch/commit/?id=3f9646058bfd91d7d0e2eda035521f97de92eabc
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

gmail importer script

[Jason A. Donenfeld]

\AllMail is the unlocalized generic version that maps to the localized
version, as are all the other back slash situations.

gmail importer script

[Jason A. Donenfeld]

On Thu, Dec 13, 2012 at 3:35 PM, fREW Schmidt  wrote:
> I just mean a general programming technique.  As you get information
> about tags you hold them in memory and then sync it later and clear
> your memory.

Oh, yea, that's a great idea. I'll play with that. Good thinking.

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Wed, Dec 12, 2012 at 10:26 PM, David Bremner  wrote:
>
> One thing that Olly Betts mentioned is that there is a recently fixed
> bug in xapian with respect to multiple threads in the same process

The script's actually single threaded.

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements  wrote:
> There should be no way to corrupt the database at this level through
> the Xapian API, which means nothing libnotmuch can do (much less users
> of libnotmuch) should be able to corrupt the database.  If you can
> reproduce the problem, it's probably a serious bug in Xapian, but it
> could also have been a file system bug or even random file system
> corruption.

Well that's... troubling.

Patrick: could you please backup and try to reproduce? Otherwise I'll
assume this was a one-off situation.


Austin-- think you could do a quick review of the script to double
check and confirm I'm not doing anything nefarious?
http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

gmail importer script

[Jason A. Donenfeld]

Please don't reply offlist. This is message #2 that's been sent offlist.

On Thu, Dec 13, 2012 at 3:22 PM, fREW Schmidt 
> One thing you might want to try is to only open the notmuch index for
> writing at a specified sync time.

How do I do this? What's the python function? I couldn't find
anything. Would be interested in doing this though if it exists.

> Also I was going to say you might want to catch SIGINT like

The library already does this.

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Wed, Dec 12, 2012 at 10:26 PM, David Bremner da...@tethera.net wrote:

 One thing that Olly Betts mentioned is that there is a recently fixed
 bug in xapian with respect to multiple threads in the same process

The script's actually single threaded.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Thu, Dec 13, 2012 at 3:35 PM, fREW Schmidt fri...@gmail.com wrote:
 I just mean a general programming technique.  As you get information
 about tags you hold them in memory and then sync it later and clear
 your memory.

Oh, yea, that's a great idea. I'll play with that. Good thinking.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

\AllMail is the unlocalized generic version that maps to the localized
version, as are all the other back slash situations.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke  
wrote:
> OK, i assumed i'd not have to download all messages because i already have 
> them
> on disk -- synced via offlineimap. but then i remembered that OI stores uses
> its own naming scheme for the files in a maildir. so thats probably why your 
> script
> did not realize theyre already there. (maybe it should? it could look up the 
> MIDs from notmuch)

Gmail bandwidth restrictions make this unfeasible. Requesting mail
header IDs via IMAP will rate limit the listing to 2 mails per second.
The best gmail will allow us to do at quasi rapid speed (~180kB/s) is
ask for the IMAP sequence number and the unique gmail 64bit number.
So, the maildir scheme winds up being "${gmail-unique-number}.gmail",
and the script doesn't redownload based on the preexistence of a file
with the corresponding name.

> i am kind of reluctant to reproduce this again (on my work-desktop).

It'd really be a huge help if you could backup and try...

> All i did was start your script, writing to the index in another terminal in 
> between (using alot)
> possibly my cronjob fired a "notmuch new; afew.." in between also.
> when I killed your script the index war broken.
>
> Can i turn on some global logging for notmuch to debug this?
> Maybe libnotmuch had trouble with so many db-writes at a time?

That is very wrong and bizarre. I open the database in read/write
mode, which is supposed to lock other programs from accessing it in
this way... Very strange.

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke patricktot...@gmail.com wrote:
 OK, i assumed i'd not have to download all messages because i already have 
 them
 on disk -- synced via offlineimap. but then i remembered that OI stores uses
 its own naming scheme for the files in a maildir. so thats probably why your 
 script
 did not realize theyre already there. (maybe it should? it could look up the 
 MIDs from notmuch)

Gmail bandwidth restrictions make this unfeasible. Requesting mail
header IDs via IMAP will rate limit the listing to 2 mails per second.
The best gmail will allow us to do at quasi rapid speed (~180kB/s) is
ask for the IMAP sequence number and the unique gmail 64bit number.
So, the maildir scheme winds up being ${gmail-unique-number}.gmail,
and the script doesn't redownload based on the preexistence of a file
with the corresponding name.

 i am kind of reluctant to reproduce this again (on my work-desktop).

It'd really be a huge help if you could backup and try...

 All i did was start your script, writing to the index in another terminal in 
 between (using alot)
 possibly my cronjob fired a notmuch new; afew.. in between also.
 when I killed your script the index war broken.

 Can i turn on some global logging for notmuch to debug this?
 Maybe libnotmuch had trouble with so many db-writes at a time?

That is very wrong and bizarre. I open the database in read/write
mode, which is supposed to lock other programs from accessing it in
this way... Very strange.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 11:04 AM, Patrick Totzke
 wrote:
> I ended up killing the process after i saw that ETA was >3h.

Gmail's bottleneck. Very unfortunate. Leave it going over night.

> This broke my index!
>
> afterwards, a `notmuch new` printed
>
> A Xapian exception occurred creating a directory: Expected block
> 485 to be level 1, not 0.
>
> I wasnt able to properly read the index anymore and had to
> restore an old index dump. not cool!


Can you provide any more information about this? The script properly
uses atomic transactions. Any corruption here is the result of a
notmuch library bug, which should be investigated.

gmail importer script

[Jason A. Donenfeld]

It seems to work, but i'm still waiting for the first run-through.

>
> Downloading messages: 457 of 22831|
>
> This takes ages. I hope it doesn't try to re-download all my messages
> everytime.
>

Nope. It's very smart about not doing that.


>
> Another thing:
> It seems that throughout its run, your script locks the notmuch database.
> This is particularly
> annoying if you want to read/tag mails while waiting for a long
> sync-process. I think you only
> really want to lock the index for a short time in the end.
> Do you write to the index directly after a msg is downloaded?
>

I tried to solve this by closing and opening the database each time, but it
actually reduced the performance considerably -- even slower than the gmail
download.
-- next part --
An HTML attachment was scrubbed...
URL: 

gmail importer script

[Jason A. Donenfeld]

On Mon, Dec 10, 2012 at 6:46 PM, Jason A. Donenfeld  wrote:

>
>
> On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke  gmail.com>wrote:
>>
>> It would, but its nicer not to load ressources you're not gonna use.
>> Also, progressbar 2.2 seems to be the newest version you see on a
>> reasonably new debian/ubuntu install.. 
>>
>
> I'll consider it.
>

--silent or -s now silences stdout.
-- next part --
An HTML attachment was scrubbed...
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20121211/3f915dd6/attachment.html>

Re: gmail importer script

[Jason A. Donenfeld]

It seems to work, but i'm still waiting for the first run-through.


 Downloading messages: 457 of 22831|

 This takes ages. I hope it doesn't try to re-download all my messages
 everytime.


Nope. It's very smart about not doing that.



 Another thing:
 It seems that throughout its run, your script locks the notmuch database.
 This is particularly
 annoying if you want to read/tag mails while waiting for a long
 sync-process. I think you only
 really want to lock the index for a short time in the end.
 Do you write to the index directly after a msg is downloaded?


I tried to solve this by closing and opening the database each time, but it
actually reduced the performance considerably -- even slower than the gmail
download.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 11:04 AM, Patrick Totzke
patricktot...@gmail.com wrote:
 I ended up killing the process after i saw that ETA was 3h.

Gmail's bottleneck. Very unfortunate. Leave it going over night.

 This broke my index!

 afterwards, a `notmuch new` printed

 A Xapian exception occurred creating a directory: Expected block
 485 to be level 1, not 0.

 I wasnt able to properly read the index anymore and had to
 restore an old index dump. not cool!


Can you provide any more information about this? The script properly
uses atomic transactions. Any corruption here is the result of a
notmuch library bug, which should be investigated.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

gmail importer script

[Jason A. Donenfeld]

On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke wrote:
>
> It would, but its nicer not to load ressources you're not gonna use. Also,
> progressbar 2.2 seems to be the newest version you see on a reasonably new
> debian/ubuntu install.. 
>

I'll consider it.


>
>
> Next exception.. do you have a bugtracker? (/me <3 github)
>

To hell with github. Sometimes I do use a bugtracker though. In light of me
being an idiot, maybe I should get one for this script. But this script is
so short, and these errors are really due to me being an idiot. Sit tight a
little longer and things will begin to work. Sorry for the prior impartial
fixes. Once this baby is running you will be quite delighted.


> TypeError: object of type 'NoneType' has no len()


Fixed.
http://git.zx2c4.com/gmail-notmuch/commit/?id=3bf8ce9698ec798a42bc2f92f596243fa8911bf8
-- next part --
An HTML attachment was scrubbed...
URL: 

gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 6:46 PM, Patrick Totzke wrote:
>
> ZeroDivisionError: float division by zero
>

Fixed it! Sorry about all this... Jeeze Louise.


>
> Another feature request:
> Could you make the progressbar stuff optinal? in the end one will want to
> run this via cronjob
> and this shouldn't print anything to stdout.
> Also, I can pretty much recommend the 'logging' module.
>

Sure, but... >/dev/null doesn't cut it? I can add some more refined logging
capabilities if you want.



>
> cheers,
> /p
>
-- next part --
An HTML attachment was scrubbed...
URL: 

Re: gmail importer script

[Jason A. Donenfeld]

On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke patricktot...@gmail.comwrote:

 It would, but its nicer not to load ressources you're not gonna use. Also,
 progressbar 2.2 seems to be the newest version you see on a reasonably new
 debian/ubuntu install.. /nitpick


I'll consider it.




 Next exception.. do you have a bugtracker? (/me 3 github)


To hell with github. Sometimes I do use a bugtracker though. In light of me
being an idiot, maybe I should get one for this script. But this script is
so short, and these errors are really due to me being an idiot. Sit tight a
little longer and things will begin to work. Sorry for the prior impartial
fixes. Once this baby is running you will be quite delighted.


 TypeError: object of type 'NoneType' has no len()


Fixed.
http://git.zx2c4.com/gmail-notmuch/commit/?id=3bf8ce9698ec798a42bc2f92f596243fa8911bf8
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Mon, Dec 10, 2012 at 6:46 PM, Jason A. Donenfeld ja...@zx2c4.com wrote:



 On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke 
 patricktot...@gmail.comwrote:

 It would, but its nicer not to load ressources you're not gonna use.
 Also, progressbar 2.2 seems to be the newest version you see on a
 reasonably new debian/ubuntu install.. /nitpick


 I'll consider it.


--silent or -s now silences stdout.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 6:46 PM, Patrick Totzke patricktot...@gmail.comwrote:

 ZeroDivisionError: float division by zero


Fixed it! Sorry about all this... Jeeze Louise.



 Another feature request:
 Could you make the progressbar stuff optinal? in the end one will want to
 run this via cronjob
 and this shouldn't print anything to stdout.
 Also, I can pretty much recommend the 'logging' module.


Sure, but... /dev/null doesn't cut it? I can add some more refined logging
capabilities if you want.




 cheers,
 /p

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 6:04 PM, Jason A. Donenfeld  wrote:
>> BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to 
>> get that far.
>
> Really? Are you in the UK? I'll have to do this dynamically using the
> XLIST imap extension.

Okay, I've updated it to dynamically find the All Mail folder.

gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 5:20 PM, Patrick Totzke  
wrote:
> I have two new errors:
>
> -
> ./gmail-notmuch.py -u patricktotzke at gmail.com -p mypwd ~/mail/gmail/
>
> I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
> as this is where my all-mail maildir is, and i also tried an absolute path, 
> all the same result.
>
> If I point it to the root of my notmuch directory I get:

Pointing it at the root directory is what you want. In fact, I'm
removing the argument from the script and having it just get that from
the config. So the correct invocation will be:

./gmail-notmuch.py -u blabla -p password

>   File "./gmail-notmuch.py", line 147, in create_progressbar
> return ProgressBar(maxval=total, widgets=[text + ": ", SimpleProgress(), 
> Bar(), Percentage(), " ", ETA(), " ", FileTransferSpeed(unit="emails")])
> TypeError: __init__() got an unexpected keyword argument 'unit'

You're probably running with an old version of python-statusbar.

>
> BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to get 
> that far.

Really? Are you in the UK? I'll have to do this dynamically using the
XLIST imap extension.

Tags with spaces

[Jason A. Donenfeld]

Very helpful! Thanks folks for the explanations.

Another thing to note is that the dump format doesn't add any quotes
to the tags, so this is something of an issue I suppose.

The Gmail Importer Script: Complete

[Jason A. Donenfeld]

Hi everyone,

My script to import messages from gmail into notmuch, including tag
information, is now polished and feature complete.


** Watch a video of it here:
** https://www.youtube.com/watch?v=e-8EHIAr7wA


Read the source here:
http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

Download the plaintext here:
http://git.zx2c4.com/gmail-notmuch/plain/gmail-notmuch.py

View the requirements here:
http://git.zx2c4.com/gmail-notmuch/tree/requirements.txt

Enjoy,
Jason

gmail importer script

[Jason A. Donenfeld]

> I actually prefer this approach, but I think it would be more useful to
> leave the syncing of the emails to a different program, and then just
> managing the labels / tags with your tool (which is notmuch territory). So
> the workflow would be:
> So, implementing the mail sync in this script would, as I see it, kind of
> reinventing the wheel.

You're misunderstanding me. Let me make it very clear what the script
does. Actually why don't you just read the source code? Please? Well,
anyway, here:

- It looks in gmail and asks it which messages gmail has
- It downloads those messages
- It applies gmail's labels to those messages as tags

The end.

Along the way it has some smart things to reduce redundant downloads.

For more information, consult the source code.

Tags with spaces

[Jason A. Donenfeld]

Do we support them? Are we going to?

This doesn't work:
notmuch search -tag:"This has spaces"

Nor does:
notmuch search "-tag:This has spaces"

Re: gmail importer script

[Jason A. Donenfeld]

 I actually prefer this approach, but I think it would be more useful to
 leave the syncing of the emails to a different program, and then just
 managing the labels / tags with your tool (which is notmuch territory). So
 the workflow would be:
 So, implementing the mail sync in this script would, as I see it, kind of
 reinventing the wheel.

You're misunderstanding me. Let me make it very clear what the script
does. Actually why don't you just read the source code? Please? Well,
anyway, here:

- It looks in gmail and asks it which messages gmail has
- It downloads those messages
- It applies gmail's labels to those messages as tags

The end.

Along the way it has some smart things to reduce redundant downloads.

For more information, consult the source code.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

The Gmail Importer Script: Complete

[Jason A. Donenfeld]

Hi everyone,

My script to import messages from gmail into notmuch, including tag
information, is now polished and feature complete.


** Watch a video of it here:
** https://www.youtube.com/watch?v=e-8EHIAr7wA


Read the source here:
http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

Download the plaintext here:
http://git.zx2c4.com/gmail-notmuch/plain/gmail-notmuch.py

View the requirements here:
http://git.zx2c4.com/gmail-notmuch/tree/requirements.txt

Enjoy,
Jason
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Tags with spaces

[Jason A. Donenfeld]

Very helpful! Thanks folks for the explanations.

Another thing to note is that the dump format doesn't add any quotes
to the tags, so this is something of an issue I suppose.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 5:20 PM, Patrick Totzke patricktot...@gmail.com wrote:
 I have two new errors:

 -
 ./gmail-notmuch.py -u patricktot...@gmail.com -p mypwd ~/mail/gmail/

 I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
 as this is where my all-mail maildir is, and i also tried an absolute path, 
 all the same result.

 If I point it to the root of my notmuch directory I get:

Pointing it at the root directory is what you want. In fact, I'm
removing the argument from the script and having it just get that from
the config. So the correct invocation will be:

./gmail-notmuch.py -u blabla -p password

   File ./gmail-notmuch.py, line 147, in create_progressbar
 return ProgressBar(maxval=total, widgets=[text + : , SimpleProgress(), 
 Bar(), Percentage(),  , ETA(),  , FileTransferSpeed(unit=emails)])
 TypeError: __init__() got an unexpected keyword argument 'unit'

You're probably running with an old version of python-statusbar.


 BTW: I had to change the hardcoded [Gmail] to [Google Mail] for it to get 
 that far.

Really? Are you in the UK? I'll have to do this dynamically using the
XLIST imap extension.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 6:04 PM, Jason A. Donenfeld ja...@zx2c4.com wrote:
 BTW: I had to change the hardcoded [Gmail] to [Google Mail] for it to 
 get that far.

 Really? Are you in the UK? I'll have to do this dynamically using the
 XLIST imap extension.

Okay, I've updated it to dynamically find the All Mail folder.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

gmail importer script

[Jason A. Donenfeld]

On Fri, Dec 7, 2012 at 2:57 PM, Rainer M Krug  wrote:
>
> >
> >
> > 2) I am downloading with the patched notmuch (to add x-keywords
> containing the labels) only
> > the "All Mail" folder - does your script use this information (if
> present) to tag the emails?
> >
> >
> > What is the patched notmuch? What does that do?
>
> Sorry - should have been "patched offlineimap" (
> https://github.com/aroig/offlineimap
> http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970and
> http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 )
>
> It is synching the gmail labels into the x-keywords header as a comma
> separated list. These could
> then be used by notmuch to be added to the labels. It also does sync
> changes from the X-Keywords
> to gmail (see
> http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970for
>  details)


I see.

I use the All Mail folder and get the tags using the X-GM-LABELS imap
extension. I don't muck with the mail headers in the process, fortunately.
-- next part --
An HTML attachment was scrubbed...
URL: 

gmail importer script

[Jason A. Donenfeld]

On Fri, Dec 7, 2012 at 2:48 PM, Jason A. Donenfeld  wrote:

>
>
>
>
>> 3) are label changes in gmail synched into the local emails?
>>
>
> It pulls labels from gmail messages into notmuch, with the labels in tact,
> once per each email.
>

Okay, it's way more robust now. Just updated it. Now it will pull gmail
labels to notmuch, each time, every time. So you can run this thing over
and over, and it will always push gmail into notmuch. It doesn't yet sync
the the other way yet, but at least this is now fairly robust.
-- next part --
An HTML attachment was scrubbed...
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20121207/32f19e81/attachment-0001.html>

gmail importer script

[Jason A. Donenfeld]

On Fri, Dec 7, 2012 at 10:23 AM, Patrick Totzke wrote:

> Is this supposed to happen?
>

Obviously not.



>
> -
> Logging in...
> Selecting all mail...
> Discovering local messages...
> Receiving message list...
> Traceback (most recent call last):
>   File "./gmail-notmuch.py", line 175, in 
> main()
>   File "./gmail-notmuch.py", line 58, in main
> new_messages = discover_new_messages(imap, old_messages)
>   File "./gmail-notmuch.py", line 80, in discover_new_messages
> typ, data = imap.fetch("1:*", "X-GM-MSGID")
>   File "/usr/lib/python2.7/imaplib.py", line 443, in fetch
> typ, dat = self._simple_command(name, message_set, message_parts)
>   File "/usr/lib/python2.7/imaplib.py", line 1070, in _simple_command
> return self._command_complete(name, self._command(name, *args))
>   File "/usr/lib/python2.7/imaplib.py", line 825, in _command
> ', '.join(Commands[name])))
> imaplib.error: command FETCH illegal in state AUTH, only allowed in states
> SELECTED
>
>
Not sure what is causing this. My best guess is that your password was
incorrect and that I'm not checking the login return value.



> --
>
>
> In [2]: imaplib.__version__
> Out[2]: '2.58'
>
> One thing you also might want to watch out for is
> exceptions due to a locked notmuch index.
>

How do I do this?



> rock on,
> /p
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
>
-- next part --
An HTML attachment was scrubbed...
URL: 

gmail importer script

[Jason A. Donenfeld]

On Fri, Dec 7, 2012 at 9:53 AM, Rainer M Krug  wrote:
>
> Sounds interesting.
>
> I don't know much python, so excuse my questions:
>
> 1) are multiple downloads of an email done, if the mail has multiple
> labels?
>

No.


> 2) I am downloading with the patched notmuch (to add x-keywords containing
> the labels) only the
> "All Mail" folder - does your script use this information (if present) to
> tag the emails?
>

What is the patched notmuch? What does that do?


> 3) are label changes in gmail synched into the local emails?
>

It pulls labels from gmail messages into notmuch, with the labels in tact,
once per each email.


> 4) I assume, local changes are not synced to gmail?
>

Correctamundo.



> 5) how are conflicts handled (local tag changes versus label changes on
> gmail)?
>

They aren't. See 3.


>
> Thanks,
>
> Rainer
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
>
> iEYEARECAAYFAlDBrqAACgkQoYgNqgF2egqrEwCeOSr5TGZgAIaRvzHCl1Nv+DBk
> ko4AnRT0FMBwYNRGzSclXxZXzaH0Q23x
> =Sx0B
> -END PGP SIGNATURE-
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
>
-- next part --
An HTML attachment was scrubbed...
URL: 

gmail importer script

[Jason A. Donenfeld]

Hi folks,

I wrote a script that imports emails and tags from gmail. It's
resumable and appears to be working reasonably well. I could use some
experienced eyes looking at it, as my first exposure to notmuch was
just a few hours ago.

http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

Thanks,
Jason

Re: gmail importer script

[Jason A. Donenfeld]

On Fri, Dec 7, 2012 at 9:53 AM, Rainer M Krug r.m.k...@gmail.com wrote:

 Sounds interesting.

 I don't know much python, so excuse my questions:

 1) are multiple downloads of an email done, if the mail has multiple
 labels?


No.


 2) I am downloading with the patched notmuch (to add x-keywords containing
 the labels) only the
 All Mail folder - does your script use this information (if present) to
 tag the emails?


What is the patched notmuch? What does that do?


 3) are label changes in gmail synched into the local emails?


It pulls labels from gmail messages into notmuch, with the labels in tact,
once per each email.


 4) I assume, local changes are not synced to gmail?


Correctamundo.



 5) how are conflicts handled (local tag changes versus label changes on
 gmail)?


They aren't. See 3.



 Thanks,

 Rainer
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

 iEYEARECAAYFAlDBrqAACgkQoYgNqgF2egqrEwCeOSr5TGZgAIaRvzHCl1Nv+DBk
 ko4AnRT0FMBwYNRGzSclXxZXzaH0Q23x
 =Sx0B
 -END PGP SIGNATURE-

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Fri, Dec 7, 2012 at 2:48 PM, Jason A. Donenfeld ja...@zx2c4.com wrote:





 3) are label changes in gmail synched into the local emails?


 It pulls labels from gmail messages into notmuch, with the labels in tact,
 once per each email.


Okay, it's way more robust now. Just updated it. Now it will pull gmail
labels to notmuch, each time, every time. So you can run this thing over
and over, and it will always push gmail into notmuch. It doesn't yet sync
the the other way yet, but at least this is now fairly robust.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Fri, Dec 7, 2012 at 2:57 PM, Rainer M Krug r.m.k...@gmail.com wrote:

 
 
  2) I am downloading with the patched notmuch (to add x-keywords
 containing the labels) only
  the All Mail folder - does your script use this information (if
 present) to tag the emails?
 
 
  What is the patched notmuch? What does that do?

 Sorry - should have been patched offlineimap (
 https://github.com/aroig/offlineimap
 http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970and
 http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 )

 It is synching the gmail labels into the x-keywords header as a comma
 separated list. These could
 then be used by notmuch to be added to the labels. It also does sync
 changes from the X-Keywords
 to gmail (see
 http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970for
  details)


I see.

I use the All Mail folder and get the tags using the X-GM-LABELS imap
extension. I don't muck with the mail headers in the process, fortunately.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Tags with spaces

[Jason A. Donenfeld]

Do we support them? Are we going to?

This doesn't work:
notmuch search -tag:This has spaces

Nor does:
notmuch search -tag:This has spaces
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

gmail importer script

[Jason A. Donenfeld]

Hi folks,

I wrote a script that imports emails and tags from gmail. It's
resumable and appears to be working reasonably well. I could use some
experienced eyes looking at it, as my first exposure to notmuch was
just a few hours ago.

http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

Thanks,
Jason
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch