gmail importer script

[Patrick Totzke]

BTW: I removed the whole maildir contents and restarted the downlowd over 
night. here's the result:

---
Traceback (most recent call last):2|
  | 26% ETA:  1:37:22   3.11  emails/s
  File "./gmail-notmuch.py", line 253, in 
main()
  File "./gmail-notmuch.py", line 89, in main
download_new_messages(imap, database, new_messages, destination)
  File "./gmail-notmuch.py", line 210, in download_new_messages
tag_message(database, dest, labels)
  File "./gmail-notmuch.py", line 168, in tag_message
raise e
notmuch.errors.FileNotEmailError
---

/p
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: signature
URL: 

Re: gmail importer script

[Patrick Totzke]

BTW: I removed the whole maildir contents and restarted the downlowd over 
night. here's the result:

---
Traceback (most recent call last):2|
  | 26% ETA:  1:37:22   3.11  emails/s
  File ./gmail-notmuch.py, line 253, in module
main()
  File ./gmail-notmuch.py, line 89, in main
download_new_messages(imap, database, new_messages, destination)
  File ./gmail-notmuch.py, line 210, in download_new_messages
tag_message(database, dest, labels)
  File ./gmail-notmuch.py, line 168, in tag_message
raise e
notmuch.errors.FileNotEmailError
---

/p


signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-16 20:44:04)
> On Sat, Dec 15, 2012 at 11:41 AM, Patrick Totzke 
> wrote:
> 
> Well, thats not the point.. the script shouldn't die like this.
> I think it's be better if the script caught that exception, deleted the
> file
> and continued..
> 
> 
> Probably, but I suspect it's related to whatever mystery error you ran into
> before with the corruption.
> 
> Does deleting that file and trying again fix it??

I don't know. How would I know which file to remove?
I just removed the newest files in new/cur/tmp
but this doesnt solve the issue.
What would really help here is some more informative error message!

I tried with the `-d` option but did not see any local path in the output.

> 
> Anyway, this is extremely stable for me and a few others at this point. I'm
> going to wait for other users to report errors. Alternatively, send me patches
> if you want things to happen.

Personally, I don't really care if 'things are happening', as I'm kind of OK 
with my current
offlineimap configuration and would seriously consider switching from OI
only if it meant an improvement in speed or stability.

I test this code because I believe that the notmuch ecosystem
would benefit from a working solution and that reviews and bug-reports
are important.

Of course, I don't expect any miracles in terms of speed at this early stage.
But the bug I reported above should be addressed at some point in order not to
scare off potential users.

Have you considered including your code in offlineimap?
I'm asking because OI already solved some of the problems you might face later 
on
if you want to continue working on this.

* multi-threaded downloads
* the ability to read passwords not as plaintext parameter..

best,
/p
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: signature
URL: 

gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 15, 2012 at 11:41 AM, Patrick Totzke wrote:
>
> Well, thats not the point.. the script shouldn't die like this.
> I think it's be better if the script caught that exception, deleted the
> file
> and continued..


Probably, but I suspect it's related to whatever mystery error you ran into
before with the corruption.

Does deleting that file and trying again fix it?


Anyway, this is extremely stable for me and a few others at this point. I'm
going to wait for other users to report errors. Alternatively, send me
patches if you want things to happen.
-- next part --
An HTML attachment was scrubbed...
URL: 

Re: gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 15, 2012 at 11:41 AM, Patrick Totzke patricktot...@gmail.comwrote:

 Well, thats not the point.. the script shouldn't die like this.
 I think it's be better if the script caught that exception, deleted the
 file
 and continued..


Probably, but I suspect it's related to whatever mystery error you ran into
before with the corruption.

Does deleting that file and trying again fix it?


Anyway, this is extremely stable for me and a few others at this point. I'm
going to wait for other users to report errors. Alternatively, send me
patches if you want things to happen.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-16 20:44:04)
 On Sat, Dec 15, 2012 at 11:41 AM, Patrick Totzke patricktot...@gmail.com
 wrote:
 
 Well, thats not the point.. the script shouldn't die like this.
 I think it's be better if the script caught that exception, deleted the
 file
 and continued..
 
 
 Probably, but I suspect it's related to whatever mystery error you ran into
 before with the corruption.
 
 Does deleting that file and trying again fix it? 

I don't know. How would I know which file to remove?
I just removed the newest files in new/cur/tmp
but this doesnt solve the issue.
What would really help here is some more informative error message!

I tried with the `-d` option but did not see any local path in the output.

 
 Anyway, this is extremely stable for me and a few others at this point. I'm
 going to wait for other users to report errors. Alternatively, send me patches
 if you want things to happen.

Personally, I don't really care if 'things are happening', as I'm kind of OK 
with my current
offlineimap configuration and would seriously consider switching from OI
only if it meant an improvement in speed or stability.

I test this code because I believe that the notmuch ecosystem
would benefit from a working solution and that reviews and bug-reports
are important.

Of course, I don't expect any miracles in terms of speed at this early stage.
But the bug I reported above should be addressed at some point in order not to
scare off potential users.

Have you considered including your code in offlineimap?
I'm asking because OI already solved some of the problems you might face later 
on
if you want to continue working on this.

* multi-threaded downloads
* the ability to read passwords not as plaintext parameter..

best,
/p


signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-15 08:22:46)
> On Thu, Dec 13, 2012 at 5:48 PM, Patrick Totzke 
> wrote:
> 
> 
> notmuch.errors.FileNotEmailError
> 
> 
> Delete the file it dies on and try again.

Well, thats not the point.. the script shouldn't die like this.
I think it's be better if the script caught that exception, deleted the file
and continued..

Thanks,
/p
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: signature
URL: 

gmail importer script

[Jason A. Donenfeld]

On Thu, Dec 13, 2012 at 5:48 PM, Patrick Totzke wrote:

>
> notmuch.errors.FileNotEmailError


Delete the file it dies on and try again.
-- next part --
An HTML attachment was scrubbed...
URL: 

notmuch python bindings corrupt db index (was: gmail importer script)

[Austin Clements]

Quoth Jason A. Donenfeld on Dec 13 at  3:32 pm:
> On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements  wrote:
> > There should be no way to corrupt the database at this level through
> > the Xapian API, which means nothing libnotmuch can do (much less users
> > of libnotmuch) should be able to corrupt the database.  If you can
> > reproduce the problem, it's probably a serious bug in Xapian, but it
> > could also have been a file system bug or even random file system
> > corruption.
> 
> Well that's... troubling.
> 
> Patrick: could you please backup and try to reproduce? Otherwise I'll
> assume this was a one-off situation.
> 
> 
> Austin-- think you could do a quick review of the script to double
> check and confirm I'm not doing anything nefarious?
> http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

In theory the only way you could cause corruption besides tickling a
bug would be to access the same database object concurrently from
different threads (since it's not thread-safe), but you don't appear
to be doing that.

I did spot something that could corrupt delivered email, though.  The
way you deliver to the Maildir is resilient to process termination,
but not to system failures such as power outages.  In particular, you
need to at least os.fsync before the os.link.  I'd recommend looking
at Python's mailbox module, which has a robust Maildir delivery
implementation (though it appears it doesn't let you control the file
name, so you probably can't use it directly).

Re: gmail importer script

[Jason A. Donenfeld]

On Thu, Dec 13, 2012 at 5:48 PM, Patrick Totzke patricktot...@gmail.comwrote:


 notmuch.errors.FileNotEmailError


Delete the file it dies on and try again.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Sat, Dec 15, 2012 at 7:18 AM, Austin Clements amdra...@mit.edu wrote:

 need to at least os.fsync before the os.link.


Fixed, thanks for the suggestion.
http://git.zx2c4.com/gmail-notmuch/commit/?id=3f9646058bfd91d7d0e2eda035521f97de92eabc
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-15 08:22:46)
 On Thu, Dec 13, 2012 at 5:48 PM, Patrick Totzke patricktot...@gmail.com
 wrote:
 
 
 notmuch.errors.FileNotEmailError
 
 
 Delete the file it dies on and try again.

Well, thats not the point.. the script shouldn't die like this.
I think it's be better if the script caught that exception, deleted the file
and continued..

Thanks,
/p


signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Austin Clements]

Quoth Jason A. Donenfeld on Dec 13 at  3:32 pm:
 On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements amdra...@mit.edu wrote:
  There should be no way to corrupt the database at this level through
  the Xapian API, which means nothing libnotmuch can do (much less users
  of libnotmuch) should be able to corrupt the database.  If you can
  reproduce the problem, it's probably a serious bug in Xapian, but it
  could also have been a file system bug or even random file system
  corruption.
 
 Well that's... troubling.
 
 Patrick: could you please backup and try to reproduce? Otherwise I'll
 assume this was a one-off situation.
 
 
 Austin-- think you could do a quick review of the script to double
 check and confirm I'm not doing anything nefarious?
 http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

In theory the only way you could cause corruption besides tickling a
bug would be to access the same database object concurrently from
different threads (since it's not thread-safe), but you don't appear
to be doing that.

I did spot something that could corrupt delivered email, though.  The
way you deliver to the Maildir is resilient to process termination,
but not to system failures such as power outages.  In particular, you
need to at least os.fsync before the os.link.  I'd recommend looking
at Python's mailbox module, which has a robust Maildir delivery
implementation (though it appears it doesn't let you control the file
name, so you probably can't use it directly).
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

gmail importer script

[Patrick Totzke]

Sorry, I'm misusing this thread as a bugtracker..


Traceback (most recent call last):4|   
| 10% ETA:  1:24:41   3.54  emails/s
  File "./gmail-notmuch.py", line 251, in 
main()
  File "./gmail-notmuch.py", line 89, in main
download_new_messages(imap, database, new_messages, destination)
  File "./gmail-notmuch.py", line 208, in download_new_messages
tag_message(database, dest, labels)
  File "./gmail-notmuch.py", line 168, in tag_message
raise e
notmuch.errors.FileNotEmailError



best,
/p

gmail importer script

[Jason A. Donenfeld]

\AllMail is the unlocalized generic version that maps to the localized
version, as are all the other back slash situations.

notmuch python bindings corrupt db index (was: gmail importer script)

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-13 14:32:53)
> On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements  wrote:
> > There should be no way to corrupt the database at this level through
> > the Xapian API, which means nothing libnotmuch can do (much less users
> > of libnotmuch) should be able to corrupt the database.  If you can
> > reproduce the problem, it's probably a serious bug in Xapian, but it
> > could also have been a file system bug or even random file system
> > corruption.
> 
> Well that's... troubling.
> 
> Patrick: could you please backup and try to reproduce? Otherwise I'll
> assume this was a one-off situation.


I tried it again, concurrently triggered some modifications
and killed the process afer a while.
(my modifications were not flushed during the run of your script as
the index was locked)
The index seems to be OK. So I'm afraid I cannot reproduce this reliably.
/p

gmail importer script

[Jason A. Donenfeld]

On Thu, Dec 13, 2012 at 3:35 PM, fREW Schmidt  wrote:
> I just mean a general programming technique.  As you get information
> about tags you hold them in memory and then sync it later and clear
> your memory.

Oh, yea, that's a great idea. I'll play with that. Good thinking.

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Wed, Dec 12, 2012 at 10:26 PM, David Bremner  wrote:
>
> One thing that Olly Betts mentioned is that there is a recently fixed
> bug in xapian with respect to multiple threads in the same process

The script's actually single threaded.

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements  wrote:
> There should be no way to corrupt the database at this level through
> the Xapian API, which means nothing libnotmuch can do (much less users
> of libnotmuch) should be able to corrupt the database.  If you can
> reproduce the problem, it's probably a serious bug in Xapian, but it
> could also have been a file system bug or even random file system
> corruption.

Well that's... troubling.

Patrick: could you please backup and try to reproduce? Otherwise I'll
assume this was a one-off situation.


Austin-- think you could do a quick review of the script to double
check and confirm I'm not doing anything nefarious?
http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

gmail importer script

[Jason A. Donenfeld]

Please don't reply offlist. This is message #2 that's been sent offlist.

On Thu, Dec 13, 2012 at 3:22 PM, fREW Schmidt 
> One thing you might want to try is to only open the notmuch index for
> writing at a specified sync time.

How do I do this? What's the python function? I couldn't find
anything. Would be interested in doing this though if it exists.

> Also I was going to say you might want to catch SIGINT like

The library already does this.

gmail importer script

[fREW Schmidt]

On Thu, Dec 13, 2012 at 03:28:30PM +0100, Jason A. Donenfeld wrote:
> On Thu, Dec 13, 2012 at 3:19 PM, fREW Schmidt  wrote:
> >> Okay, I've updated it to dynamically find the All Mail folder.
> >
> > I can't seem to find it now, but I read that Gmail actually localizes
> > the name of various folders, so I think All Mail will actually be
> > called something else for non-english speakers.  It might be a good
> > idea for you to at least make it a config option.
>
> A) Why are you replying off list?
>
> B) Did you totally miss my message directly above yours where I wrote
> "Okay, I've updated it to dynamically find the All Mail folder"?

Right, I did see that and I did look at the code; all I'm saying is
that Gmail may localize AllMail as I'm fairly sure they localize the
other builtin labels
(http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py#n214)

I could certainly be wrong though, I haven't looked at this at such a
low level before.

--
fREW Schmidt
http://blog.afoolishmanifesto.com
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: 

gmail importer script

[fREW Schmidt]

On Thu, Dec 13, 2012 at 03:30:19PM +0100, Jason A. Donenfeld wrote:
> Please don't reply offlist. This is message #2 that's been sent offlist.

Oh I'm sorry, that was an accident.

> On Thu, Dec 13, 2012 at 3:22 PM, fREW Schmidt 
> > One thing you might want to try is to only open the notmuch index for
> > writing at a specified sync time.
>
> How do I do this? What's the python function? I couldn't find
> anything. Would be interested in doing this though if it exists.

I just mean a general programming technique.  As you get information
about tags you hold them in memory and then sync it later and clear
your memory.

> > Also I was going to say you might want to catch SIGINT like
>
> The library already does this.

--
fREW Schmidt
http://blog.afoolishmanifesto.com
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: 

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Wed, Dec 12, 2012 at 10:26 PM, David Bremner da...@tethera.net wrote:

 One thing that Olly Betts mentioned is that there is a recently fixed
 bug in xapian with respect to multiple threads in the same process

The script's actually single threaded.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[fREW Schmidt]

On Thu, Dec 13, 2012 at 03:30:19PM +0100, Jason A. Donenfeld wrote:
 Please don't reply offlist. This is message #2 that's been sent offlist.

Oh I'm sorry, that was an accident.

 On Thu, Dec 13, 2012 at 3:22 PM, fREW Schmidt fri...@gmail.com
  One thing you might want to try is to only open the notmuch index for
  writing at a specified sync time.

 How do I do this? What's the python function? I couldn't find
 anything. Would be interested in doing this though if it exists.

I just mean a general programming technique.  As you get information
about tags you hold them in memory and then sync it later and clear
your memory.

  Also I was going to say you might want to catch SIGINT like

 The library already does this.

--
fREW Schmidt
http://blog.afoolishmanifesto.com


pgpYfiABySGYw.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Thu, Dec 13, 2012 at 3:35 PM, fREW Schmidt fri...@gmail.com wrote:
 I just mean a general programming technique.  As you get information
 about tags you hold them in memory and then sync it later and clear
 your memory.

Oh, yea, that's a great idea. I'll play with that. Good thinking.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[fREW Schmidt]

On Thu, Dec 13, 2012 at 03:28:30PM +0100, Jason A. Donenfeld wrote:
 On Thu, Dec 13, 2012 at 3:19 PM, fREW Schmidt fri...@gmail.com wrote:
  Okay, I've updated it to dynamically find the All Mail folder.
 
  I can't seem to find it now, but I read that Gmail actually localizes
  the name of various folders, so I think All Mail will actually be
  called something else for non-english speakers.  It might be a good
  idea for you to at least make it a config option.

 A) Why are you replying off list?

 B) Did you totally miss my message directly above yours where I wrote
 Okay, I've updated it to dynamically find the All Mail folder?

Right, I did see that and I did look at the code; all I'm saying is
that Gmail may localize AllMail as I'm fairly sure they localize the
other builtin labels
(http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py#n214)

I could certainly be wrong though, I haven't looked at this at such a
low level before.

--
fREW Schmidt
http://blog.afoolishmanifesto.com


pgpqDMiLkFVHO.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

\AllMail is the unlocalized generic version that maps to the localized
version, as are all the other back slash situations.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-13 14:32:53)
 On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements amdra...@mit.edu wrote:
  There should be no way to corrupt the database at this level through
  the Xapian API, which means nothing libnotmuch can do (much less users
  of libnotmuch) should be able to corrupt the database.  If you can
  reproduce the problem, it's probably a serious bug in Xapian, but it
  could also have been a file system bug or even random file system
  corruption.
 
 Well that's... troubling.
 
 Patrick: could you please backup and try to reproduce? Otherwise I'll
 assume this was a one-off situation.


I tried it again, concurrently triggered some modifications
and killed the process afer a while.
(my modifications were not flushed during the run of your script as
the index was locked)
The index seems to be OK. So I'm afraid I cannot reproduce this reliably.
/p


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Patrick Totzke]

Sorry, I'm misusing this thread as a bugtracker..


Traceback (most recent call last):4|   
| 10% ETA:  1:24:41   3.54  emails/s
  File ./gmail-notmuch.py, line 251, in module
main()
  File ./gmail-notmuch.py, line 89, in main
download_new_messages(imap, database, new_messages, destination)
  File ./gmail-notmuch.py, line 208, in download_new_messages
tag_message(database, dest, labels)
  File ./gmail-notmuch.py, line 168, in tag_message
raise e
notmuch.errors.FileNotEmailError



best,
/p
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke  
wrote:
> OK, i assumed i'd not have to download all messages because i already have 
> them
> on disk -- synced via offlineimap. but then i remembered that OI stores uses
> its own naming scheme for the files in a maildir. so thats probably why your 
> script
> did not realize theyre already there. (maybe it should? it could look up the 
> MIDs from notmuch)

Gmail bandwidth restrictions make this unfeasible. Requesting mail
header IDs via IMAP will rate limit the listing to 2 mails per second.
The best gmail will allow us to do at quasi rapid speed (~180kB/s) is
ask for the IMAP sequence number and the unique gmail 64bit number.
So, the maildir scheme winds up being "${gmail-unique-number}.gmail",
and the script doesn't redownload based on the preexistence of a file
with the corresponding name.

> i am kind of reluctant to reproduce this again (on my work-desktop).

It'd really be a huge help if you could backup and try...

> All i did was start your script, writing to the index in another terminal in 
> between (using alot)
> possibly my cronjob fired a "notmuch new; afew.." in between also.
> when I killed your script the index war broken.
>
> Can i turn on some global logging for notmuch to debug this?
> Maybe libnotmuch had trouble with so many db-writes at a time?

That is very wrong and bizarre. I open the database in read/write
mode, which is supposed to lock other programs from accessing it in
this way... Very strange.

notmuch python bindings corrupt db index (was: gmail importer script)

[David Bremner]

Austin Clements  writes:

>
> There should be no way to corrupt the database at this level through
> the Xapian API, which means nothing libnotmuch can do (much less users
> of libnotmuch) should be able to corrupt the database.  If you can
> reproduce the problem, it's probably a serious bug in Xapian, but it
> could also have been a file system bug or even random file system
> corruption.

One thing that Olly Betts mentioned is that there is a recently fixed
bug in xapian with respect to multiple threads in the same process

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695542

I don't _think_ this is relevant here, but I mention it for
completeness.

d

notmuch python bindings corrupt db index (was: gmail importer script)

[Austin Clements]

Quoth Jason A. Donenfeld on Dec 12 at  7:35 pm:
> On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke  
> wrote:
> > i am kind of reluctant to reproduce this again (on my work-desktop).
> 
> It'd really be a huge help if you could backup and try...
> 
> > All i did was start your script, writing to the index in another terminal 
> > in between (using alot)
> > possibly my cronjob fired a "notmuch new; afew.." in between also.
> > when I killed your script the index war broken.
> >
> > Can i turn on some global logging for notmuch to debug this?
> > Maybe libnotmuch had trouble with so many db-writes at a time?
> 
> That is very wrong and bizarre. I open the database in read/write
> mode, which is supposed to lock other programs from accessing it in
> this way... Very strange.

There should be no way to corrupt the database at this level through
the Xapian API, which means nothing libnotmuch can do (much less users
of libnotmuch) should be able to corrupt the database.  If you can
reproduce the problem, it's probably a serious bug in Xapian, but it
could also have been a file system bug or even random file system
corruption.

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke patricktot...@gmail.com wrote:
 OK, i assumed i'd not have to download all messages because i already have 
 them
 on disk -- synced via offlineimap. but then i remembered that OI stores uses
 its own naming scheme for the files in a maildir. so thats probably why your 
 script
 did not realize theyre already there. (maybe it should? it could look up the 
 MIDs from notmuch)

Gmail bandwidth restrictions make this unfeasible. Requesting mail
header IDs via IMAP will rate limit the listing to 2 mails per second.
The best gmail will allow us to do at quasi rapid speed (~180kB/s) is
ask for the IMAP sequence number and the unique gmail 64bit number.
So, the maildir scheme winds up being ${gmail-unique-number}.gmail,
and the script doesn't redownload based on the preexistence of a file
with the corresponding name.

 i am kind of reluctant to reproduce this again (on my work-desktop).

It'd really be a huge help if you could backup and try...

 All i did was start your script, writing to the index in another terminal in 
 between (using alot)
 possibly my cronjob fired a notmuch new; afew.. in between also.
 when I killed your script the index war broken.

 Can i turn on some global logging for notmuch to debug this?
 Maybe libnotmuch had trouble with so many db-writes at a time?

That is very wrong and bizarre. I open the database in read/write
mode, which is supposed to lock other programs from accessing it in
this way... Very strange.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Austin Clements]

Quoth Jason A. Donenfeld on Dec 12 at  7:35 pm:
 On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke patricktot...@gmail.com 
 wrote:
  i am kind of reluctant to reproduce this again (on my work-desktop).
 
 It'd really be a huge help if you could backup and try...
 
  All i did was start your script, writing to the index in another terminal 
  in between (using alot)
  possibly my cronjob fired a notmuch new; afew.. in between also.
  when I killed your script the index war broken.
 
  Can i turn on some global logging for notmuch to debug this?
  Maybe libnotmuch had trouble with so many db-writes at a time?
 
 That is very wrong and bizarre. I open the database in read/write
 mode, which is supposed to lock other programs from accessing it in
 this way... Very strange.

There should be no way to corrupt the database at this level through
the Xapian API, which means nothing libnotmuch can do (much less users
of libnotmuch) should be able to corrupt the database.  If you can
reproduce the problem, it's probably a serious bug in Xapian, but it
could also have been a file system bug or even random file system
corruption.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[David Bremner]

Austin Clements amdra...@mit.edu writes:


 There should be no way to corrupt the database at this level through
 the Xapian API, which means nothing libnotmuch can do (much less users
 of libnotmuch) should be able to corrupt the database.  If you can
 reproduce the problem, it's probably a serious bug in Xapian, but it
 could also have been a file system bug or even random file system
 corruption.

One thing that Olly Betts mentioned is that there is a recently fixed
bug in xapian with respect to multiple threads in the same process

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695542

I don't _think_ this is relevant here, but I mention it for
completeness.

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

notmuch python bindings corrupt db index (was: gmail importer script)

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-11 13:47:32)
> On Tue, Dec 11, 2012 at 11:04 AM, Patrick Totzke
>  wrote:
> > I ended up killing the process after i saw that ETA was >3h.
> 
> Gmail's bottleneck. Very unfortunate. Leave it going over night.

OK, i assumed i'd not have to download all messages because i already have them
on disk -- synced via offlineimap. but then i remembered that OI stores uses
its own naming scheme for the files in a maildir. so thats probably why your 
script
did not realize theyre already there. (maybe it should? it could look up the 
MIDs from notmuch)

> > This broke my index!
> >
> > afterwards, a `notmuch new` printed
> >
> > A Xapian exception occurred creating a directory: Expected block
> > 485 to be level 1, not 0.
> >
> > I wasnt able to properly read the index anymore and had to
> > restore an old index dump. not cool!
> 
> 
> Can you provide any more information about this? The script properly
> uses atomic transactions. Any corruption here is the result of a
> notmuch library bug, which should be investigated.

i am kind of reluctant to reproduce this again (on my work-desktop).
All i did was start your script, writing to the index in another terminal in 
between (using alot)
possibly my cronjob fired a "notmuch new; afew.." in between also.
when I killed your script the index war broken.

Can i turn on some global logging for notmuch to debug this?
Maybe libnotmuch had trouble with so many db-writes at a time?

/p


-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: signature
URL: 

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 11:04 AM, Patrick Totzke
 wrote:
> I ended up killing the process after i saw that ETA was >3h.

Gmail's bottleneck. Very unfortunate. Leave it going over night.

> This broke my index!
>
> afterwards, a `notmuch new` printed
>
> A Xapian exception occurred creating a directory: Expected block
> 485 to be level 1, not 0.
>
> I wasnt able to properly read the index anymore and had to
> restore an old index dump. not cool!


Can you provide any more information about this? The script properly
uses atomic transactions. Any corruption here is the result of a
notmuch library bug, which should be investigated.

gmail importer script

[Jason A. Donenfeld]

It seems to work, but i'm still waiting for the first run-through.

>
> Downloading messages: 457 of 22831|
>
> This takes ages. I hope it doesn't try to re-download all my messages
> everytime.
>

Nope. It's very smart about not doing that.


>
> Another thing:
> It seems that throughout its run, your script locks the notmuch database.
> This is particularly
> annoying if you want to read/tag mails while waiting for a long
> sync-process. I think you only
> really want to lock the index for a short time in the end.
> Do you write to the index directly after a msg is downloaded?
>

I tried to solve this by closing and opening the database each time, but it
actually reduced the performance considerably -- even slower than the gmail
download.
-- next part --
An HTML attachment was scrubbed...
URL: 

gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-11 09:12:56)
> It seems to work, but i'm still waiting for the first run-through.
> 
> 
> Downloading messages: 457 of 22831|
> 
> This takes ages. I hope it doesn't try to re-download all my messages
> everytime.
> 
> 
> Nope. It's very smart about not doing that.
> ?

hmmk

I ended up killing the process after i saw that ETA was >3h. This broke my 
index!
afterwards, a `notmuch new` printed 

A Xapian exception occurred creating a directory: Expected block
485 to be level 1, not 0.

I wasnt able to properly read the index anymore and had to
restore an old index dump. not cool!

/p

gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-11 07:06:22)
> On Mon, Dec 10, 2012 at 6:46 PM, Jason A. Donenfeld  
> wrote:
> 
> 
> 
> On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke  gmail.com>
> wrote:
> 
> It would, but its nicer not to load ressources you're not gonna use.
> Also, progressbar 2.2 seems to be the newest version you see on a
> reasonably new debian/ubuntu install.. 
> 
> 
> I'll consider it.
> 
> 
> --silent or -s now silences stdout.

It seems to work, but i'm still waiting for the first run-through.

Downloading messages: 457 of 22831|

This takes ages. I hope it doesn't try to re-download all my messages everytime.

Another thing:
It seems that throughout its run, your script locks the notmuch database. This 
is particularly
annoying if you want to read/tag mails while waiting for a long sync-process. I 
think you only
really want to lock the index for a short time in the end.
Do you write to the index directly after a msg is downloaded?

/p

gmail importer script

[Jason A. Donenfeld]

On Mon, Dec 10, 2012 at 6:46 PM, Jason A. Donenfeld  wrote:

>
>
> On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke  gmail.com>wrote:
>>
>> It would, but its nicer not to load ressources you're not gonna use.
>> Also, progressbar 2.2 seems to be the newest version you see on a
>> reasonably new debian/ubuntu install.. 
>>
>
> I'll consider it.
>

--silent or -s now silences stdout.
-- next part --
An HTML attachment was scrubbed...
URL: 

Re: gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-11 07:06:22)
 On Mon, Dec 10, 2012 at 6:46 PM, Jason A. Donenfeld ja...@zx2c4.com wrote:
 
 
 
 On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke patricktot...@gmail.com
 wrote:
 
 It would, but its nicer not to load ressources you're not gonna use.
 Also, progressbar 2.2 seems to be the newest version you see on a
 reasonably new debian/ubuntu install.. /nitpick
 
 
 I'll consider it.
 
 
 --silent or -s now silences stdout.

It seems to work, but i'm still waiting for the first run-through.

Downloading messages: 457 of 22831|

This takes ages. I hope it doesn't try to re-download all my messages everytime.

Another thing:
It seems that throughout its run, your script locks the notmuch database. This 
is particularly
annoying if you want to read/tag mails while waiting for a long sync-process. I 
think you only
really want to lock the index for a short time in the end.
Do you write to the index directly after a msg is downloaded?

/p
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

It seems to work, but i'm still waiting for the first run-through.


 Downloading messages: 457 of 22831|

 This takes ages. I hope it doesn't try to re-download all my messages
 everytime.


Nope. It's very smart about not doing that.



 Another thing:
 It seems that throughout its run, your script locks the notmuch database.
 This is particularly
 annoying if you want to read/tag mails while waiting for a long
 sync-process. I think you only
 really want to lock the index for a short time in the end.
 Do you write to the index directly after a msg is downloaded?


I tried to solve this by closing and opening the database each time, but it
actually reduced the performance considerably -- even slower than the gmail
download.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-11 09:12:56)
 It seems to work, but i'm still waiting for the first run-through.
 
 
 Downloading messages: 457 of 22831|
 
 This takes ages. I hope it doesn't try to re-download all my messages
 everytime.
 
 
 Nope. It's very smart about not doing that.
  

hmmk

I ended up killing the process after i saw that ETA was 3h. This broke my 
index!
afterwards, a `notmuch new` printed 

A Xapian exception occurred creating a directory: Expected block
485 to be level 1, not 0.

I wasnt able to properly read the index anymore and had to
restore an old index dump. not cool!

/p
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 11:04 AM, Patrick Totzke
patricktot...@gmail.com wrote:
 I ended up killing the process after i saw that ETA was 3h.

Gmail's bottleneck. Very unfortunate. Leave it going over night.

 This broke my index!

 afterwards, a `notmuch new` printed

 A Xapian exception occurred creating a directory: Expected block
 485 to be level 1, not 0.

 I wasnt able to properly read the index anymore and had to
 restore an old index dump. not cool!


Can you provide any more information about this? The script properly
uses atomic transactions. Any corruption here is the result of a
notmuch library bug, which should be investigated.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-11 13:47:32)
 On Tue, Dec 11, 2012 at 11:04 AM, Patrick Totzke
 patricktot...@gmail.com wrote:
  I ended up killing the process after i saw that ETA was 3h.
 
 Gmail's bottleneck. Very unfortunate. Leave it going over night.

OK, i assumed i'd not have to download all messages because i already have them
on disk -- synced via offlineimap. but then i remembered that OI stores uses
its own naming scheme for the files in a maildir. so thats probably why your 
script
did not realize theyre already there. (maybe it should? it could look up the 
MIDs from notmuch)

  This broke my index!
 
  afterwards, a `notmuch new` printed
 
  A Xapian exception occurred creating a directory: Expected block
  485 to be level 1, not 0.
 
  I wasnt able to properly read the index anymore and had to
  restore an old index dump. not cool!
 
 
 Can you provide any more information about this? The script properly
 uses atomic transactions. Any corruption here is the result of a
 notmuch library bug, which should be investigated.

i am kind of reluctant to reproduce this again (on my work-desktop).
All i did was start your script, writing to the index in another terminal in 
between (using alot)
possibly my cronjob fired a notmuch new; afew.. in between also.
when I killed your script the index war broken.

Can i turn on some global logging for notmuch to debug this?
Maybe libnotmuch had trouble with so many db-writes at a time?

/p




signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

gmail importer script

[Jason A. Donenfeld]

On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke wrote:
>
> It would, but its nicer not to load ressources you're not gonna use. Also,
> progressbar 2.2 seems to be the newest version you see on a reasonably new
> debian/ubuntu install.. 
>

I'll consider it.


>
>
> Next exception.. do you have a bugtracker? (/me <3 github)
>

To hell with github. Sometimes I do use a bugtracker though. In light of me
being an idiot, maybe I should get one for this script. But this script is
so short, and these errors are really due to me being an idiot. Sit tight a
little longer and things will begin to work. Sorry for the prior impartial
fixes. Once this baby is running you will be quite delighted.


> TypeError: object of type 'NoneType' has no len()


Fixed.
http://git.zx2c4.com/gmail-notmuch/commit/?id=3bf8ce9698ec798a42bc2f92f596243fa8911bf8
-- next part --
An HTML attachment was scrubbed...
URL: 

gmail importer script

[Rainer M Krug]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 08/12/12 14:15, Jason A. Donenfeld wrote:
>> I actually prefer this approach, but I think it would be more useful to 
>> leave the syncing of
>> the emails to a different program, and then just managing the labels / tags 
>> with your tool
>> (which is notmuch territory). So the workflow would be: So, implementing the 
>> mail sync in
>> this script would, as I see it, kind of reinventing the wheel.
> 
> You're misunderstanding me. Let me make it very clear what the script does. 
> Actually why don't
> you just read the source code? Please? Well,

Because I have never used python, and you know what your script is doing. I 
prefer to know what
the script is doing, before I try it out and it changes up my local emails and 
my notmuch
databases in unexpected ways, because I did not understand the source code 
properly?

> anyway, here:
> 
> - It looks in gmail and asks it which messages gmail has

OK.

> - It downloads those messages

Here is my question - see below.

> - It applies gmail's labels to those messages as tags

Perfect.

Now my final question: after reading your info which is useful, I looked in 
more detail at the
source code and it made now sense to me. My question is:

If I only want to aply the tags and *not download* emails, can I simply remove 
the line 88:

  download_new_messages(imap, database, new_messages, destination)

Is this correct? In this case, the script would ony tag y messages which are 
already downloaded?

If this is the case, I would immediately start using the script.

Rainer

> 
> The end.
> 
> Along the way it has some smart things to reduce redundant downloads.
> 
> For more information, consult the source code.
> 

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlDFp5kACgkQoYgNqgF2egqfVACfWckdCl3SOmue6EEHF1cv6d44
IKgAmwV9cQH/VxCmPHmmU6z3akQ9pFZV
=vZxS
-END PGP SIGNATURE-

gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-09 23:13:12)
> On Sat, Dec 8, 2012 at 6:46 PM, Patrick Totzke  
> wrote:
> 
> ZeroDivisionError: float division by zero
> 
> 
> Fixed it! Sorry about all this... Jeeze?Louise.?

thx

> 
> Another feature request:
> Could you make the progressbar stuff optinal? in the end one will want to
> run this via cronjob
> and this shouldn't print anything to stdout.
> Also, I can pretty much recommend the 'logging' module.
> 
> 
> Sure, but... >/dev/null doesn't cut it? I can add some more refined logging
> capabilities if you want.
It would, but its nicer not to load ressources you're not gonna use. Also, 
progressbar 2.2 seems to be the newest version you see on a reasonably new 
debian/ubuntu install.. 


Next exception.. do you have a bugtracker? (/me <3 github)

Logging in...
Selecting all mail...
Receiving message list: 24455 of 
24455||100% Time: 0:00:12   2.04 
kemails/s
Parsing message list and labels...
Searching for local messages...
Traceback (most recent call last):
  File "./gmail-notmuch.py", line 248, in 
main()
  File "./gmail-notmuch.py", line 75, in main
if len(new_messages) == 0:
TypeError: object of type 'NoneType' has no len()



/p

gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 6:46 PM, Patrick Totzke wrote:
>
> ZeroDivisionError: float division by zero
>

Fixed it! Sorry about all this... Jeeze Louise.


>
> Another feature request:
> Could you make the progressbar stuff optinal? in the end one will want to
> run this via cronjob
> and this shouldn't print anything to stdout.
> Also, I can pretty much recommend the 'logging' module.
>

Sure, but... >/dev/null doesn't cut it? I can add some more refined logging
capabilities if you want.



>
> cheers,
> /p
>
-- next part --
An HTML attachment was scrubbed...
URL: 

Re: gmail importer script

[Rainer M Krug]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 08/12/12 14:15, Jason A. Donenfeld wrote:
 I actually prefer this approach, but I think it would be more useful to 
 leave the syncing of
 the emails to a different program, and then just managing the labels / tags 
 with your tool
 (which is notmuch territory). So the workflow would be: So, implementing the 
 mail sync in
 this script would, as I see it, kind of reinventing the wheel.
 
 You're misunderstanding me. Let me make it very clear what the script does. 
 Actually why don't
 you just read the source code? Please? Well,

Because I have never used python, and you know what your script is doing. I 
prefer to know what
the script is doing, before I try it out and it changes up my local emails and 
my notmuch
databases in unexpected ways, because I did not understand the source code 
properly?

 anyway, here:
 
 - It looks in gmail and asks it which messages gmail has

OK.

 - It downloads those messages

Here is my question - see below.

 - It applies gmail's labels to those messages as tags

Perfect.

Now my final question: after reading your info which is useful, I looked in 
more detail at the
source code and it made now sense to me. My question is:

If I only want to aply the tags and *not download* emails, can I simply remove 
the line 88:

  download_new_messages(imap, database, new_messages, destination)

Is this correct? In this case, the script would ony tag y messages which are 
already downloaded?

If this is the case, I would immediately start using the script.

Rainer

 
 The end.
 
 Along the way it has some smart things to reduce redundant downloads.
 
 For more information, consult the source code.
 

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlDFp5kACgkQoYgNqgF2egqfVACfWckdCl3SOmue6EEHF1cv6d44
IKgAmwV9cQH/VxCmPHmmU6z3akQ9pFZV
=vZxS
-END PGP SIGNATURE-

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-09 23:13:12)
 On Sat, Dec 8, 2012 at 6:46 PM, Patrick Totzke patricktot...@gmail.com 
 wrote:
 
 ZeroDivisionError: float division by zero
 
 
 Fixed it! Sorry about all this... Jeeze Louise. 

thx

 
 Another feature request:
 Could you make the progressbar stuff optinal? in the end one will want to
 run this via cronjob
 and this shouldn't print anything to stdout.
 Also, I can pretty much recommend the 'logging' module.
 
 
 Sure, but... /dev/null doesn't cut it? I can add some more refined logging
 capabilities if you want.
It would, but its nicer not to load ressources you're not gonna use. Also, 
progressbar 2.2 seems to be the newest version you see on a reasonably new 
debian/ubuntu install.. /nitpick


Next exception.. do you have a bugtracker? (/me 3 github)

Logging in...
Selecting all mail...
Receiving message list: 24455 of 
24455||100% Time: 0:00:12   2.04 
kemails/s
Parsing message list and labels...
Searching for local messages...
Traceback (most recent call last):
  File ./gmail-notmuch.py, line 248, in module
main()
  File ./gmail-notmuch.py, line 75, in main
if len(new_messages) == 0:
TypeError: object of type 'NoneType' has no len()



/p
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke patricktot...@gmail.comwrote:

 It would, but its nicer not to load ressources you're not gonna use. Also,
 progressbar 2.2 seems to be the newest version you see on a reasonably new
 debian/ubuntu install.. /nitpick


I'll consider it.




 Next exception.. do you have a bugtracker? (/me 3 github)


To hell with github. Sometimes I do use a bugtracker though. In light of me
being an idiot, maybe I should get one for this script. But this script is
so short, and these errors are really due to me being an idiot. Sit tight a
little longer and things will begin to work. Sorry for the prior impartial
fixes. Once this baby is running you will be quite delighted.


 TypeError: object of type 'NoneType' has no len()


Fixed.
http://git.zx2c4.com/gmail-notmuch/commit/?id=3bf8ce9698ec798a42bc2f92f596243fa8911bf8
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Rainer M Krug]

On 7 December 2012 16:32, Jason A. Donenfeld ja...@zx2c4.com wrote:



 On Fri, Dec 7, 2012 at 2:57 PM, Rainer M Krug r.m.k...@gmail.com wrote:

  
 
  2) I am downloading with the patched notmuch (to add x-keywords
 containing the labels) only
  the All Mail folder - does your script use this information (if
 present) to tag the emails?
 
 
  What is the patched notmuch? What does that do?

 Sorry - should have been patched offlineimap (
 https://github.com/aroig/offlineimap

 http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970and
 http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 )

 It is synching the gmail labels into the x-keywords header as a comma
 separated list. These could
 then be used by notmuch to be added to the labels. It also does sync
 changes from the X-Keywords
 to gmail (see
 http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970for
  details)


 I see.

 I use the All Mail folder and get the tags using the X-GM-LABELS imap
 extension. I don't muck with the mail headers in the process, fortunately.


I actually prefer this approach, but I think it would be more useful to
leave the syncing of the emails to a different program, and then just
managing the labels / tags with your tool (which is notmuch territory). So
the workflow would be:

Initial or download (only remote changes):

1) Download email (All Mail folder) with whatever you are using
(offlinemail, ...)
2) call your script to index the downloaded emails (All Mail folder) with
notmuch and get the labels and set / remove them in notmuch accordingly.

Upload (only local changes):

1) upload emails using whatever (offlineimap, ...)
2) upload the changed tags and create the needed new labels (and possibly
delete empty ons?)

The tricky part will be the synching when changes on both sides were
observed.
The mail sync is implemented in e.g. offlineimap and works reilably. But I
don't know how one could handle tag / label changes on both sides.

So, implementing the mail sync in this script would, as I see it, kind of
reinventing the wheel. One can leave it in, but I would like to have an
option to leave the synching of the emails to a separate program.

Cheers,

Rainer



-- 
NEW GERMAN FAX NUMBER!!!

Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology,
UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa

Cell:   +27 - (0)83 9479 042
Fax:+27 - (0)86 516 2782
Fax:+49 - (0)321 2125 2244
email:  rai...@krugs.de

Skype:  RMkrug
Google: r.m.k...@gmail.com
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Mon, Dec 10, 2012 at 6:46 PM, Jason A. Donenfeld ja...@zx2c4.com wrote:



 On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke 
 patricktot...@gmail.comwrote:

 It would, but its nicer not to load ressources you're not gonna use.
 Also, progressbar 2.2 seems to be the newest version you see on a
 reasonably new debian/ubuntu install.. /nitpick


 I'll consider it.


--silent or -s now silences stdout.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 6:46 PM, Patrick Totzke patricktot...@gmail.comwrote:

 ZeroDivisionError: float division by zero


Fixed it! Sorry about all this... Jeeze Louise.



 Another feature request:
 Could you make the progressbar stuff optinal? in the end one will want to
 run this via cronjob
 and this shouldn't print anything to stdout.
 Also, I can pretty much recommend the 'logging' module.


Sure, but... /dev/null doesn't cut it? I can add some more refined logging
capabilities if you want.




 cheers,
 /p

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 6:04 PM, Jason A. Donenfeld  wrote:
>> BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to 
>> get that far.
>
> Really? Are you in the UK? I'll have to do this dynamically using the
> XLIST imap extension.

Okay, I've updated it to dynamically find the All Mail folder.

gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 5:20 PM, Patrick Totzke  
wrote:
> I have two new errors:
>
> -
> ./gmail-notmuch.py -u patricktotzke at gmail.com -p mypwd ~/mail/gmail/
>
> I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
> as this is where my all-mail maildir is, and i also tried an absolute path, 
> all the same result.
>
> If I point it to the root of my notmuch directory I get:

Pointing it at the root directory is what you want. In fact, I'm
removing the argument from the script and having it just get that from
the config. So the correct invocation will be:

./gmail-notmuch.py -u blabla -p password

>   File "./gmail-notmuch.py", line 147, in create_progressbar
> return ProgressBar(maxval=total, widgets=[text + ": ", SimpleProgress(), 
> Bar(), Percentage(), " ", ETA(), " ", FileTransferSpeed(unit="emails")])
> TypeError: __init__() got an unexpected keyword argument 'unit'

You're probably running with an old version of python-statusbar.

>
> BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to get 
> that far.

Really? Are you in the UK? I'll have to do this dynamically using the
XLIST imap extension.

gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-08 17:04:48)
> On Sat, Dec 8, 2012 at 5:20 PM, Patrick Totzke  
> wrote:
> > I have two new errors:
> >
> > -
> > ./gmail-notmuch.py -u patricktotzke at gmail.com -p mypwd ~/mail/gmail/
> >
> > I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
> > as this is where my all-mail maildir is, and i also tried an absolute path, 
> > all the same result.
> >
> > If I point it to the root of my notmuch directory I get:
> 
> Pointing it at the root directory is what you want. In fact, I'm
> removing the argument from the script and having it just get that from
> the config. So the correct invocation will be:
> 
> ./gmail-notmuch.py -u blabla -p password

Okay..
> 
> >   File "./gmail-notmuch.py", line 147, in create_progressbar
> > return ProgressBar(maxval=total, widgets=[text + ": ", 
> > SimpleProgress(), Bar(), Percentage(), " ", ETA(), " ", 
> > FileTransferSpeed(unit="emails")])
> > TypeError: __init__() got an unexpected keyword argument 'unit'
> 
> You're probably running with an old version of python-statusbar.

I just installed it from ubuntu 12.04's repository. the version was 2.2-2.
Installed from pip now, and it seems ok. you'll want to make more specific 
dependencies at some
point..

> >
> > BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to 
> > get that far.
> 
> Really? Are you in the UK? I'll have to do this dynamically using the
> XLIST imap extension.

Yea, but its a *really* old account I first got from google.de.
Reading this dynamically would def. help.


next try..
-

Traceback (most recent call last):
  File "./gmail-notmuch.py", line 246, in 
main()
  File "./gmail-notmuch.py", line 74, in main
new_messages = retag_old_messages(database, messages, destination)
  File "./gmail-notmuch.py", line 176, in retag_old_messages
progressbar.start()
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 
311, in start
self.update(0)
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 
283, in update
self.fd.write(self._format_line() + '\r')
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 
243, in _format_line
widgets = ''.join(self._format_widgets())
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 
223, in _format_widgets
widget = format_updatable(widget, self)
  File "/usr/local/lib/python2.7/dist-packages/progressbar/widgets.py", line 
38, in format_updatable
if hasattr(updatable, 'update'): return updatable.update(pbar)
  File "/usr/local/lib/python2.7/dist-packages/progressbar/widgets.py", line 
184, in update
return '%3d%%' % pbar.percentage()
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 
208, in percentage
return self.currval * 100.0 / self.maxval
ZeroDivisionError: float division by zero
--



Another feature request:
Could you make the progressbar stuff optinal? in the end one will want to run 
this via cronjob
and this shouldn't print anything to stdout.
Also, I can pretty much recommend the 'logging' module.

cheers,
/p
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: signature
URL: 

gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-07 13:49:46)
> Not sure what is causing this. My best guess is that your password was
> incorrect and that I'm not checking the login return value.

Yes, you're right, it was an incorrect passwd.

> One thing you also might want to watch out for is
> exceptions due to a locked notmuch index.
> 
> 
> How do I do this?

See e.g. here: https://github.com/pazz/alot/blob/master/alot/db/manager.py#L163
I'm sure "afew" (https://github.com/teythoon/afew) does a similar thing.


I have two new errors:

-
./gmail-notmuch.py -u patricktotzke at gmail.com -p mypwd ~/mail/gmail/
Logging in...
Selecting all mail...
Error opening database at /home/pazz/mail/gmail/.notmuch: No such file or 
directory
Traceback (most recent call last):
  File "./gmail-notmuch.py", line 225, in 
main()
  File "./gmail-notmuch.py", line 56, in main
database = notmuch.Database(destination, False, 
notmuch.Database.MODE.READ_WRITE)
  File "/home/pazz/.local/lib/python2.7/site-packages/notmuch/database.py", 
line 154, in __init__
self.open(path, mode)
  File "/home/pazz/.local/lib/python2.7/site-packages/notmuch/database.py", 
line 214, in open
raise NotmuchError(status)
notmuch.errors.FileError
-

I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
as this is where my all-mail maildir is, and i also tried an absolute path, all 
the same result.

If I point it to the root of my notmuch directory I get:
-
[~/projects/gmail-notmuch] ./gmail-notmuch.py -u patricktotzke at gmail.com -p 
mypwd /home/pazz/mail
Logging in...
Selecting all mail...
Traceback (most recent call last):
  File "./gmail-notmuch.py", line 225, in 
main()
  File "./gmail-notmuch.py", line 58, in main
messages = discover_messages(imap, total)
  File "./gmail-notmuch.py", line 104, in discover_messages
new_readline.progressbar = create_progressbar("Receiving message list", 
total).start()
  File "./gmail-notmuch.py", line 147, in create_progressbar
return ProgressBar(maxval=total, widgets=[text + ": ", SimpleProgress(), 
Bar(), Percentage(), " ", ETA(), " ", FileTransferSpeed(unit="emails")])
TypeError: __init__() got an unexpected keyword argument 'unit'

BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to get 
that far.


HTH,
/p
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: signature
URL: 

The Gmail Importer Script: Complete

[Jason A. Donenfeld]

Hi everyone,

My script to import messages from gmail into notmuch, including tag
information, is now polished and feature complete.


** Watch a video of it here:
** https://www.youtube.com/watch?v=e-8EHIAr7wA


Read the source here:
http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

Download the plaintext here:
http://git.zx2c4.com/gmail-notmuch/plain/gmail-notmuch.py

View the requirements here:
http://git.zx2c4.com/gmail-notmuch/tree/requirements.txt

Enjoy,
Jason

gmail importer script

[Jason A. Donenfeld]

> I actually prefer this approach, but I think it would be more useful to
> leave the syncing of the emails to a different program, and then just
> managing the labels / tags with your tool (which is notmuch territory). So
> the workflow would be:
> So, implementing the mail sync in this script would, as I see it, kind of
> reinventing the wheel.

You're misunderstanding me. Let me make it very clear what the script
does. Actually why don't you just read the source code? Please? Well,
anyway, here:

- It looks in gmail and asks it which messages gmail has
- It downloads those messages
- It applies gmail's labels to those messages as tags

The end.

Along the way it has some smart things to reduce redundant downloads.

For more information, consult the source code.

gmail importer script

[Rainer M Krug]

On 7 December 2012 16:32, Jason A. Donenfeld  wrote:

>
>
> On Fri, Dec 7, 2012 at 2:57 PM, Rainer M Krug  wrote:
>>
>>  >
>> >
>> > 2) I am downloading with the patched notmuch (to add x-keywords
>> containing the labels) only
>> > the "All Mail" folder - does your script use this information (if
>> present) to tag the emails?
>> >
>> >
>> > What is the patched notmuch? What does that do?
>>
>> Sorry - should have been "patched offlineimap" (
>> https://github.com/aroig/offlineimap
>>
>> http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970and
>> http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 )
>>
>> It is synching the gmail labels into the x-keywords header as a comma
>> separated list. These could
>> then be used by notmuch to be added to the labels. It also does sync
>> changes from the X-Keywords
>> to gmail (see
>> http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970for
>>  details)
>
>
> I see.
>
> I use the All Mail folder and get the tags using the X-GM-LABELS imap
> extension. I don't muck with the mail headers in the process, fortunately.
>

I actually prefer this approach, but I think it would be more useful to
leave the syncing of the emails to a different program, and then just
managing the labels / tags with your tool (which is notmuch territory). So
the workflow would be:

Initial or download (only remote changes):

1) Download email ("All Mail" folder) with whatever you are using
(offlinemail, ...)
2) call your script to index the downloaded emails (All Mail" folder) with
notmuch and get the labels and set / remove them in notmuch accordingly.

Upload (only local changes):

1) upload emails using whatever (offlineimap, ...)
2) upload the changed tags and create the needed new labels (and possibly
delete empty ons?)

The tricky part will be the synching when changes on both sides were
observed.
The mail sync is implemented in e.g. offlineimap and works reilably. But I
don't know how one could handle tag / label changes on both sides.

So, implementing the mail sync in this script would, as I see it, kind of
reinventing the wheel. One can leave it in, but I would like to have an
option to leave the synching of the emails to a separate program.

Cheers,

Rainer



-- 
NEW GERMAN FAX NUMBER!!!

Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology,
UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa

Cell:   +27 - (0)83 9479 042
Fax:+27 - (0)86 516 2782
Fax:+49 - (0)321 2125 2244
email:  Rainer at krugs.de

Skype:  RMkrug
Google: R.M.Krug at gmail.com
-- next part --
An HTML attachment was scrubbed...
URL: 

Re: gmail importer script

[Jason A. Donenfeld]

 I actually prefer this approach, but I think it would be more useful to
 leave the syncing of the emails to a different program, and then just
 managing the labels / tags with your tool (which is notmuch territory). So
 the workflow would be:
 So, implementing the mail sync in this script would, as I see it, kind of
 reinventing the wheel.

You're misunderstanding me. Let me make it very clear what the script
does. Actually why don't you just read the source code? Please? Well,
anyway, here:

- It looks in gmail and asks it which messages gmail has
- It downloads those messages
- It applies gmail's labels to those messages as tags

The end.

Along the way it has some smart things to reduce redundant downloads.

For more information, consult the source code.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

The Gmail Importer Script: Complete

[Jason A. Donenfeld]

Hi everyone,

My script to import messages from gmail into notmuch, including tag
information, is now polished and feature complete.


** Watch a video of it here:
** https://www.youtube.com/watch?v=e-8EHIAr7wA


Read the source here:
http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

Download the plaintext here:
http://git.zx2c4.com/gmail-notmuch/plain/gmail-notmuch.py

View the requirements here:
http://git.zx2c4.com/gmail-notmuch/tree/requirements.txt

Enjoy,
Jason
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-07 13:49:46)
 Not sure what is causing this. My best guess is that your password was
 incorrect and that I'm not checking the login return value.

Yes, you're right, it was an incorrect passwd.

 One thing you also might want to watch out for is
 exceptions due to a locked notmuch index.
 
 
 How do I do this?

See e.g. here: https://github.com/pazz/alot/blob/master/alot/db/manager.py#L163
I'm sure afew (https://github.com/teythoon/afew) does a similar thing.


I have two new errors:

-
./gmail-notmuch.py -u patricktot...@gmail.com -p mypwd ~/mail/gmail/
Logging in...
Selecting all mail...
Error opening database at /home/pazz/mail/gmail/.notmuch: No such file or 
directory
Traceback (most recent call last):
  File ./gmail-notmuch.py, line 225, in module
main()
  File ./gmail-notmuch.py, line 56, in main
database = notmuch.Database(destination, False, 
notmuch.Database.MODE.READ_WRITE)
  File /home/pazz/.local/lib/python2.7/site-packages/notmuch/database.py, 
line 154, in __init__
self.open(path, mode)
  File /home/pazz/.local/lib/python2.7/site-packages/notmuch/database.py, 
line 214, in open
raise NotmuchError(status)
notmuch.errors.FileError
-

I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
as this is where my all-mail maildir is, and i also tried an absolute path, all 
the same result.

If I point it to the root of my notmuch directory I get:
-
[~/projects/gmail-notmuch] ./gmail-notmuch.py -u patricktot...@gmail.com -p 
mypwd /home/pazz/mail
Logging in...
Selecting all mail...
Traceback (most recent call last):
  File ./gmail-notmuch.py, line 225, in module
main()
  File ./gmail-notmuch.py, line 58, in main
messages = discover_messages(imap, total)
  File ./gmail-notmuch.py, line 104, in discover_messages
new_readline.progressbar = create_progressbar(Receiving message list, 
total).start()
  File ./gmail-notmuch.py, line 147, in create_progressbar
return ProgressBar(maxval=total, widgets=[text + : , SimpleProgress(), 
Bar(), Percentage(),  , ETA(),  , FileTransferSpeed(unit=emails)])
TypeError: __init__() got an unexpected keyword argument 'unit'

BTW: I had to change the hardcoded [Gmail] to [Google Mail] for it to get 
that far.


HTH,
/p


signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 5:20 PM, Patrick Totzke patricktot...@gmail.com wrote:
 I have two new errors:

 -
 ./gmail-notmuch.py -u patricktot...@gmail.com -p mypwd ~/mail/gmail/

 I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
 as this is where my all-mail maildir is, and i also tried an absolute path, 
 all the same result.

 If I point it to the root of my notmuch directory I get:

Pointing it at the root directory is what you want. In fact, I'm
removing the argument from the script and having it just get that from
the config. So the correct invocation will be:

./gmail-notmuch.py -u blabla -p password

   File ./gmail-notmuch.py, line 147, in create_progressbar
 return ProgressBar(maxval=total, widgets=[text + : , SimpleProgress(), 
 Bar(), Percentage(),  , ETA(),  , FileTransferSpeed(unit=emails)])
 TypeError: __init__() got an unexpected keyword argument 'unit'

You're probably running with an old version of python-statusbar.


 BTW: I had to change the hardcoded [Gmail] to [Google Mail] for it to get 
 that far.

Really? Are you in the UK? I'll have to do this dynamically using the
XLIST imap extension.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 6:04 PM, Jason A. Donenfeld ja...@zx2c4.com wrote:
 BTW: I had to change the hardcoded [Gmail] to [Google Mail] for it to 
 get that far.

 Really? Are you in the UK? I'll have to do this dynamically using the
 XLIST imap extension.

Okay, I've updated it to dynamically find the All Mail folder.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-08 17:04:48)
 On Sat, Dec 8, 2012 at 5:20 PM, Patrick Totzke patricktot...@gmail.com 
 wrote:
  I have two new errors:
 
  -
  ./gmail-notmuch.py -u patricktot...@gmail.com -p mypwd ~/mail/gmail/
 
  I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
  as this is where my all-mail maildir is, and i also tried an absolute path, 
  all the same result.
 
  If I point it to the root of my notmuch directory I get:
 
 Pointing it at the root directory is what you want. In fact, I'm
 removing the argument from the script and having it just get that from
 the config. So the correct invocation will be:
 
 ./gmail-notmuch.py -u blabla -p password

Okay..
 
File ./gmail-notmuch.py, line 147, in create_progressbar
  return ProgressBar(maxval=total, widgets=[text + : , 
  SimpleProgress(), Bar(), Percentage(),  , ETA(),  , 
  FileTransferSpeed(unit=emails)])
  TypeError: __init__() got an unexpected keyword argument 'unit'
 
 You're probably running with an old version of python-statusbar.

I just installed it from ubuntu 12.04's repository. the version was 2.2-2.
Installed from pip now, and it seems ok. you'll want to make more specific 
dependencies at some
point..

 
  BTW: I had to change the hardcoded [Gmail] to [Google Mail] for it to 
  get that far.
 
 Really? Are you in the UK? I'll have to do this dynamically using the
 XLIST imap extension.

Yea, but its a *really* old account I first got from google.de.
Reading this dynamically would def. help.


next try..
-

Traceback (most recent call last):
  File ./gmail-notmuch.py, line 246, in module
main()
  File ./gmail-notmuch.py, line 74, in main
new_messages = retag_old_messages(database, messages, destination)
  File ./gmail-notmuch.py, line 176, in retag_old_messages
progressbar.start()
  File /usr/local/lib/python2.7/dist-packages/progressbar/__init__.py, line 
311, in start
self.update(0)
  File /usr/local/lib/python2.7/dist-packages/progressbar/__init__.py, line 
283, in update
self.fd.write(self._format_line() + '\r')
  File /usr/local/lib/python2.7/dist-packages/progressbar/__init__.py, line 
243, in _format_line
widgets = ''.join(self._format_widgets())
  File /usr/local/lib/python2.7/dist-packages/progressbar/__init__.py, line 
223, in _format_widgets
widget = format_updatable(widget, self)
  File /usr/local/lib/python2.7/dist-packages/progressbar/widgets.py, line 
38, in format_updatable
if hasattr(updatable, 'update'): return updatable.update(pbar)
  File /usr/local/lib/python2.7/dist-packages/progressbar/widgets.py, line 
184, in update
return '%3d%%' % pbar.percentage()
  File /usr/local/lib/python2.7/dist-packages/progressbar/__init__.py, line 
208, in percentage
return self.currval * 100.0 / self.maxval
ZeroDivisionError: float division by zero
--



Another feature request:
Could you make the progressbar stuff optinal? in the end one will want to run 
this via cronjob
and this shouldn't print anything to stdout.
Also, I can pretty much recommend the 'logging' module.

cheers,
/p


signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

gmail importer script

[Jason A. Donenfeld]

On Fri, Dec 7, 2012 at 2:57 PM, Rainer M Krug  wrote:
>
> >
> >
> > 2) I am downloading with the patched notmuch (to add x-keywords
> containing the labels) only
> > the "All Mail" folder - does your script use this information (if
> present) to tag the emails?
> >
> >
> > What is the patched notmuch? What does that do?
>
> Sorry - should have been "patched offlineimap" (
> https://github.com/aroig/offlineimap
> http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970and
> http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 )
>
> It is synching the gmail labels into the x-keywords header as a comma
> separated list. These could
> then be used by notmuch to be added to the labels. It also does sync
> changes from the X-Keywords
> to gmail (see
> http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970for
>  details)


I see.

I use the All Mail folder and get the tags using the X-GM-LABELS imap
extension. I don't muck with the mail headers in the process, fortunately.
-- next part --
An HTML attachment was scrubbed...
URL: 

gmail importer script

[Jason A. Donenfeld]

On Fri, Dec 7, 2012 at 2:48 PM, Jason A. Donenfeld  wrote:

>
>
>
>
>> 3) are label changes in gmail synched into the local emails?
>>
>
> It pulls labels from gmail messages into notmuch, with the labels in tact,
> once per each email.
>

Okay, it's way more robust now. Just updated it. Now it will pull gmail
labels to notmuch, each time, every time. So you can run this thing over
and over, and it will always push gmail into notmuch. It doesn't yet sync
the the other way yet, but at least this is now fairly robust.
-- next part --
An HTML attachment was scrubbed...
URL: 

gmail importer script

[Rainer M Krug]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 07/12/12 14:48, Jason A. Donenfeld wrote:
> 
> 
> On Fri, Dec 7, 2012 at 9:53 AM, Rainer M Krug  >
> wrote:
> 
> Sounds interesting.
> 
> I don't know much python, so excuse my questions:
> 
> 1) are multiple downloads of an email done, if the mail has multiple labels?
> 
> 
> No.

Good.

> 
> 
> 2) I am downloading with the patched notmuch (to add x-keywords containing 
> the labels) only
> the "All Mail" folder - does your script use this information (if present) to 
> tag the emails?
> 
> 
> What is the patched notmuch? What does that do?

Sorry - should have been "patched offlineimap" ( 
https://github.com/aroig/offlineimap
http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970 and
http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 )

It is synching the gmail labels into the x-keywords header as a comma separated 
list. These could
then be used by notmuch to be added to the labels. It also does sync changes 
from the X-Keywords
to gmail (see 
http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970 for 
details)

> 
> 
> 3) are label changes in gmail synched into the local emails?
> 
> 
> It pulls labels from gmail messages into notmuch, with the labels in tact, 
> once per each
> email.

Good.

> 
> 
> 4) I assume, local changes are not synced to gmail?
> 
> 
> Correctamundo.
> 

OK.

> 
> 
> 5) how are conflicts handled (local tag changes versus label changes on 
> gmail)?
> 
> 
> They aren't. See 3.
> 

Thanks,

Rainer


> 
> 
> Thanks,
> 
> Rainer -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) 
> Comment: Using GnuPG
> with Mozilla - http://www.enigmail.net/
> 
> iEYEARECAAYFAlDBrqAACgkQoYgNqgF2egqrEwCeOSr5TGZgAIaRvzHCl1Nv+DBk 
> ko4AnRT0FMBwYNRGzSclXxZXzaH0Q23x =Sx0B -END PGP SIGNATURE-
> 
> ___ notmuch mailing list notmuch 
> at notmuchmail.org
>  
> http://notmuchmail.org/mailman/listinfo/notmuch
> 
> 


- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, 
UCT), Dipl. Phys.
(Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :   +33 - (0)9 53 10 27 44
Cell:   +33 - (0)6 85 62 59 98
Fax :   +33 - (0)9 58 10 27 44

Fax (D):+49 - (0)3 21 21 25 22 44

email:  Rainer at krugs.de

Skype:  RMkrug
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlDB9cMACgkQoYgNqgF2egrEkACeLD9p4CUW3EhliFQ44cigpZdF
o8AAnA9M8tmw52W3m62QFJaFuoOLdto2
=6Zrk
-END PGP SIGNATURE-

gmail importer script

[Jason A. Donenfeld]

On Fri, Dec 7, 2012 at 10:23 AM, Patrick Totzke wrote:

> Is this supposed to happen?
>

Obviously not.



>
> -
> Logging in...
> Selecting all mail...
> Discovering local messages...
> Receiving message list...
> Traceback (most recent call last):
>   File "./gmail-notmuch.py", line 175, in 
> main()
>   File "./gmail-notmuch.py", line 58, in main
> new_messages = discover_new_messages(imap, old_messages)
>   File "./gmail-notmuch.py", line 80, in discover_new_messages
> typ, data = imap.fetch("1:*", "X-GM-MSGID")
>   File "/usr/lib/python2.7/imaplib.py", line 443, in fetch
> typ, dat = self._simple_command(name, message_set, message_parts)
>   File "/usr/lib/python2.7/imaplib.py", line 1070, in _simple_command
> return self._command_complete(name, self._command(name, *args))
>   File "/usr/lib/python2.7/imaplib.py", line 825, in _command
> ', '.join(Commands[name])))
> imaplib.error: command FETCH illegal in state AUTH, only allowed in states
> SELECTED
>
>
Not sure what is causing this. My best guess is that your password was
incorrect and that I'm not checking the login return value.



> --
>
>
> In [2]: imaplib.__version__
> Out[2]: '2.58'
>
> One thing you also might want to watch out for is
> exceptions due to a locked notmuch index.
>

How do I do this?



> rock on,
> /p
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
>
-- next part --
An HTML attachment was scrubbed...
URL: 

gmail importer script

[Jason A. Donenfeld]

On Fri, Dec 7, 2012 at 9:53 AM, Rainer M Krug  wrote:
>
> Sounds interesting.
>
> I don't know much python, so excuse my questions:
>
> 1) are multiple downloads of an email done, if the mail has multiple
> labels?
>

No.


> 2) I am downloading with the patched notmuch (to add x-keywords containing
> the labels) only the
> "All Mail" folder - does your script use this information (if present) to
> tag the emails?
>

What is the patched notmuch? What does that do?


> 3) are label changes in gmail synched into the local emails?
>

It pulls labels from gmail messages into notmuch, with the labels in tact,
once per each email.


> 4) I assume, local changes are not synced to gmail?
>

Correctamundo.



> 5) how are conflicts handled (local tag changes versus label changes on
> gmail)?
>

They aren't. See 3.


>
> Thanks,
>
> Rainer
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
>
> iEYEARECAAYFAlDBrqAACgkQoYgNqgF2egqrEwCeOSr5TGZgAIaRvzHCl1Nv+DBk
> ko4AnRT0FMBwYNRGzSclXxZXzaH0Q23x
> =Sx0B
> -END PGP SIGNATURE-
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
>
-- next part --
An HTML attachment was scrubbed...
URL: 

gmail importer script

[Rainer M Krug]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 07/12/12 05:05, Kushal Kumaran wrote:
> "Jason A. Donenfeld"  writes:
> 
>> Hi folks,
>> 
>> I wrote a script that imports emails and tags from gmail. It's resumable and 
>> appears to be
>> working reasonably well. I could use some experienced eyes looking at it, as 
>> my first
>> exposure to notmuch was just a few hours ago.
>> 
>> http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py
>> 
> 
> Not specific to notmuch, but not every gmail user has a @gmail.com or 
> @googlemail.com email
> address.  Google Apps users can have arbitrary email addresses, and still use 
> gmail's imap and
> web interface.
> 
> I finished syncing my gmail locally (using offlineimap), and creating tags by 
> hand not one day
> ago, or I would certainly have tried out this script.
> 

Sounds interesting.

I don't know much python, so excuse my questions:

1) are multiple downloads of an email done, if the mail has multiple labels?
2) I am downloading with the patched notmuch (to add x-keywords containing the 
labels) only the
"All Mail" folder - does your script use this information (if present) to tag 
the emails?
3) are label changes in gmail synched into the local emails?
4) I assume, local changes are not synced to gmail?
5) how are conflicts handled (local tag changes versus label changes on gmail)?

Thanks,

Rainer
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlDBrqAACgkQoYgNqgF2egqrEwCeOSr5TGZgAIaRvzHCl1Nv+DBk
ko4AnRT0FMBwYNRGzSclXxZXzaH0Q23x
=Sx0B
-END PGP SIGNATURE-

gmail importer script

[Kushal Kumaran]

"Jason A. Donenfeld"  writes:

> Hi folks,
>
> I wrote a script that imports emails and tags from gmail. It's
> resumable and appears to be working reasonably well. I could use some
> experienced eyes looking at it, as my first exposure to notmuch was
> just a few hours ago.
>
> http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py
>

Not specific to notmuch, but not every gmail user has a @gmail.com or
@googlemail.com email address.  Google Apps users can have arbitrary
email addresses, and still use gmail's imap and web interface.

I finished syncing my gmail locally (using offlineimap), and creating
tags by hand not one day ago, or I would certainly have tried out this
script.

-- 
regards,
kushal

gmail importer script

[Patrick Totzke]

Is this supposed to happen?

-
Logging in...
Selecting all mail...
Discovering local messages...
Receiving message list...
Traceback (most recent call last):
  File "./gmail-notmuch.py", line 175, in 
main()
  File "./gmail-notmuch.py", line 58, in main
new_messages = discover_new_messages(imap, old_messages)
  File "./gmail-notmuch.py", line 80, in discover_new_messages
typ, data = imap.fetch("1:*", "X-GM-MSGID")
  File "/usr/lib/python2.7/imaplib.py", line 443, in fetch
typ, dat = self._simple_command(name, message_set, message_parts)
  File "/usr/lib/python2.7/imaplib.py", line 1070, in _simple_command
return self._command_complete(name, self._command(name, *args))
  File "/usr/lib/python2.7/imaplib.py", line 825, in _command
', '.join(Commands[name])))
imaplib.error: command FETCH illegal in state AUTH, only allowed in states 
SELECTED

--


In [2]: imaplib.__version__
Out[2]: '2.58'

One thing you also might want to watch out for is
exceptions due to a locked notmuch index.
rock on,
/p

gmail importer script

[Jason A. Donenfeld]

Hi folks,

I wrote a script that imports emails and tags from gmail. It's
resumable and appears to be working reasonably well. I could use some
experienced eyes looking at it, as my first exposure to notmuch was
just a few hours ago.

http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

Thanks,
Jason

Re: gmail importer script

[Rainer M Krug]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 07/12/12 05:05, Kushal Kumaran wrote:
 Jason A. Donenfeld ja...@zx2c4.com writes:
 
 Hi folks,
 
 I wrote a script that imports emails and tags from gmail. It's resumable and 
 appears to be
 working reasonably well. I could use some experienced eyes looking at it, as 
 my first
 exposure to notmuch was just a few hours ago.
 
 http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py
 
 
 Not specific to notmuch, but not every gmail user has a @gmail.com or 
 @googlemail.com email
 address.  Google Apps users can have arbitrary email addresses, and still use 
 gmail's imap and
 web interface.
 
 I finished syncing my gmail locally (using offlineimap), and creating tags by 
 hand not one day
 ago, or I would certainly have tried out this script.
 

Sounds interesting.

I don't know much python, so excuse my questions:

1) are multiple downloads of an email done, if the mail has multiple labels?
2) I am downloading with the patched notmuch (to add x-keywords containing the 
labels) only the
All Mail folder - does your script use this information (if present) to tag 
the emails?
3) are label changes in gmail synched into the local emails?
4) I assume, local changes are not synced to gmail?
5) how are conflicts handled (local tag changes versus label changes on gmail)?

Thanks,

Rainer
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlDBrqAACgkQoYgNqgF2egqrEwCeOSr5TGZgAIaRvzHCl1Nv+DBk
ko4AnRT0FMBwYNRGzSclXxZXzaH0Q23x
=Sx0B
-END PGP SIGNATURE-

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Patrick Totzke]

Is this supposed to happen?

-
Logging in...
Selecting all mail...
Discovering local messages...
Receiving message list...
Traceback (most recent call last):
  File ./gmail-notmuch.py, line 175, in module
main()
  File ./gmail-notmuch.py, line 58, in main
new_messages = discover_new_messages(imap, old_messages)
  File ./gmail-notmuch.py, line 80, in discover_new_messages
typ, data = imap.fetch(1:*, X-GM-MSGID)
  File /usr/lib/python2.7/imaplib.py, line 443, in fetch
typ, dat = self._simple_command(name, message_set, message_parts)
  File /usr/lib/python2.7/imaplib.py, line 1070, in _simple_command
return self._command_complete(name, self._command(name, *args))
  File /usr/lib/python2.7/imaplib.py, line 825, in _command
', '.join(Commands[name])))
imaplib.error: command FETCH illegal in state AUTH, only allowed in states 
SELECTED

--


In [2]: imaplib.__version__
Out[2]: '2.58'

One thing you also might want to watch out for is
exceptions due to a locked notmuch index.
rock on,
/p
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Fri, Dec 7, 2012 at 9:53 AM, Rainer M Krug r.m.k...@gmail.com wrote:

 Sounds interesting.

 I don't know much python, so excuse my questions:

 1) are multiple downloads of an email done, if the mail has multiple
 labels?


No.


 2) I am downloading with the patched notmuch (to add x-keywords containing
 the labels) only the
 All Mail folder - does your script use this information (if present) to
 tag the emails?


What is the patched notmuch? What does that do?


 3) are label changes in gmail synched into the local emails?


It pulls labels from gmail messages into notmuch, with the labels in tact,
once per each email.


 4) I assume, local changes are not synced to gmail?


Correctamundo.



 5) how are conflicts handled (local tag changes versus label changes on
 gmail)?


They aren't. See 3.



 Thanks,

 Rainer
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

 iEYEARECAAYFAlDBrqAACgkQoYgNqgF2egqrEwCeOSr5TGZgAIaRvzHCl1Nv+DBk
 ko4AnRT0FMBwYNRGzSclXxZXzaH0Q23x
 =Sx0B
 -END PGP SIGNATURE-

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Rainer M Krug]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 07/12/12 14:48, Jason A. Donenfeld wrote:
 
 
 On Fri, Dec 7, 2012 at 9:53 AM, Rainer M Krug r.m.k...@gmail.com 
 mailto:r.m.k...@gmail.com
 wrote:
 
 Sounds interesting.
 
 I don't know much python, so excuse my questions:
 
 1) are multiple downloads of an email done, if the mail has multiple labels?
 
 
 No.

Good.

 
 
 2) I am downloading with the patched notmuch (to add x-keywords containing 
 the labels) only
 the All Mail folder - does your script use this information (if present) to 
 tag the emails?
 
 
 What is the patched notmuch? What does that do?

Sorry - should have been patched offlineimap ( 
https://github.com/aroig/offlineimap
http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970 and
http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 )

It is synching the gmail labels into the x-keywords header as a comma separated 
list. These could
then be used by notmuch to be added to the labels. It also does sync changes 
from the X-Keywords
to gmail (see 
http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970 for 
details)

 
 
 3) are label changes in gmail synched into the local emails?
 
 
 It pulls labels from gmail messages into notmuch, with the labels in tact, 
 once per each
 email.

Good.

 
 
 4) I assume, local changes are not synced to gmail?
 
 
 Correctamundo.
 

OK.

 
 
 5) how are conflicts handled (local tag changes versus label changes on 
 gmail)?
 
 
 They aren't. See 3.
 

Thanks,

Rainer


 
 
 Thanks,
 
 Rainer -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) 
 Comment: Using GnuPG
 with Mozilla - http://www.enigmail.net/
 
 iEYEARECAAYFAlDBrqAACgkQoYgNqgF2egqrEwCeOSr5TGZgAIaRvzHCl1Nv+DBk 
 ko4AnRT0FMBwYNRGzSclXxZXzaH0Q23x =Sx0B -END PGP SIGNATURE-
 
 ___ notmuch mailing list 
 notmuch@notmuchmail.org
 mailto:notmuch@notmuchmail.org 
 http://notmuchmail.org/mailman/listinfo/notmuch
 
 


- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, 
UCT), Dipl. Phys.
(Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :   +33 - (0)9 53 10 27 44
Cell:   +33 - (0)6 85 62 59 98
Fax :   +33 - (0)9 58 10 27 44

Fax (D):+49 - (0)3 21 21 25 22 44

email:  rai...@krugs.de

Skype:  RMkrug
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlDB9cMACgkQoYgNqgF2egrEkACeLD9p4CUW3EhliFQ44cigpZdF
o8AAnA9M8tmw52W3m62QFJaFuoOLdto2
=6Zrk
-END PGP SIGNATURE-
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Fri, Dec 7, 2012 at 2:48 PM, Jason A. Donenfeld ja...@zx2c4.com wrote:





 3) are label changes in gmail synched into the local emails?


 It pulls labels from gmail messages into notmuch, with the labels in tact,
 once per each email.


Okay, it's way more robust now. Just updated it. Now it will pull gmail
labels to notmuch, each time, every time. So you can run this thing over
and over, and it will always push gmail into notmuch. It doesn't yet sync
the the other way yet, but at least this is now fairly robust.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

On Fri, Dec 7, 2012 at 2:57 PM, Rainer M Krug r.m.k...@gmail.com wrote:

 
 
  2) I am downloading with the patched notmuch (to add x-keywords
 containing the labels) only
  the All Mail folder - does your script use this information (if
 present) to tag the emails?
 
 
  What is the patched notmuch? What does that do?

 Sorry - should have been patched offlineimap (
 https://github.com/aroig/offlineimap
 http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970and
 http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 )

 It is synching the gmail labels into the x-keywords header as a comma
 separated list. These could
 then be used by notmuch to be added to the labels. It also does sync
 changes from the X-Keywords
 to gmail (see
 http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970for
  details)


I see.

I use the All Mail folder and get the tags using the X-GM-LABELS imap
extension. I don't muck with the mail headers in the process, fortunately.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

gmail importer script

[Jason A. Donenfeld]

Hi folks,

I wrote a script that imports emails and tags from gmail. It's
resumable and appears to be working reasonably well. I could use some
experienced eyes looking at it, as my first exposure to notmuch was
just a few hours ago.

http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

Thanks,
Jason
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Kushal Kumaran]

Jason A. Donenfeld ja...@zx2c4.com writes:

 Hi folks,

 I wrote a script that imports emails and tags from gmail. It's
 resumable and appears to be working reasonably well. I could use some
 experienced eyes looking at it, as my first exposure to notmuch was
 just a few hours ago.

 http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py


Not specific to notmuch, but not every gmail user has a @gmail.com or
@googlemail.com email address.  Google Apps users can have arbitrary
email addresses, and still use gmail's imap and web interface.

I finished syncing my gmail locally (using offlineimap), and creating
tags by hand not one day ago, or I would certainly have tried out this
script.

-- 
regards,
kushal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch