Re: [PATCH 3/3] lib: Reject multi-message mboxes and deprecate single-message mbox
On Sun, Nov 25 2012, Austin Clements amdra...@mit.edu wrote: Previously, we would treat multi-message mboxes as one giant email, which, besides the obvious incorrect indexing, often led to out-of-memory errors for archival mboxes. Now we explicitly reject multi-message mboxes. For historical reasons, we retain support for single-message mboxes, but official deprecate this behavior. The series looks good to me -- but I don't know about deprecating single-message mboxes: * If we someday support (read-only?) mbox format, then single-message mboxes are normal again. * Some naïve mb2md scripts could leave the 'From ' -line intact: for example `formail -bz -s head -3 $MAIL`(*) can be used to demonstrate this * Some people may have large collection of single-file messages starting with 'From ' currently indexed. If those are to be re-indexed later without single-message mbox support that is somewhat of a burden to the users (**) (*) my mb2md wannabe does gnus-like $formail -bz -R 'From ' X-From-Line: ... (**) Something like the following could be used to mangle single-file mboxes... find . -type f | xargs perl -e 'foreach (@ARGV) { open IO, +, $_ or next; sysread IO, $buf, 5; if ($buf eq From ) { sysseek IO, 0, 0; syswrite IO, Fro:; }}' This breaks the multi-message mbox nicely... ;) Tomi ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 3/3] lib: Reject multi-message mboxes and deprecate single-message mbox
Quoth Tomi Ollila on Nov 25 at 3:26 pm: On Sun, Nov 25 2012, Austin Clements amdra...@mit.edu wrote: Previously, we would treat multi-message mboxes as one giant email, which, besides the obvious incorrect indexing, often led to out-of-memory errors for archival mboxes. Now we explicitly reject multi-message mboxes. For historical reasons, we retain support for single-message mboxes, but official deprecate this behavior. The series looks good to me -- but I don't know about deprecating single-message mboxes: * If we someday support (read-only?) mbox format, then single-message mboxes are normal again. If notmuch does gain mbox support, then its handling of single-message mboxes will *definitely* change because it will stop doing maildir-like things to them (flag sync, moving from new to cur, etc), which people may currently be depending on. This was one of the motivations for deprecating the current handling of single-message mboxes. * Some naïve mb2md scripts could leave the 'From ' -line intact: for example `formail -bz -s head -3 $MAIL`(*) can be used to demonstrate this I would call that buggy, rather than naïve. ]:--8) * Some people may have large collection of single-file messages starting with 'From ' currently indexed. If those are to be re-indexed later without single-message mbox support that is somewhat of a burden to the users (**) That's why this only deprecates them (with a warning) and doesn't drop support for them. The idea is to keep the historical handling for a few releases and then we'll have the flexibility to do what we want with single-message mboxes (including supporting them as real mbox). It's probably a good idea to include a script or a wiki pointer for fixing single-message mboxes in the NEWS. As long as the file name is kept the same, notmuch won't reindex it. (*) my mb2md wannabe does gnus-like $formail -bz -R 'From ' X-From-Line: ... (**) Something like the following could be used to mangle single-file mboxes... find . -type f | xargs perl -e 'foreach (@ARGV) { open IO, +, $_ or next; sysread IO, $buf, 5; if ($buf eq From ) { sysseek IO, 0, 0; syswrite IO, Fro:; }}' This breaks the multi-message mbox nicely... ;) Tomi ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 3/3] lib: Reject multi-message mboxes and deprecate single-message mbox
On Sun, Nov 25 2012, Austin Clements amdra...@mit.edu wrote: Quoth Tomi Ollila on Nov 25 at 3:26 pm: On Sun, Nov 25 2012, Austin Clements amdra...@mit.edu wrote: Previously, we would treat multi-message mboxes as one giant email, which, besides the obvious incorrect indexing, often led to out-of-memory errors for archival mboxes. Now we explicitly reject multi-message mboxes. For historical reasons, we retain support for single-message mboxes, but official deprecate this behavior. The series looks good to me -- but I don't know about deprecating single-message mboxes: * If we someday support (read-only?) mbox format, then single-message mboxes are normal again. If notmuch does gain mbox support, then its handling of single-message mboxes will *definitely* change because it will stop doing maildir-like things to them (flag sync, moving from new to cur, etc), which people may currently be depending on. This was one of the motivations for deprecating the current handling of single-message mboxes. * Some naïve mb2md scripts could leave the 'From ' -line intact: for example `formail -bz -s head -3 $MAIL`(*) can be used to demonstrate this I would call that buggy, rather than naïve. ]:--8) * Some people may have large collection of single-file messages starting with 'From ' currently indexed. If those are to be re-indexed later without single-message mbox support that is somewhat of a burden to the users (**) That's why this only deprecates them (with a warning) and doesn't drop support for them. The idea is to keep the historical handling for a few releases and then we'll have the flexibility to do what we want with single-message mboxes (including supporting them as real mbox). It's probably a good idea to include a script or a wiki pointer for fixing single-message mboxes in the NEWS. As long as the file name is kept the same, notmuch won't reindex it. Ok, I'm convinced. +1 Tomi (*) my mb2md wannabe does gnus-like $formail -bz -R 'From ' X-From-Line: ... (**) Something like the following could be used to mangle single-file mboxes... find . -type f | xargs perl -e 'foreach (@ARGV) { open IO, +, $_ or next; sysread IO, $buf, 5; if ($buf eq From ) { sysseek IO, 0, 0; syswrite IO, Fro:; }}' This breaks the multi-message mbox nicely... ;) Tomi ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch