[Mailman-Developers] Re: Using hyperkitty_import for archives downloaded from HK
On Mon, Sep 7, 2020, at 4:26 PM, Goran Terzic wrote: > Thanks Abhilash for your quick response > >> You can just take a database dump and migrate that. If you are using >> postgres, there is a pg_dump command that I know of. I am sure something >> exists for Mysql/mariadb too. I would recommend using this method for >> migrations. >> You can also use django's dumpdata[1] and loaddata commands to export and >> import data from one installation to other. It should work when moving from >> one database type to other in theory. I haven't tested it to comment on how >> successful it would be. > > Using pg_dump would work when migrating all lists from one server to another, > but can't be used for migration of a single list. > However, I guess it would be possible to write SQL script for that. > >> >> It turns out HK strips the angle brackets `< >` when internally storing the >> message-id and in-reply-to header. It adds them back to Message-ID when >> exporting but doesn't add it back to In-Reply-To. I have opened [1] for the >> issue. >> >> [1]: https://gitlab.com/mailman/hyperkitty/-/issues/311 >> >> Because of that, when the header is folded, an additional whitespace is >> added to the start resulting in broken threads. > > Thanks for confirming this issue. I also opened the fix [1] for the issue, you can apply it to the source server and export the archives that will import correctly. [1]: https://gitlab.com/mailman/hyperkitty/-/merge_requests/282 > > Regards, > Goran T -- thanks, Abhilash Raj (maxking) ___ Mailman-Developers mailing list -- mailman-developers@python.org To unsubscribe send an email to mailman-developers-le...@python.org https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3 Security Policy: https://wiki.list.org/x/QIA9
[Mailman-Developers] Re: Using hyperkitty_import for archives downloaded from HK
Thanks Abhilash for your quick response > You can just take a database dump and migrate that. If you are using > postgres, there is a pg_dump command that I know of. I am sure something > exists for Mysql/mariadb too. I would recommend using this method for > migrations. You can also use django's dumpdata[1] and loaddata commands to export and > import data from one installation to other. It should work when moving from > one database type to other in theory. I haven't tested it to comment on how > successful it would be. > Using pg_dump would work when migrating all lists from one server to another, but can't be used for migration of a single list. However, I guess it would be possible to write SQL script for that. > It turns out HK strips the angle brackets `< >` when internally storing > the message-id and in-reply-to header. It adds them back to Message-ID when > exporting but doesn't add it back to In-Reply-To. I have opened [1] for the > issue. > > [1]: https://gitlab.com/mailman/hyperkitty/-/issues/311 > > Because of that, when the header is folded, an additional whitespace is > added to the start resulting in broken threads. > > Thanks for confirming this issue. Regards, Goran T ___ Mailman-Developers mailing list -- mailman-developers@python.org To unsubscribe send an email to mailman-developers-le...@python.org https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3 Security Policy: https://wiki.list.org/x/QIA9
[Mailman-Developers] Re: Using hyperkitty_import for archives downloaded from HK
On Mon, Sep 7, 2020, at 2:25 PM, goran.ter...@gmail.com wrote: > I noticed that hyperkitty_import has issues with In-Reply-To header > when importing archives downloaded from Hyperkitty itself. > > In some cases in_reply_to field in hyperkitty_email table will have > extra white space at the start. > For example In-Reply-To: > cafv2ywofzrhedttz3f6emnwx8sh4_ar7+jmr7xvoad8_eoj...@mail.gmail.com > will be saved as ' CAFV2YWOfZRHEDTTz3F6Emnwx8Sh4_Ar7+JMR7XvOAD8_eOjqaQ'. It turns out HK strips the angle brackets `< >` when internally storing the message-id and in-reply-to header. It adds them back to Message-ID when exporting but doesn't add it back to In-Reply-To. I have opened [1] for the issue. [1]: https://gitlab.com/mailman/hyperkitty/-/issues/311 Because of that, when the header is folded, an additional whitespace is added to the start resulting in broken threads. > This results in broken threads, as parent of this message cannot be found. > > I also noticed that hyperkitty_import doesn't have this problem with > archives from MM2, but in MM2 archives, message id in in-reply-to > header is > surrounded with brackets, like this: In-Reply-To: > > > When I look at original email source, in-reply-to header doesn't have > brackets; it looks like in the archive made by HK. > > Is the issue here with import or export (download) process? > Is hyperkitty_import right tool for migrating lists from one MM3 server > to another? > > I was thinking should I post this to the mm3-users list or here, I hope > I made a right decision. > > Regards, > Goran T. > ___ > Mailman-Developers mailing list -- mailman-developers@python.org > To unsubscribe send an email to mailman-developers-le...@python.org > https://mail.python.org/mailman3/lists/mailman-developers.python.org/ > Mailman FAQ: https://wiki.list.org/x/AgA3 > > Security Policy: https://wiki.list.org/x/QIA9 > -- thanks, Abhilash Raj (maxking) ___ Mailman-Developers mailing list -- mailman-developers@python.org To unsubscribe send an email to mailman-developers-le...@python.org https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3 Security Policy: https://wiki.list.org/x/QIA9
[Mailman-Developers] Re: Using hyperkitty_import for archives downloaded from HK
On Mon, Sep 7, 2020, at 2:25 PM, goran.ter...@gmail.com wrote: > I noticed that hyperkitty_import has issues with In-Reply-To header > when importing archives downloaded from Hyperkitty itself. > > In some cases in_reply_to field in hyperkitty_email table will have > extra white space at the start. > For example In-Reply-To: > cafv2ywofzrhedttz3f6emnwx8sh4_ar7+jmr7xvoad8_eoj...@mail.gmail.com > will be saved as ' CAFV2YWOfZRHEDTTz3F6Emnwx8Sh4_Ar7+JMR7XvOAD8_eOjqaQ'. > > This results in broken threads, as parent of this message cannot be found. > > I also noticed that hyperkitty_import doesn't have this problem with > archives from MM2, but in MM2 archives, message id in in-reply-to > header is > surrounded with brackets, like this: In-Reply-To: > > > When I look at original email source, in-reply-to header doesn't have > brackets; it looks like in the archive made by HK. > > Is the issue here with import or export (download) process? > Is hyperkitty_import right tool for migrating lists from one MM3 server > to another? The import process should be fine, the issues are more likely with download/export process. Internally, Hyperkitty doesn't store the original email objects, only some of the headers and body of the message. It then generates the export archive on-demand by creating the Email object and stitching them together for mbox archive. You can just take a database dump and migrate that. If you are using postgres, there is a pg_dump command that I know of. I am sure something exists for Mysql/mariadb too. I would recommend using this method for migrations. You can also use django's dumpdata[1] and loaddata commands to export and import data from one installation to other. It should work when moving from one database type to other in theory. I haven't tested it to comment on how successful it would be. [1]: https://docs.djangoproject.com/en/3.0/ref/django-admin/#dumpdata > I was thinking should I post this to the mm3-users list or here, I hope > I made a right decision. > > Regards, > Goran T. > ___ > Mailman-Developers mailing list -- mailman-developers@python.org > To unsubscribe send an email to mailman-developers-le...@python.org > https://mail.python.org/mailman3/lists/mailman-developers.python.org/ > Mailman FAQ: https://wiki.list.org/x/AgA3 > > Security Policy: https://wiki.list.org/x/QIA9 > -- thanks, Abhilash Raj (maxking) ___ Mailman-Developers mailing list -- mailman-developers@python.org To unsubscribe send an email to mailman-developers-le...@python.org https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3 Security Policy: https://wiki.list.org/x/QIA9
