Re: [Mailman-Developers] Mailman Suite 3.0rc1 awaiting an upstream fix
On Mon, Apr 20, 2015 at 10:53:09AM -0400, Sumana Harihareswara wrote: At the PyCon sprint last week, we made a lot of progress towards a 3.0 release and we discussed next steps. I'm very glad so many of us could make it and pitch in. The 3.0rc1 depends on a new Falcon release because there's a bugfix awaiting release in Falcon that fixes a blocker bug for us: https://bugs.launchpad.net/postorius/+bug/1444504 Once that's out, we can put out a release candidate. In order to release 3.0.0 final, we need core native releases for all 5 components of the Mailman suite, with tested installation instructions and with all the components tagged, tested, and on PyPI. We'd also like some infrastructure and workflow improvements before we do that -- for one thing, we ought to set up continuous integration to run our automated tests on every merge. We will probably request a CI server/VM from the Python Software Foundation. We could actually set up CI for HyperKitty (and probably our other components as well) right away using the (no guarantee of uptime/service level) Jenkins installation at http://jenkins.cloud.fedoraproject.org/ if someone followed the instructions at https://fedoraproject.org/wiki/Jenkins@infra#Can_I_add_my_project_to_Jenkins.3F and add ticket to https://fedorahosted.org/fedora-infrastructure/ . We need to have an i18n setup and I've agreed to take lead on that. And there's further potential wrangling around Launchpad, branches, Bazaar, Git, etc. that I'll write up tomorrow. We are (as in Patrick is) currently working on getting a Fedora 22 builder added to the jenkins instance so that we have a python 3.4 available for you folks if you so desire :) Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [hyperkitty-devel] Mailman Suite 3.0rc1 awaiting an upstream fix
On Mon, Apr 20, 2015 at 08:19:29PM +0200, Aurelien Bompard wrote: We are (as in Patrick is) currently working on getting a Fedora 22 builder added to the jenkins instance so that we have a python 3.4 available for you folks if you so desire :) Fedora 21 has Python 3.4, so that would be enough. That is true, but currently we only have a F20 builder, but good to know that if we do not manage to get the F22 one running, we can also try getting a F21 one up. Thanks, Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Python 3
On Fri, Dec 26, 2014 at 04:38:08PM +0100, Aurelien Bompard wrote: [...] then I'll merge this to trunk and do a 3.0b5 release as a Python 3-only application. Mailman 3 will not be bilingual so Python 2 support will be dropped. Wow. I am very surprised. So we went from Python3 compatibility is not on the blocker list for Mailman 3.0 to Mailman 3.0 will only work on Python 3. That's quite a change. This means I must now port HyperKitty and Kittystore to Python3, check that we don't use Py2-only libraries, etc. And mailman.client must be ported too, since it does import stuff from mailman for testing purposes. I think Postorious is safe, but that must be verified (I believe it imports the testing framework from mailman.client and thus must be ported too). This also means that mailman 3 will pretty much only not work on server-oriented distro (except probably Debian that likely ships a python 3 version, although not being involved in Debian, I do not know for sure). For the record, neither RHEL7 nor SL7 nor CentOS7 currently have a python3 stack... And I thought the 3.0 release was almost there. I must say this is rather discouraging. Same for me. Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Archives on MM3
On Fri, Aug 08, 2014 at 04:51:25PM -0300, Alex Azevedo wrote: Hey guys, Do we have a web interface to get list archives? Are they available (archives) on MM3 ? You may be interested in the HyperKitty project: https://fedorahosted.org/hyperkitty/ Demo instance: https://lists.stg.fedoraproject.org/archives/ Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] GSOC 2013 - Introduction and Project Discussion
On Sat, 2013-04-06 at 15:03 +0530, Udit Saxena wrote: 2. Web Posting Interface. I haven't really followed the GSoC ideas for MM3 this year but that line poped-up on my radar. Isn't this similar/overlapping to what HyperKitty already does? I don't think one would want to embed posting messages from the admin interface of mailman (postorious), so posting from an interface would have to deal with archives as well (since one might want to reply to an existing thread). That's the reasoning that brought me to the question above :) Did I miss something? Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Developers] Logo license
Good morning everyone (© Farnsworth), I was wondering if there is a specific license for the Mailman logo. I could find the svg file on the website [1] but it does not mention a license. Basically, we have someone interested in working on an icon for hyperkitty as part of the Google Code-In and I was wondering if we could reuse/modify/use part of the mailman logo. Thanks, Pierre [1] http://www.gnu.org/software/mailman/otherstuff.html ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Speaking about kitties (or archivers)
On Fri, 2012-06-01 at 21:33 -0400, Barry Warsaw wrote: * Look at the IMessageStore API. Is this complete? IOW, could you build a purely Python-level archiver like HyperKitty on top of this API? Here's where proper attachment handling would probably be necessary. With the help of wacky on #mailman this week-end we came up with this patch. Attached is both the diff and the file itself. This way we can start discussing :) Pierre === modified file 'src/mailman/interfaces/messages.py' --- src/mailman/interfaces/messages.py 2012-01-01 19:14:46 + +++ src/mailman/interfaces/messages.py 2012-06-03 16:23:46 + @@ -65,35 +65,125 @@ def add(message): Add the message to the store. -:param message: An email.message.Message instance containing at least -a unique Message-ID header. The message will be given an -X-Message-ID-Hash header, overriding any existing such header. -:returns: The calculated X-Message-ID-Hash header. -:raises ValueError: if the message is missing a Message-ID header. -The storage service is also allowed to raise this exception if it -find, but disallows collisions. - - -def get_message_by_id(message_id): -Return the message with a matching Message-ID. - -:param message_id: The Message-ID header contents to search for. -:returns: The message, or None if no matching message was found. - - -def get_message_by_hash(message_id_hash): -Return the message with the matching X-Message-ID-Hash. - -:param message_id_hash: The X-Message-ID-Hash header contents to -search for. -:returns: The message, or None if no matching message was found. +:param message: An email.message.Message instance containing at +least a unique Message-ID header. The message will be given +an X-Message-ID-Hash header, overriding any existing such +header. +:returns: The calculated X-Message-ID-Hash header. +:raises ValueError: if the message is missing a Message-ID +header. +The storage service is also allowed to raise this exception +if it find, but disallows collisions. + + +def add_to_list(list_name, message): +Add the message to a specific list of the store. + +:param list_name: The fully qualified list name to which the +message should be added. +:param message: An email.message.Message instance containing at +least a unique Message-ID header. The message will be given +an X-Message-ID-Hash header, overriding any existing such +header. +:returns: The calculated X-Message-ID-Hash header. +:raises ValueError: if the message is missing a Message-ID +header. +The storage service is also allowed to raise this exception +if it find, but disallows collisions. def delete_message(message_id): Remove the given message from the store. -:param message: The Message-ID of the mesage to delete from the store. -:raises LookupError: if there is no such message. +:param message: The Message-ID of the mesage to delete from the +store. +:raises LookupError: if there is no such message. + + +def delete_message_from_list(list_name, message_id): +Remove the given message for a specific list from the store. + +:param list_name: The fully qualified list name to which the +message should be added. +:param message: The Message-ID of the mesage to delete from the +store. +:raises LookupError: if there is no such message. + + +def get_list_size(list_name): + Return the number of emails stored for a given mailing list. + +:arg list_name, name of the mailing list in which this email +should be searched. + + +def get_message_by_hash(message_id_hash): +Return the message with the matching X-Message-ID-Hash. + +:param message_id_hash: The X-Message-ID-Hash header contents to +search for. +:returns: The message, or None if no matching message was found. + + +def get_message_by_hash_from_list(list_name, message_id_hash): +Return the message with the matching X-Message-ID-Hash. + +:param message_id_hash: The X-Message-ID-Hash header contents to +search for. +:returns: The message, or None if no matching message was found. + + +def get_message_by_id(message_id): +Return the message with a matching Message-ID. + +:param message_id: The Message-ID header contents to search for. +:returns: The message, or None if no matching message was found. + + +def get_message_by_id_from_list(list_name, message_id): +Return the
Re: [Mailman-Developers] Speaking about kitties (or archivers)
On Fri, 2012-06-01 at 21:33 -0400, Barry Warsaw wrote: * Look at the IMessageStore API. Is this complete? IOW, could you build a purely Python-level archiver like HyperKitty on top of this API? Here's where proper attachment handling would probably be necessary. I have defined a number of functions in the kittystore interface which could go into the IMessageStore: https://github.com/pypingou/kittystore/blob/master/kittystore/__init__.py Main difference compare to what is in the IMessageStore at the moment is that I require the fq list name as first argument of each function (ie: I have one table for each list and need to know where you would like me to search) * How would you want to expose the IMessageStore interface into the REST API? My sense is that you could probably take a fairly straightforward translation of IMessageStore into REST and *that* would be what you'd build the various archiver UIs on top of. REST needs to answer questions like batching which are necessary for efficient transfer of data over HTTP but not for direct Python calls. +1 for the straightforward translation of the interface to REST. Archiver could then use either the implementation of the interface or REST. * Should threading information be part of the IMessageStore, or a separate interface? If the prototype archiver becomes the default implementation for the IMessageStore, it probably needs to grow a lot more functionality to support threading information. Keeping some info about threading in the database, the KittyStore interface allows to retrieve for example all the messages of a thread or the length of a thread. I can work on merging KittyStore and IMessageStore and providing the REST interface for it. Should I provide also an implementation ? Using PostgreSQL as backend ? Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Speaking about kitties (or archivers)
On Sun, 2012-06-03 at 14:49 +0200, Pierre-Yves Chibon wrote: Main difference compare to what is in the IMessageStore at the moment is that I require the fq list name as first argument of each function (ie: I have one table for each list and need to know where you would like me to search) hm, the fact that I would require the full list name will cause a problem regarding the `messages` attributes on the IMessageStore. Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Speaking about kitties (or archivers)
On Thu, 2012-05-31 at 10:28 -0700, Toshio Kuratomi wrote: The idea is that now, we can have different backend and each module needing access to the emails can use the API directly without having to bother about which storage system is behind. Wacky looked at this today and asked if we should have the x-message-id-hash as another key value to look up an email. I am all for extending the API as much as we need, and I am sure the different modules have different needs, so please let us know what you would like to have access to. Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Developers] HyperKitty MongoDB vs PostgreSQL.
Dear all, It has been some time since the last info regarding HyperKitty, but the project has made some progress. I have implemented to database interface that I was presented already a month ago [1]. So now there is a standalone project/library, KittyStore [2] which provides an interface to the database. It defines an interface which can then be implemented for whatever database system you would like. At the moment it covers MongoDB and PostgreSQL. Once I implemented this interface I tried to solve the question of which database system should we primarily focus on. I wrote a small comparison test of the two systems on an RHEL6 system [3] (I still have to publish the results for F17). The difference between the two databases system is not so large (1s for a query that already takes 6 seconds) and there was not any tuning of the servers. So I think the advantage of having only one back-end for mailman and its archive is worth this time difference in the results (which might anyway get even better). Thus we will move forward with PostgreSQL as a back-end for HyperKitty. The good news is that HyperKitty already works fine with PostgreSQL and KittyStore (if you use the correct branch [4]). However, we have to rebuild our test server, so we cannot show you how the latest version works right now. So all is nicely getting in place for Aamir to start working on his project and for HK to get further down the road :) I think that's about all I wanted to say, fire away if you have questions! Best regards, Pierre [1] http://mail.python.org/pipermail/mailman-developers/2012-April/022012.html [2] https://github.com/pypingou/kittystore [3] http://blog.pingoured.fr/index.php?post/2012/05/20/PostgreSQL-vs-MongoDB [4] http://bzr.fedorahosted.org/bzr/hyperkitty/rdbms/files ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Speaking about kitties (or archivers)
On Tue, 2012-04-24 at 11:12 -0700, Toshio Kuratomi wrote: [mailman3 core] -- maintainance of list metadata, sending and receiving, provides a REST API [Web UIs] -- web ui to Core functions [Archive-stores] -- stores the messages sent to the mailing lists. Provides a (REST?) API to apps built on top of it [Archiver UIs] -- web ui, nntp interface, REST API (if not implemented at the storage layer), etc to the archive-store By splitting the archive storage from the archive UI similar to how mailman3-core splits with the web ui, we can allow a sysadmin to choose one archive-storage for all of the archive front-ends that they run on their systems. Thank you Toshio for explaining this in a better than I was able to do. Question: Where do we start? I think that we'll either succeed or fail very quickly by trying to define what the API between archive-store and archiver-ui should look like. We'll either be able to agree on a common set of features there (from which we'll be able to go forth and create our own archive-storage plugins) or we'll decide that we all need/want to do different things that no common API can address. If there's no common API definition then we won't be able to do any of the rest of this so there won't be any sense continuing down that path. The current version of HK relies on mongodb for the storage, but I want to test HK with a traditionnal SQL backend. So I have started to work on this. The interface I defined is there: https://github.com/pypingou/kittystore/blob/master/kittystore/__init__.py And its implementation using SQLAlchemy is there: https://github.com/pypingou/kittystore/blob/master/kittystore/kittysastore.py The mongodb implementation isn't done yet but should be quite trivial to do (most function from the API were coming from it). The idea is that now, we can have different backend and each module needing access to the emails can use the API directly without having to bother about which storage system is behind. I hope this helps, Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Speaking about kitties (or archivers)
Resent as the first one doesn't seem to want to arrive. On Tue, 2012-04-24 at 11:12 -0700, Toshio Kuratomi wrote: [mailman3 core] -- maintainance of list metadata, sending and receiving, provides a REST API [Web UIs] -- web ui to Core functions [Archive-stores] -- stores the messages sent to the mailing lists. Provides a (REST?) API to apps built on top of it [Archiver UIs] -- web ui, nntp interface, REST API (if not implemented at the storage layer), etc to the archive-store By splitting the archive storage from the archive UI similar to how mailman3-core splits with the web ui, we can allow a sysadmin to choose one archive-storage for all of the archive front-ends that they run on their systems. Thank you Toshio for explaining this in a better than I was able to do. Question: Where do we start? I think that we'll either succeed or fail very quickly by trying to define what the API between archive-store and archiver-ui should look like. We'll either be able to agree on a common set of features there (from which we'll be able to go forth and create our own archive-storage plugins) or we'll decide that we all need/want to do different things that no common API can address. If there's no common API definition then we won't be able to do any of the rest of this so there won't be any sense continuing down that path. The current version of HK relies on mongodb for the storage, but I want to test HK with a traditionnal SQL backend. So I have started to work on this. The interface I defined is there: https://github.com/pypingou/kittystore/blob/master/kittystore/__init__.py And its implementation using SQLAlchemy is there: https://github.com/pypingou/kittystore/blob/master/kittystore/kittysastore.py The mongodb implementation isn't done yet but should be quite trivial to do (most function from the API were coming from it). The idea is that now, we can have different backend and each module needing access to the emails can use the API directly without having to bother about which storage system is behind. I hope this helps, Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Developers] Speaking about kitties (or archivers)
Meeow miaou* We spoke on IRC about the archiver the other day and I said that I should present here my thoughts about it. So here they are (beware that might be long). First I think we should think about the structure/architecture of things. We have a number of component which need to be archives aware, without being exhaustive I'm thinking about: - the archiver itself (which present the archive (ie: mails and threads) - the NNTP bits which should be able to return emails and/or threads - the stats module which want to give information to the user about the health of the list itself (emails/month, last threads, biggest threads...) - archives retrieval (we probably want to give the user a way to download the archives since the creation of the list/the last year/month) All of these components needs to be aware about the archives. We agreed that the core does not want to know about it. So we have several solutions: - each module becomes an archiver wrt to core, meaning each module has its own way to storing the archives (and eventually its own system to do so) - we create a archive-core module which manage the archives and provides an API to access, modify, extend them. Of course, we prefer the second solution :) So we would have the following architecture: mm-core (handles the lists themselves) --send emails to archivers-- archive-core (store the emails and expose them through an API) -- archivers/stats/NNTP The questions are then: - how do we store the emails ? - how do we expose the API ? - how to make it such that it becomes easy to extend ? (ie: the stats module wants to read the db, but probably also to store information on it) Having played with mongodb (HK relies on it atm), I quite like the possibilities it gives us. We can easily store the emails in it, query them and since it is a NoSQL database system extending it becomes also easy. On the other hand, having the archiver-core relying on the same system as the core itself would be nicer from a sysadmin pov. I have not tried to upload archives to a RDBMS and test its speed, but for mongodb the results of the tests are presented at [1]. The challenge will be speed and designing an API which allow each component to do its work. I think it would be nice if we could reach some kind of agreement before the GSoC starts (even if we change our mind later on) to be sure that if we get students their work don't overlap too much. The second point I want to present is with respect to the archiver itself. At the moment we have HyperKitty (HK), the current version: - exposes single emails - exposes single threads - presents the archives for one month or day - allows to search the archives using the sender, subject, content or subject and content - presents a summary of the recent activities on the list (including the evolution of the number of post sent over the last month) I think these are the basis functionality that we would like to see in an archiver. But HK aims at much more, the ultimate goal of HK is to provide a forum-like interface to the mailing-lists, with it HK would provide a number of option (social-web like) allowing to like or dislike a post or a thread, allowing to +1 someone, allowing to tag the mails or assign them categories. These are all nice feature but, imho, they go beyond what one would want from a basic archiver. So what I would like to propose is to split HK into a sub-project (MiniKitty?) which would provide these basic functionality. We would keep HyperKitty as a more extensive archiver and try to bring HK to its ultimate goal. This will need some more work and time as we will have to make HK speak with core for authentication, find a way to send emails to core/the lists and of course add all the other features (tags, categories...) Comments welcome :) Thanks, Pierre [1] http://blog.pingoured.fr/index.php?post/2012/03/16/Mailman-archives-and-mongodb * Hi everyone ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Developers] From the creation of a ThreadID
Hi, In HyperKitty to be able to easily retrieve from the database all the threads of a given month or just all the emails of a thread, I created a Field in the database called ThreadID. When I load the archives from mailman into mongo, I look for the absence of the headers 'References' or 'In-Reply-To' to define an email that starts a new thread. Then all emails which have the header 'References' or 'In-Reply-To' will look for the preceding email and extract the ThreadID from it. This seems to work fine. At the beginning I was using a simple integer as identifier but of course if you changed the order in which the archives are loaded or just if you miss like one month than the ThreadID is not consistent anymore. So I changed to use the Message-ID of the first email of the Thread as ThreadID. Problem is of course, if the admin removes the first email of a thread for x or y reasons, then when reloading the archives (for z or a reasons), we will loose the ThreadID and actually, the integrity of the Thread (each reply to the first email will be split into their own thread). Would anyone have an idea on how to generate a stable and delete/reload proof ThreadID? The other solution of course being that I regenerate the thread on the fly based on the first email (which is still easy to find), but that will be a lot of db querying. Thinking about it, generating the thread on the fly would also give the possibility to regenerate the thread view from anywhere (so you could generate a thread view for only a sub-thread). Do you have any suggestions/preferences ? Thanks, Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] From the creation of a ThreadID
On Fri, 2012-04-06 at 00:10 +0900, Stephen J. Turnbull wrote: On Thu, Apr 5, 2012 at 10:41 PM, Pierre-Yves Chibon pin...@pingoured.fr wrote: In HyperKitty to be able to easily retrieve from the database all the threads of a given month or just all the emails of a thread, I created a Field in the database called ThreadID. When I load the archives from mailman into mongo, I look for the absence of the headers 'References' or 'In-Reply-To' to define an email that starts a new thread. This fails when a thread crosses channels. Eg, To: Pierre From: Steve Message-Id: x@y.z is followed by To: Steve From: Pierre Cc: SomeList References: x@y.z Message-Id: a@b.c Would anyone have an idea on how to generate a stable and delete/reload proof ThreadID? I don't see how this can be possible. Eg, in the above scenario you construct a thread based on your reply to me. Then I go, oh, really I should have posted to mm-dev and repost the thread. So the Message-ID of root message fails, and I don't see an alternative that can be predicted. So it may as well be arbitrary (eg, any message in the thread) and stored in the database with appropriate linkage from thread IDs to message IDs (one-to-many), and vice versa (many-to-one). Ok, I missed a something here. So when it parses the email, it checks for 'References' or 'In-Reply-To'. - If it finds them, it looks for the preceding email - if it finds the preceding email, then the current email gets the ThreadID from the preceding email - if it does not find the preceding email, then the current email is assumed to be a new thread and thus its ThreadID is its Message-ID - if it does not find 'References' or 'In-Reply-To', then the current email is assumed to be a new thread and thus its ThreadID is its Message-ID So for the example you give, the archiver will receive your email and make a new thread out of it. The other solution of course being that I regenerate the thread on the fly based on the first email (which is still easy to find), but that will be a lot of db querying. I haven't thought about it deeply, but I would say just give the thread an arbitrary ID in the database. Message-IDs are supposed to universally unique, so what's wrong with keeping the thread in the database as a tree of message IDs? Some Message-IDs will not have corresponding messages but that's always a problem with threading (see http://www.jwz.org/doc/threading.html, and RFC 5256). The idea of using the Message-ID for ThreadID (instead of a integer) is that, if I whether I load one months or two months of archives into the database, the link to the thread (http://mm3test.fedoraproject.org/thread/packaging@fp.o/XU7HT5JC5GND2O4JII7MTQILLTB4IN4S) will remain the same (so consistent urls). There are other problems with threading that need to be dealt with as well, such as References being inconsistent across messages in the same thread and people who continue a thread with a new message, etc. For these I am not sure I can do something (at least automatically, we could always allow an admin to edit the field). Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] GSoC
On Thu, 2012-03-29 at 10:49 +0900, Stephen J. Turnbull wrote: Hyperkitty is the demo that a few people have been working on, including Toshio at the sprints, but there are others in play (eg, recent posts to this list). Also, Hyperkitty is currently based on somewhat limited technology (the notmuch indexer). I'm not sure it would be up to the demands of data-mining. Of course, if Toshio says it is, listen to him, not me. :-) For the record, the current version of HyperKitty has moved from a notmuch backend to a mongo database allowing a much more flexible data storage scheme. This is the one you can see online at: http://mm3test.fedoraproject.org/ The demo is getting more and more complete with month view, recent activities, thread view, basic search functions and categories. My only problem is that we seem to have some memory leaks at the moment :-/ Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] GSoC
On Thu, 2012-03-29 at 21:49 +0200, Pierre-Yves Chibon wrote: On Thu, 2012-03-29 at 10:49 +0900, Stephen J. Turnbull wrote: Hyperkitty is the demo that a few people have been working on, including Toshio at the sprints, but there are others in play (eg, recent posts to this list). Also, Hyperkitty is currently based on somewhat limited technology (the notmuch indexer). I'm not sure it would be up to the demands of data-mining. Of course, if Toshio says it is, listen to him, not me. :-) For the record, the current version of HyperKitty has moved from a notmuch backend to a mongo database allowing a much more flexible data storage scheme. This is the one you can see online at: http://mm3test.fedoraproject.org/ The demo is getting more and more complete with month view, recent activities, thread view, basic search functions and categories. My only problem is that we seem to have some memory leaks at the moment :-/ We might still get a third version of this email. As it seemed that they were not reaching (which apparently they do now), I try to resend them. Sorry for the noise. Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
On Mon, 2012-03-26 at 17:11 -0700, Toshio Kuratomi wrote: Grackle is another archiver for mailman that doesn't have the UI bells and whistles of hyperkitty but it does make an effort to expose a REST UI to the world. I think that's a beautiful thing. I started a small thing on hyperkitty there: http://mm3test.fedoraproject.org/api/ Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] GSoC
On Thu, 2012-03-29 at 10:49 +0900, Stephen J. Turnbull wrote: Hyperkitty is the demo that a few people have been working on, including Toshio at the sprints, but there are others in play (eg, recent posts to this list). Also, Hyperkitty is currently based on somewhat limited technology (the notmuch indexer). I'm not sure it would be up to the demands of data-mining. Of course, if Toshio says it is, listen to him, not me. :-) For the record, the current version of HyperKitty has moved from a notmuch backend to a mongo database allowing a much more flexible data storage scheme. This is the one you can see online at: http://mm3test.fedoraproject.org/ The demo is getting more and more complete with month view, recent activities, thread view, basic search functions and categories. My only problem is that we seem to have some memory leaks at the moment :-/ Pierre ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
