Re: [Mailman-Developers] Google Summer of Code: Integration of Search Code
On Tue, Mar 27, 2012 at 1:11 AM, Toshio Kuratomi a.bad...@gmail.com wrote: On Mon, Mar 26, 2012 at 04:57:44PM +0530, Shayan Md wrote: Hi, I am Shayan, I am doing my masters from IISc Bangalore. I want to take part in GSoC from mailman organization. I have fairly good experience in python. I worked on whoosh library for my own project. I have experience with django also. I am planning to work on integrating systers search code into mailman. I believe for whole summer this task alone is very small. I'll include couple of more tasks in the proposal. I went through search code. Code looks pretty standard. As far I understood mailman.Searcher.IndexSchema has whoosh schema mailman.Seracher.Indexer contains indexing functions which are later used in genindex.py mailman.Seracher.Mailocate has search function for indexed content generated by genindex.py mailman.bin.genindex.main() is the main function which indexes the archives I am still trying to figure out how mailman client library interacts with mailman server. I know how to use client library but I couldn't understand internal working of mailman server. I have been reading the code but I wasn't able understand much. It would be awesome if you can point me to any documentation to understand this. Some api documentation or anything to understand code. Is this integration to be done with mailman2 or mailman3? In mailman3, the archivers are separated from the mailman core. I was working on mm3. But systers' indexer/searcher was implemented for mailman2. So it must be easy for to integrate it with mm2. Looks like archiver for mm3 is still in development stage. As far as I understand searcher depends on the srchiver, right? Not completely but it somewhat depends on archiver. I am not sure if searcher can be implemented without archiver. If possible I can implement for mm3 also. -Toshio ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
I brought up a quick CSLA injest of some mailman-dev posts to show off some basic features. I'm having a little trouble with the swish-e code, so the features that depend on text-indexing arn't working at the moment (search, author search). The current UI uses user-cookies to choose between a 2-pane frames mode and a noframes mode. I set the frames-mode as the default, which is inspired by the old google-groups usenet multi-message tree. Use prefs to check out the noframes mode, which is more like pipermail. Take a peek more for general capabilities than UI details (unless you see something you really like). I'm planning to change the UI to be more like the gmail-esq conversation style UIs I've been using since then... (where most/all of a thread's messages are rendered into a single page) I like the outgoing google-groups UI pretty well (not the new AJAXy one). 1) basic home for a list with bymonth and top-author metrics, RSS. Extended top-author page. http://dj1.willowmail.com/csla/Mailman-Developers http://dj1.willowmail.com/csla/Mailman-Developers/top_authors?date=2012-03 2) Prefs, including timezone, read-status key, browse-style.. (not sure if the read state is working on my install) http://dj1.willowmail.com/csla/Mailman-Developers/prefs 3) threadlist for the month (frames or noframes in prefs) http://dj1.willowmail.com/csla/Mailman-Developers/browse/month/2012-03/107 4) single-thread-list display (frames or noframes in prefs) http://dj1.willowmail.com/csla/Mailman-Developers/threads?start=1 5) here is a good example of a thread with redundant text eliding http://dj1.willowmail.com/csla/Mailman-Developers/browse_frm/thread/4/221 ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
On Tue, Mar 27, 2012 at 2:37 PM, David Jeske dav...@gmail.com wrote: Can you share something about dependency philosophy (besides licensing) in Mailman? Well, for the official poop you'll have to wait for Barry, but AFAICS archivers aren't restricted to Storm + RESTish (which is what Mailman itself uses) because they're separate applications. If the archiver/web UI is going to be distributed *with* Mailman, Barry would probably prefer Storm + Django because that's what Mailman/Protorius (core and admin web UI, resp.) are using. But I imagine that's negotiable as long as everything is free software. I don't think the database backend much matters, but Mailman and Django are both happy to use sqlite, I believe. I'm not a fan of javascript client-side rendering because of the generally poor performance, poor mobile compatibility, and lack of benefit for this kind of application. Not my pidgin, you'll need to talk to Toshio/Barry about that. AIUI, the current philosophy is that (1) the communication from Mailman to the archivers will be via LMTP/SMTP, including a Mailman-specific header to identify the message's permalink (currently the SHA1 hash of the message ID in BASE32 format, IIRC); and Using SMTP for an included archiver would require it be a long running server instead of merely a handler. Sure, but you're already running a sufficient server, namely your MTA. Of course that may not be efficient. If an mbox or maildir storage is sufficient, any decent MTA/MDA can do that for you. If not, I don't think that writing a separate handler is a big deal; I'm just saying that AIUI the configurable Handler provided with Mailman will certainly know how to do LMTP/ SMTP. Are we talking about third-party-site archivers here? That is the motivation for choosing ?MTP as the default transport to archivers, yes. Steve ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] [Bug 937154] Re: bin/disabled.py is nonfunctional
On Mar 26, 2012, at 08:03 PM, Andrea Crotti wrote: Should it provide exactly the same command line interface? Not necessarily. Looking at the code now I think the long options are probably fine, but I'm not sure the short options are great (e.g. -o is usually reserved for output to a file; not relevant for this particular script, but I don't like appropriating commonly understood options if at all possible). Also --unknown won't be useful now; that was a nod to some bug in MM2.1 we never figured out. ;) If you make it a bin/mailman subcommand, you won't need to re-implement -C here. - Does it need to be independent from the rest of the code or should it be added to the mailman commands?? I think I would make it a bin/mailman subcommand. One of the main use cases for this script is to implement a cron job that site admins can enable to process users in various statuses. But I think a cron could invoke a subcommand just fine. Still, that use case should inform its cli. Anything else that I should know? One thing to keep in mind is that in mm3, I'm trying to reduce the amount of logic in the actual commands. Meaning, it would be better to move as much as possible into core services, and have the command do as little as possible, over cli parsing and such. ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
On Mar 27, 2012 5:32 AM, Stephen J. Turnbull step...@xemacs.org wrote: If the archiver/web UI is going to be distributed *with* Mailman, Barry would probably prefer Storm + Django because that's what Mailman/Protorius (core and admin web UI, resp.) are using. But I imagine that's negotiable as long as everything is free software. I'll take a look at the Mailman2/3 code and see what the pattern looks like. Storm-ORM looks like it allows any primary key so that's good. From a quick glance it looks very similar to the Clearsilver-odb-orm in syntax and function. My primary beef with django-orm (last time I looked at it) was it requiring a unique-id primary key which is pretty poor for multi-row fetch performance design. Clearsilver templating has a slightly more aggressive enforcement of 'no code in template' than django, because it has an explicit intermediate dataset called HDF (it doesn't allow access to arbitrary python lists, structs, classes, and methods). This allows it to work from any language (not just python), and makes the binding from the ORM a bit more explicit. However, it also means it's a c-module, and I can see the advantage to keeping c-compile dependencies to a minimum. AIUI, the current philosophy is that Using SMTP for an included archiver would require it be a long running server instead of merely a handler. Sure, but you're already running a sufficient server, namely your MTA. Gotcha. I thought you were saying Mailman wanted the archiver to be a long-running server accepting SMTP/LMTP on a socket. I normally manage the archive queue with the MTA, and have it launch the archive-injestor for each message.. (or for higher volume applications, long-running injester on a Maildir).. which sounds like what you're saying.. After I finish some quick cleanup and UI changes to CSLA I want to do, I'll take a look at MM2/3 and see if I have better questions once I'm informed. Thanks for the great overview. ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Google Summer of Code: Integration of Search Code
On Tue, Mar 27, 2012 at 03:01:41PM +0530, Shayan Md wrote: On Tue, Mar 27, 2012 at 1:11 AM, Toshio Kuratomi a.bad...@gmail.com wrote: Is this integration to be done with mailman2 or mailman3? In mailman3, the archivers are separated from the mailman core. I was working on mm3. But systers' indexer/searcher was implemented for mailman2. So it must be easy for to integrate it with mm2. Looks like archiver for mm3 is still in development stage. As far as I understand searcher depends on the srchiver, right? Not completely but it somewhat depends on archiver. I am not sure if searcher can be implemented without archiver. If possible I can implement for mm3 also. The searcher wouldn't be much use without an archiver. There is a sample archiver in mailman core -- if enabled, it stores the messages to lists in maildirs. It does not have a frontend for retrieving or otherwise displaying the archives. In some of the other threads here, it was brought up that a builtin archiver with the same feature set of pipermail could be desirable. If you're interested in integrating search into mailman I'd watch (and participate!) in that thread to see what the outcome of that discussion is. -Toshio pgp13W2e2H6ZJ.pgp Description: PGP signature ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Google Summer of Code: Integration of Search Code
On 03/27/2012 03:31 AM, Shayan Md wrote: I was working on mm3. But systers' indexer/searcher was implemented for mailman2. So it must be easy for to integrate it with mm2. Actually, the systers indexer was designed to work with mboxes (because I had a pile of data in that format that the students could use) but otherwise knows pretty much nothing about mailman 2 or 3. Other than handling mbox instead of maildir, which is only a matter of changing parsers, it shouldn't matter which it's integrated with. This was a design decision at the time, as Mailman 3 was coming but still too incomplete to test with when the code was written. Looks like archiver for mm3 is still in development stage. As far as I understand searcher depends on the srchiver, right? Not completely but it somewhat depends on archiver. I am not sure if searcher can be implemented without archiver. If possible I can implement for mm3 also. Searcher and archiver are interdependent *if* we want to share caches and data stores, which we probably do for any installation with larger archives where storing 2 copies vs 4 of each message would make a difference. Plus, many archive views may be basically searches messages in the last month messages which are replies to messageid $foo etc. Ideally, anyone working on search will interact heavily with the archiver and probably usability folk at the beginning so that you can figure out what data structures you need to store and index and what use cases you'll need to make fast. Terri ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Developers] GSoC 2012 - NNTP archive access
Hi, I would like to participate at the google summer of code this year for mailman. While reading through the ideas in the wiki, the NNTP archive access look very interesting. I took part in the gsoc last year for vlc, developing a c library for accessing a sony minidisc player from linux. This year vlc unfortunately got rejected from google. What are the next steps you would propose. I unfortunately not up to date with the development of mailman 3. But I am a little bit familiar with the mailman 2 source code. Thanks, Alexander pgpdjk2tHe8bM.pgp Description: PGP signature ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
On 03/26/2012 11:37 PM, David Jeske wrote: CSLA doesn't currently have any concept server-auth. The only stateful features it has are view-preferences and read-state, neither of which are important enough to require a password. It uses a password-less system which uses cookies for prefs and a 'read state userid' which a user can manually set if they want. I like it, because it doesn't require login to get some basic browsing prefs and features. Hooking up an auth system would be necessary for some of the editing ideas in the document I read, or to allow online posting. So Postorius (the webUI) has a sketch of an auth system using BrowserID at the moment. BrowserID is convenient 'cause it proves you have ownership of a given email address, but we should have OpenID working soon once we've got the code to confirm that a given OpenID can be associated with an email address. We should do a little thinking about how to make sure that the archives system can make use of the webui authentication. In theory, you could just use the same browserID/etc. and perform authentication again to provide a single sign on with the same tokens, but we can probably do something nicer by sharing the webui django accounts. Terri ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
On Mar 26, 2012, at 03:20 PM, David Jeske wrote: I'm writing to find out the state of and philosophy surrounding pipermail in mailman, to see if there is a productive way to provide some code/development-time to that part of mailman. That's awesome. Pipermail (in its current state) is ancient, and it shows. It's biggest flaw IMO is the lack of stable URL support, meaning if you regenerate the archive from the underlying mbox file, you'll likely break all your links. Pipermail has lots of other problems, but this one for example is why it was ditched for Launchpad. I think the time is ripe for new archivers, and Mailman 3's philosophy is to provide a framework in which developers can easily experiment with new archiver technology, integrated easily into Mailman. I personally think it would be best to spend effort on one of the many archiver projects being worked on, and mentioned here, rather than trying to improve Pipermail. One possible connection though would be to consider how a site would migrate from a Pipermail-based archiver to one of the new ones. Do you keep the old URLs alive but just not add anything new to Pipermail? Do you provide a mapping from Pipermail URLs to new-archiver URLs? etc. I've written code for this a number of times (eGroups, Yahoo Groups, Google Groups). I also released an open-source python/clearsilver/sqlite based archiver with redundant text-eliding, a few different thread views, and search... ( http://www.clearsilver.net/archive/ ) which is hardly used both because I don't try to popularize it, and because many sites just leave the default (pipermail). Neat. Perhaps you'd like to contribute an implementation of the IArchiver interface in Mailman 3 that would send posted messages to ClearSilver? Here's the current interface definition: http://tinyurl.com/cup7amw and some example implementations: http://tinyurl.com/cynyfou a) pipermail is fine... if you want to fix a bug or two submit a patch, but we don't want to improve it Note that before I ripped Pipermail out of the Mailman 3 trunk branch, I created a project on Launchpad and pushed a semi-sanitized branch: https://code.launchpad.net/pipermail I don't personally plan to touch this, but if someone is really motivated to hack on Pipermail specifically, I'd happily give you access. b) we're ditching pipermail entirely... in the future sites will have to choose an install an external archiver Yes, but remember, you can choose one or more archivers in Mailman 3. So it would be easy to archive to something local, and The Mail Archive, and Gmane, etc. c) we'd love pipermail to be improved... but we still want it to be simple, static-html, and dependency free d) we'd love a dynamic-ui replacement for pipermail... as long as it uses the same cgi/templating model as mailman ui Because Postorius (the official, but not required MM3 web ui) is Django-based, I think it should be pretty flexible for customizing the look and feel. I won't dictate what a new archiver should look like, but some principles that I personally think should be followed include: - Modern web technologies, with flexible templating, so that sites can customize the look and feel as needed. - JavaScript not required: an archiver should be navigable with a text-based simple browser, but that also doesn't mean it has to be feature-equivalent if you *do* have JavaScript. I think progressive enhancement is the right term? I.e. make it awesome for today's web, but usable in older browsers, screen readers (for accessibility), etc. - Dynamic generation of pages with caching. I'd love to see an enhanceable approach to the actual HTML generation. Let's say for example, that I suddenly want to recognize and hyperlink bug numbers so they point to my tracker. I should be able to drop in some extension to do that. Or maybe the spammers have cracked my email obfuscation algorithm. I should be able to drop in a replacement, invalidate the cache, and all my new page would automatically get the new obfuscation algorithm. - Separate, dynamic support for take down notices. You posted your personal information and have asked the site's postmasters to take that down (either the full message or the personal information parts of it). It should be really simple for the site admins to do this, while possibly retaining the original message in a publicly inaccessible location for forensic purposes. - Support for private archivers, so configurable authentication would be necessary. - Merging of forums, archives, newsgroups, and IMAP. - A REST API for querying information about the site, its lists, individual archived messages, metrics, etc. Maybe even some control (e.g. take downs) for users with the proper permissions. - Stable URLs, RFC 5064 + X-Message-ID-Hash. See the above links. If you can implement the `IArchiver.permalink()` method and ensure that even if completely wiped and regenerated from the
Re: [Mailman-Developers] RELEASED: GNU Mailman 3.0 beta 1 andPostorius 1.0 alpha 1
What is the incantation for enabling an external archiving service? Currently I only see this in mailman.cfg after following 5 minute guide. [runner.archive] class: mailman.runners.archive.ArchiveRunner archivers are configured site-wide, so there's almost nothing to expose in the web-ui. I'm worried about confusion. The last thing we want is for a list to be accidentally archived contrary to the list administrator's wish. It sounds scary to me not to have any indication whatsoever in the web interface. Along similar lines, there seems opportunity for confusion if there are two independent mechanisms for archival; site wide configuration and also manually subscribing an archival subscriber such as arch...@mail-archive.com. I can imagine someone turning off just one of these two mechanisms then being surprised that it had no practical effect. Finally, it sounds like there are architectural reasons for having archiving a site-wide configuration. But I do think list admins would appreciate some sort per list GUI option, to easily distinguish between public and private lists. These are often different folks from the sysadmin who can apt-get install mailman without giving a first glance at the mailman.cfg file. -Jeff ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Google Summer of Code: Integration of Search Code
On Wed, Mar 28, 2012 at 4:21 AM, Terri Oda te...@zone12.com wrote: Looks like archiver for mm3 is still in development stage. As far as I understand searcher depends on the srchiver, right? Not completely but it somewhat depends on archiver. I am not sure if searcher can be implemented without archiver. If possible I can implement for mm3 also. Searcher and archiver are interdependent *if* we want to share caches and data stores, which we probably do for any installation with larger archives where storing 2 copies vs 4 of each message would make a difference. Plus, many archive views may be basically searches messages in the last month messages which are replies to messageid $foo etc. Actually, as far as I can see, the summary/search/index/retrieval functions depend only on the API for the message store. If you want, you can split this into the database layer and a presentation layer, of course. However, the database layer is surely going to have its own schema optimized for the kinds of retrieval its designer considers important. If the designer emphasizes threads, however, she is *not* going to try to store messages in thread order or anything like that. Rather, any reasonable store will be message-ID-addressable. The only tricky issue is that we *do* have to worry about message-ID collisions of truly different messages and about messages without message IDs, especially for converted historical archives. So the API needs to be able to deal with these issues, probably by returning a set or sequence of messages. Oh, and we probably ought to have a more general notion of retrievable object rather than just messages, as some archive/retrieval backends may store some types of MIME part separately. Hopefully these would be presented to us as MIME parts with external bodies and content IDs. I would guess she'll probably store messages in YY-MM/MSGID, or as git does in unpacked XX/... format, where XX are the first two digits of the hash ID, and YY... are the remaining ones). But it could easily be backed by an IMAP store or something more specialized; we don't really care as long as it's object-ID-addressable. And that's all we want to say about the archiver and the associated message-retrieval logic, I think. (In fact, it occurs to me that maybe we should say RFC 3501 and be done with it. I don't mean that we necessarily implement IMAP protocol per se, but some subset of its functionality probably is what we need from an archiver.) Then the schema-specific stuff will use hash IDs to represent message objects in a portable but schema-specific way. As it's schema-specific, I don't really see how data structures can be shared by different searchers. So I would say not to worry about the archiver side at all. If large installations want to implement specialized message- retrieval, bully for them. But we can go with simple backends, maildir, mbox, and maybe IMAP, I think. ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
On Mar 27, 2012, at 10:34 AM, Stephen J. Turnbull wrote: Pipermail is going the way of the dodo, yes, but there will be something bundled with Mailman, I'm pretty sure. fsvo bundled :) (1) the communication from Mailman to the archivers will be via LMTP/SMTP, including a Mailman-specific header to identify the message's permalink (currently the SHA1 hash of the message ID in BASE32 format, IIRC); and Kind of. Communication *into* Mailman is via LMTP, but sending a message out of mm3 into the archiver can be hidden behind the IArchiver interface. The prototype archiver, just opens a maildir and adds the new message to that. The mailarchive implementation drops the message into the outgoing queue, but with a recipient as specified in the configuration file. The mhonarc archiver calls a subprocess and pipes the message's bytes to that process's stdin. The moral is that whatever you need to do to get the message into the archiver, you can do it. You're right about the algorithm for calculating the X-Message-Id-Hash. Full details are on the wiki. The intent here is to be able to calculate the URL to the message in the archiver without actually having to communicate with the archiver. (2) the summary, search, and retrieval UI will be a separate application from Mailman per se, which will communicate with Mailman via the REST API for authentication and user configuration purposes. Yep. I've also been thinking; we have an IMessageStore interface which currently isn't much hooked up in any way. It's possible that a specific archiver could satisfy the IMessageStore API as well, and that eventually we could implement things like a mail command that, given a Message-ID (or its hash) would send you back the message from the store. Some ideas that have *not* been elaborated as philosophy, but which I think are consistent with various desiderata and Barry's general approach and specific statements are (3) an archiver Handler for the pipeline will be provided, which will store posts in maildir format and do nothing else; and I've been thinking that the prototype archiver could be this thing. Well, it already essentially *is* that thing! So, it's not in a handler, but gets invoked from the ArchiveRunner. (4) a simple default summary/search/retrieval application will be bundled with (but have a separate development cycle from) Mailman. +1 Even if that's not the current philosophy wink/ that's the direction I would propose. So if you have already developed some stuff, I certainly would love to see you put it on the table as a candidate for the Mailman 3 default archive web UI. Definitely. My intent with mm3 is to provide an enabling platform for archiver developers. I think there's lots of opportunity for experimentation here, so let's make sure we get the API between mm3 and a generic archiver right. We'll see what pans out and then we can choose a preferred default that we can bless and bundle in some kind of sumo release. Cheers, -Barry ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
On Mar 27, 2012, at 09:32 PM, Stephen J. Turnbull wrote: Well, for the official poop you'll have to wait for Barry, but AFAICS archivers aren't restricted to Storm + RESTish (which is what Mailman itself uses) because they're separate applications. If the archiver/web UI is going to be distributed *with* Mailman, Barry would probably prefer Storm + Django because that's what Mailman/Protorius (core and admin web UI, resp.) are using. But I imagine that's negotiable as long as everything is free software. I don't think the database backend much matters, but Mailman and Django are both happy to use sqlite, I believe. What Stephen says above is pretty accurate. See also my previous follow ups in this thread. The core engine uses Storm as its ORM, which I think officially supports SQLite, PostgreSQL, and MySQL, though it's probably compatible with others out of the box or easily so. Mailman itself currently only supports SQLite out of the box because that comes with Python, and we have donated support for PostgreSQL. I'd love for someone to donate MySQL support; I think it would be easy for someone with MySQL experience, there isn't much that would need to be added to Mailman, there are good examples now, and I would of course be happy to help. The biggest problem with multiple database support is ensuring we're getting good testing with them. I do almost all my tests with SQLite and we don't have a buildbot/jenkins farm yet to test all combinations (and Python 2.6/2.7 :). I'm not a fan of javascript client-side rendering because of the generally poor performance, poor mobile compatibility, and lack of benefit for this kind of application. Not my pidgin, you'll need to talk to Toshio/Barry about that. AIUI, the current philosophy is that As a third party archiver, I can't tell you what to do, but my personal preference would be that JS is fine if it enhances the user experience, but that it be possible to perform basic archiver interactions without it. Cheers, -Barry ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
On Mar 27, 2012, at 08:53 AM, David Jeske wrote: Storm-ORM looks like it allows any primary key so that's good. From a quick glance it looks very similar to the Clearsilver-odb-orm in syntax and function. My primary beef with django-orm (last time I looked at it) was it requiring a unique-id primary key which is pretty poor for multi-row fetch performance design. Here's my take on ORMs. Early on mm3 went through phases with both SQLObject and SQLAlchemy, which I'd say are the other two most popular Python ORMs that aren't specifically tied to a CMS. It's been a few years now, but both had lots of problems that manifested as unacceptably slow velocity for making changes to Mailman. What I like about Storm is that it's so lightweight. When something goes wrong (e.g. not being able to store flufl.enum.Enums in the database or UTC timezone-aware datetimes into SQLite), I've always found it pretty trivial to dig into Storm, find the problem, maybe fix it, or at least easily work around it. The well-written, thin-ish layer of Python code makes up for the lack of comprehensive documentation. UTSL. Storm is also heavily used within Canonical so I know who to beg with or scream at if I need something :). Seriously though, it powers Launchpad among other tech, so it's pretty well battle-tested. But I'm the first to admit I'm not a database expert, so while I can make stuff work in Storm, it may not be optimal. Two examples: I've gotten donations to make some of the queries more efficient and I'll always welcome those types of contributions. Second, I know that some of the constraints I have expressed in the SQLite schema can't be expressed in a more strict database such as PostgreSQL. It would be nice for someone to dig in and fix those. The other thing to keep in mind is that within the core, I've tried very hard to again, use formal interfaces between components so that it could almost not matter. This hasn't been field tested so maybe I failed, but the idea is that you could potentially rip out the model and database layers and provide alternative implementations of IDatabase and other interfaces, and the rest of Mailman should Just Work. I try to be very careful not to break those abstractions, and I think the Zope Component Architecture is the big win here. Clearsilver templating has a slightly more aggressive enforcement of 'no code in template' than django, because it has an explicit intermediate dataset called HDF (it doesn't allow access to arbitrary python lists, structs, classes, and methods). This allows it to work from any language (not just python), and makes the binding from the ORM a bit more explicit. However, it also means it's a c-module, and I can see the advantage to keeping c-compile dependencies to a minimum. While I want to keep the Mailman core as pure Python, and so far have seen no reason to think it can't be, we already have some C dependencies in our stack. Some of the libraries that Mailman depends on have C extensions that are either required or improve performance. This already means that to deploy a mm3 server you currently need a C compiler (and the development bit of Python, e.g. Python.h). Now, eventually you'll probably get this all from your distro, so won't have to worry about it, but that's the state of affairs atm. After I finish some quick cleanup and UI changes to CSLA I want to do, I'll take a look at MM2/3 and see if I have better questions once I'm informed. mm2 and mm3 will differ greatly in the parts you're going to care about. My suggestion is to forget about mm2 and concentrate on mm3. Cheers, -Barry ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
On Mar 26, 2012, at 05:11 PM, Toshio Kuratomi wrote: We'd love to have work done on the archier! I know that we're ditching pipermail entirely and that archivers are becoming separate from the core mailman. What I don't know is whether mailman3 will eventually have a standard archiver which lives outside of the core mailman but is recommended for installation along with it. Yeah, who knows? :) At PyCon a few weeks ago, I demoed hyperkitty which showed some of the things that a next generation archiver could do. hyperkitty is continuing to be developed. As I was talking about hyperkitty we touched briefly on what I think is one of the central conundrums about having only unofficial third party archivers: how to have a consistent programatic interface available over http. Grackle is another archiver for mailman that doesn't have the UI bells and whistles of hyperkitty but it does make an effort to expose a REST UI to the world. I think that's a beautiful thing. But I don't like that a site that wanted both would need to run two archivers that were saving mail into two sets of storage. Really excellent points Toshio. I think there's several ways we could go about this. * We could create a standard REST API that archivers were free but encouraged to implement. * We could expand the python API that archivers needed to expose and then implement the REST API inside of mailman Core (or a re-envisioned, lightweight Grackle). * We could promote a standard archiver much as we're going to promote posterius as the standard admin front-end and that archiver would provide a standard REST API that others could then emulate. And very good suggestions too. I'm not sure what the best thing to do right now, but I've long thought that the core needs a basic message store (as you'll see in the IMessageStore interface, which probably sucks ;). It's possible that the prototype archiver morphs into this thing and that as we expose the IMessageStore to the core's REST API, we'll start to define what we need from an archiver. I agree that having such an API in front of the archiver is a truly beautiful thing. (One thing I notice about grackle now is that it's AGPL... that's going to be unpleasant for some sites to run. Perhaps we can change that or get some changes added to the AGPL.) It may be difficult to change. Canonical's default license on FLOSS it releases is AGPL. If we can make a good case for wanting something different, it may be possible to change. On a separate track, at Pycon you made a persuasive argument about the AGPL's flaws, and since that's an official FSF license, it would be good if someone would explore addressing those problems with them. That probably won't be me wink. -Barry signature.asc Description: PGP signature ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
On Mar 26, 2012, at 06:07 PM, David Jeske wrote: I highly recommend reconsidering this and including a standard archiver with mailman. If the number of sites that use pipermail is any indication, I think failing to include something will basically mean lots of lists without any archives. I think you're right. People want a turnkey solution - download one thing, run one command, and you've got everything you need. Of course, with mm2, you *don't* though because Pipermail never provided searching for example. There are lots of questions to ask about how we in the Mailman project would provide that kind of turnkey solution that includes our best-of-breed services. OTOH, it's also powerful to provide choice and not require any specific bits. My gut tells me that we're on the right track with Postorius, but that we're pretty far away from being able to bless an archiver right now. It's definitely not something I want to hold up the 3.0 final release for, so we have to find the right way to manage our users expectations. Understand though that we're constrained in other ways when we start thinking about bundling or officially blessing certain components. Mailman is a GNU project and the core's copyright has been owned by the FSF for over a decade. We require copyright assignments to the FSF. Postorius falls under the same administrative structure, so there's no problem calling it the official GNU Mailman web ui. A bless, bundled archiver will have to probably adhere to the same constraints, or at least we'd have to have that conversation with the FSF about it. But that absolutely shouldn't stop any other third party archiver from being Mailman 3 compatible. As I said, like the Python standard library, it's both a blessing and a curse. :) As for the features it doesn't have from your list: Editing would be easy to add because it's sqlite (deciding on the auth system is probably more of an issue than the editing). Anti-Crawl code is really an issue of configuration for cheap in-memory state-management. NNTP is well. that would be a big job that I doubt will be bitten off by something as small as a list archiver. Why can't we kill off Gmane while we're killing off Gmail, and *Groups? :). What is the REST UI used by? CSLA supports RSS. When it comes to a more involved REST UI, what software would be hitting it? I don't think I'll understand your other API/REST points until I see an answer to this. I'm a list owner and someone requests that a post containing private information be taken down. As a drive-by archive user, I want to request that a message get sent to me so that I can reply to it in my mail reader as if I had received the original. I run a question/answer forum that gateways a list, and I want to +1 really helpful messages, or give some extra kudos to really helpful users. -Barry ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] RELEASED: GNU Mailman 3.0 beta 1 andPostorius 1.0 alpha 1
On Wed, Mar 28, 2012 at 9:20 AM, Jeff Breidenbach j...@jab.org wrote: archivers are configured site-wide, so there's almost nothing to expose in the web-ui. I'm worried about confusion. Indeed. I think Barry misspoke here. But remember, we're barely out of alpha test, and we don't actually have a standard archiver, just a simple handler to support further development. The last thing we want is for a list to be accidentally archived contrary to the list administrator's wish. It sounds scary to me not to have any indication whatsoever in the web interface. However, this is unavoidable, since in an open-subscription or confirm-only list anybody can request Gmane or mail-archive.com to subscribe, and of course users will save their mail in many cases. I think it's arguable that it's best to archive everything (eg, so you can demonstrate that a particular piece of spam did or did not come through your list), and lock it down at the archive-viewer level. Along similar lines, there seems opportunity for confusion if there are two independent mechanisms for archival; site wide configuration and also manually subscribing an archival subscriber Personally, I think only the latter should be provided by default since it can't easily be prevented unless the list owner vets all subscribers very carefully (including having access to their .forwards!) Of course individual sites or lists can provide their own archiving handlers, and for the purpose of developing archivers and archive browsers offline, it may be useful to have a simple archive-to-maildir handler. (Note that the usual MTA/MDA suites can be used to deliver to maildirs, so for a list that's actually online, to get a local archive you just set up a $LIST-archiver virtual user delivering to maildir. AFAICS, this will be sufficiently efficient.) ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
