Re: [Mailman-Developers] Integrating HyperKitty with Mailman3
On Monday, April 9, 2012, Barry Warsaw wrote: So I can turn this question around and ask, what's the best way to get messages into ClearSilver? Drop it into a Maildir, so that the version of Python we use for CSLA isn't locked to the version of Python for MM3. Perhaps the MM3 CSLA handler can do things like manage the maildir, start/stop CSLA, etc. I'll take a look. Python imports are not version-dependent (like C-shlibs are), so it seems dubious to expect an external archiver to necessarily be compatible with the same version of python that MM3 is. I know I've run into this problem in the past, especially because of how much the python MIME message classes changed over each python release (though hopefully they are more stable now) Well, until email6 is released, Mailman 3 is ported to Python 3, and we can all (finally) do email the right way in Python. :) I'll believe Python will stabilize the email APIs when I see it. :) A decade ago we resorted to copying all the email handling classes out of the python dist and into our own code so we could keep using the ones that we depended on while upgrading python. ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Integrating HyperKitty with Mailman3
Are you expecting this direct python configuration import to actually be an archiver, or simply to be a configuration shim to get data to an archiver? Python imports are not version-dependent (like C-shlibs are), so it seems dubious to expect an external archiver to necessarily be compatible with the same version of python that MM3 is. I know I've run into this problem in the past, especially because of how much the python MIME message classes changed over each python release (though hopefully they are more stable now) On Apr 8, 2012 9:39 AM, Barry Warsaw ba...@list.org wrote: -snip snip- [archiver.hyperkitty] class: python.path.to.hyperkitty.HyperKitty -snip snip- Of course, you'd probably want to `enable` it too. One tricky thing here is that the `class` value names a Python dotted-module path, so the class must be importable. Ensuring that the hyperkitty module (and this is just a suggestion, YMMV) is importable by the core engine may not be fully baked. For now, just set $PYTHONPATH. ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] From the creation of a ThreadID
On Apr 6, 2012 10:49 AM, Toshio Kuratomi a.bad...@gmail.com wrote: 1) don't publish thread-ids, but just message-ids... for example, a thread URL could be allowed to reference the message-id of 'any' message in the thread They could then include more than one message-id, making them resiliant to a lost messageid later. if some messageid are lost, hopefully a url someone is holding onto has another messageid that was not lost. This sounds good. So instead of relying on the first message-id of the thread we internally keep a mapping of all message-ids and stableurl hashes to either an internal message-id or a tree of messages in the thread. I think of this as keeping a mapping from rfc822 message-id to internal thread-id. I think you are using different words to say the same thing. When deleting messages, always retain the message-id and stableurl hashes for that message in the mapping. That way a url that pointed to the thread by that message-id will continue to function even though the message itself has been deleted. Perhaps I misunderstood. If you are going to have a record of the deletion (i.e. you can keep a deleted message around in some form), this problem becomes much easier. I thought this desire was to have stable urls and threads when you rebuild and a message is missing. Absolutly if there is a message 'deletion' feature, it should delete the message contents but leave a 'stub' that links the message-id and references/in-reply-to, so it can help hold the thread together during a rebuild. My memory is foggy, but I think we used a technique like this in Yahoo Groups. ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] From the creation of a ThreadID
On Apr 5, 2012 6:42 AM, Pierre-Yves Chibon pin...@pingoured.fr wrote: So I changed to use the Message-ID of the first email of the Thread as ThreadID. Problem is of course, if the admin removes the first email of a thread for x or y reasons, then when reloading the archives (for z or a reasons), we will loose the ThreadID and actually, the integrity of the Thread (each reply to the first email will be split into their own thread). Would anyone have an idea on how to generate a stable and delete/reload proof ThreadID? I believe deletion proof (i.e. stable thread-ids in the case of arbitrary deletions) may be provably not possible. If you really want to be resiliant to arbitrary deletions/reloads, I think your solution is ultimately going to involve referencing more than one message in thread URLs.. For example, here is a scheme where 'messages in the thread name the thread': 1) don't publish thread-ids, but just message-ids... for example, a thread URL could be allowed to reference the message-id of 'any' message in the thread They could then include more than one message-id, making them resiliant to a lost messageid later. if some messageid are lost, hopefully a url someone is holding onto has another messageid that was not lost. As for how to pick the message-ids, paged display could include a messageid for a message on the page, in addition to the 'first' messageid of the thread. 2) create an 'internal only threadid' which you use to correlate messages together into a thread. (don't show this to anyone) you could generate this as a GUID, Hash, or the message-id of the message..doesn't matter, since nobody will see it... 3) when indexing messages, search in both directions (references/in-reply-to - messageid, and vice-versa) to find out if the message belongs in a thread.. if it does, then adopt the 'internal thread id'.. if you find two different threadids in the two directions, then rewrite/combine into a single internal-thread-id - urls can be somewhat resiliant of deleted/missing messages within a thread... and completely resilient to changes in other threads - threads can be manually edited and merged/split after the fact, with some level of success - could be designed to 'break down' threads that get too big, again with minimal damage, and some url compatibility ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] mailman / archive-ui / licensing questions
This thread is slowing down my coding! :)(it's been really helpful though all, thanks for the many perspectives!) On Tue, Apr 3, 2012 at 11:19 PM, Terri Oda te...@zone12.com wrote: It occurs to me that it's perfectly reasonable to assume that people who *package* mailman for different distributions may choose different recommended/required archive software, since they can (and with the license hassle likely should)) be separate packages. So what works for the FSF, what works for us as a dev team, and what works for the distributions may actually be different things. I agree I'm coming around to the sensibility of possibly not including an archiver with MM3, just so long as there actually ARE solid and working archivers that plug right in with nothing more than an apt-get (or equiv). It's just as fine if the distribution maintainers pick which one to include, and this gets around all this FSF/GPL/whatsit stuff... without bascally getting a pipermail default. I still think it's dangerous for people landing on Mailman's website and downloading source.. So no matter what, having David release his work is potentially going to lead to people getting it as a default, somewhere along the line, if he's got a great solution available. I know this thread is long and in pieces, but just to clarify, my code is already released and has been S-BSD for **ten** years. The UI is a little dated, so I'm cleaning up both the UI and the code right now, but I just want folks to know the code is already out there.. http://www.clearsilver.net/archive/ http://dj1.willowmail.com/csla/Mailman-Developers ...this discussion is all just about whether mailman wants to bundle (or reference) near-future updates to this stuff. I was hoping that rather than create my own separate OSS-y website and such for it, I could just hang out here and roll it into Mailman-land. You guys have done great work. If this GPL/S-BSD issue turns out to be a blocker, then I'll just make my own site and maintain (my version) there because I want to release my code S-BSD. Also, there will be *zero* ill-will if you folks want to wrap it up in a GPL license and stick it into mailman... i just won't be maintaining that, or assigning copyright, and any patches I make will be into my S-BSD tree. Perhaps not ideal, but still seems a better outcome than pipermail. ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] mailman / archive-ui / licensing questions
On Apr 3, 2012 8:14 PM, Bob Puff b...@nleaudio.com wrote: I think it would be a mistake to bundle any archiver with mailman3. Listing the available archiver options and their features and shortcomings would be a better way to go. -1 I think the majority of MM users will be simply using the RPM that comes with their distro, and there is a real benefit to stuff working right out of the box. This includes the Archiving functions. Its great to have options, and giving a list of possible alternatives for users is excellent, but I think releasing MM 3 without -any- archiver is a down-grade from the current MM 2.x. I agree. If MM2 and pipermail is any indication of how often admins just 'leave the defaults', then bunding no archive interface with MM3 would mean most mailing lists would have no archive. I'd personally like to see a better archiver rolled into an MM2 point release, as well as upcoming MM3 development. (I understand pipermail URL compat would be nice in that case). ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] mailman / archive-ui / licensing questions
On Apr 3, 2012 11:58 AM, Toshio Kuratomi a.bad...@gmail.com wrote: The question is would you BUNDLE another archiver even if the licenses don't match? Where could your archiver fit into that sequence of impressions? I'm not entirely sure. I think that it probably couldn't be bundled into the same tarball with mailman core due to mailman being an FSF project. I'm just going to charge down the path I was on and finish up something that's a great drop in for MM2/MM3. I'll even try to add some pipermail URL compatibility. It'll be S-BSD, so (if you like it) the MM devs and the FSF can wrestle with issues of whether you want to bundle it as is, put a rubber GPL stamp on it, or just point to it like you would any other archiver. I honestly expected to have an updated UI to show by now. I've been busy with some code-restructuring, and an unbelievable amount of life-stuff came across my bow in the past week. It shouldn't be too long now. But pointing to it from list.org or blessing it as the standard archiver for mailman3 is probably something that could be discussed by the core devs and yourself. I'm a bit scared of a world where MM3 does not include any archiver. If pipermail popularity is any indication of how often admins 'stick with the bundled defaults', we could have an unreasonable number of MM3 lists with no archives at all. Obviously the team is free to bless any archiver it wants, mine or others. Also, I'm certainly NOT trying to get anyone to agree to bless an archiver before they've even seen it working and kicking butt. I was just trying to understand the many issues as I'm cleaning up my code and trying to find it a home with a bit more utility. I think I have a great idea from all the disussions here.. THANKS! ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] mailman / archive-ui / licensing questions
On Apr 2, 2012 3:07 PM, Terri Oda te...@zone12.com wrote: This agrees with my view of the situation as well. Which leads to the question, is the above approach interesting/viable for Mailman-team? (assuming the code does something awesome that people want) If the question is just would you like another archiver even if the licenses don't match? then I believe the answer is yes. The question i would you BUNDLE another archiver even if the licenses don't match? My archiver has been available for download (like many others) for ten years. All these sites are still running a limping pipermail archive, because it's bundled. I want to get Mailman a better bundled archive. But... since you arrived around the same time GSoC started, I should ask whether you were hoping to do this as a GSoC project? Perhaps it would make things more clear if I expledin why I'm here... I'm not a student. I've been working in software for 15 years, programming for almost 30 (since I was 9). I wrote large portions of eGroups / Yahoo Groups / Google Groups. I'm a successful post-Google entrepreneur. Since leaving Google I've been angel investing mostly in tech stuff (see my Angel List).. I've been donating notable chunks of money and time to open source projects (with my blender donations working out the best so far). Given my history, and the fact that I keep wanting to tear my hear out reading mailing list archives in pipermail, I thought I'd give you folks an archiver that would be nice. HOWEVER, I personally will not write GPL code. I might submit a tiny patch or bugfix, but I'm simply opposed to restrictions on how someone uses something that I'm trying to donate to the software community. (i.e. you're never going to turn me into a mailman developer, the best you'd get is me writing my own mailman-ish and releassing it under S-BSD.. if you want that, let me know) ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
- Stable URLs, RFC 5064 + X-Message-ID-Hash. See the above links. If you can implement the `IArchiver.permalink()` method and ensure that even if completely wiped and regenerated from the underlying raw messages, your URLs will remain stable, I think you will have won. :) This is really trivial to implement. HOWEVER, it also has two notable problems. First, if the sender-selected-messageID is trusted to make a unique-id and permalink, then you are trusting senders, which IMO is a bad idea. What happens when senders intentionally duplicate a message-id? It's pretty hard to do anything smart when the listprocessor doesn't track messageids. Second, when message-id uniqueness breaks, it's often not an isolated instance, but a pattern of software doing bad things. I prefer generating a trustably unique message IDs, even if those are only used internally. I've used partial-content hashes in the past, but they have their tradeoffs. They are hard to abuse, since matching a content hash tends to mean supplying the same content, which kinda prevents abuse. However, if the content is being changed at all (intentionally or not), they breakdown. --- I like your ideas about plug-in text-match-formatters. We have something like that in the willowmail system, and it's pretty useful. (auto-linking fedex tracking numbers in email is one example) - Merging of forums, archives, newsgroups, and IMAP. You like to bite off the big ones eh? NNTP, then IMAP. Does anyone even use fat-mail-clients anymore? Even the IMAP clients I've used in recent years have been web-based. which is kinda already unified with web-based archives since they are both in the browser. :) Seriously though, what is the goal here? I suppose I've had one thought over the years, which is that sometimes it seems weird that I have all these mailing lists delivering into my mail client. In theory my mail client could be 'faking' the whole thing with NNTP to a known archive location. However, that only works when online. If you have to do delivery to my client, who cares if it's SMTP/NNTP/IMAP? I say pick the simplest one (SMTP), which works with all mail clients, no changes, and allows you to keep your messages when the archive goes down. As for mail-client integration features... I like mail-client push-button unsubscribe. I like the idea of making it trivial for mail clients to recognize mailing lists posts and auto-configure filter-to-folder for them. Neither of these require NNTP/IMAP in the archive though. ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Developers] mailman / archive-ui / licensing questions
Thanks for the super-detailed replies... I'm separating these discussions, so here I have some questions about licensing and bundling.. So in some sense, CSLA needn't become *the* Mailman archiver, but it should definitely be *a* Mailman archiver. Then you can make all the engineering and design decisions you prefer, but with the confidence that it will Just Work with Mailman 3. Sure but this isn't why I'm here. CSLA is already *a* mailman archiver..I think we first released it in 2004. A few of us ex-egroups folks hacked it out because we used it for private projects. We open-sourced it so we could use it across organizational boundaries and because we were happy to give it away to anyone who wanted it. We're just all primarily focused on startup and commercial endeavors, so we havn't done much to package and popularize it. Right now I'm in between entrepreneural endeavors and spending some time 'giving back' and coding/donating-to/helping several open-source projects. As I engage with these projects, all of them are using Mailman, which is fantastic. However, nearly all of them are also using pipermail, which is not so great. They are using it because it's the default, so it was easy. I started to talk to one of them about installing CSLA (or MHonArc, or anything really), and realized I should see if you folks are interested in a great bundled archiver, to fix the problem at the source. I'm not particularly interested in promoting or maintaining an open-source project around this, so if you folks don't want a shiny new (S-BSD licensed) archiver to bundle, I'll probably just fix a few things, bump the CSLA archiver to 0.3 and move on. --- I admit that even with a pretty good knowledge of these many licenses, I'm not familiar with the intracacies of FSF copyright assignment and non-GPL free licenses. The ClearsilverArchiver code (written by me and two others) is released under the Simplified BSD license and totally free. It's important to me that any code I release be similarly free-and-unrestricted (i.e. BSD/Python/Artistic/PublicDomain), not free under certain conditions (i.e. GPL/LGPL). It's not possible to assert GPL restrictions on totally-free code, because it's already totally free. FSF says S-BSD is GPL-Compatible, which I believe means they are saying they have no problem with GPL code depending on and being combined with (i.e. linked with) S-BSD code, because the S-BSD code is fully open-source and does not put restrictions on the use of the GPL code. It's also my understanding that the primary reason for FSF copyright assignment is to provide a coherent entity to enforce the terms of the GPL by challenging violators who don't redistribute source something which is not necessary for S-BSD. (Though I suppose they could enforce that folks include the S-BSD copyright notices.) So I guess this all drives to the following question: Is Mailman-team is interested in having a better built-in archiver that is included in the distribution, but licensed under the less-restrictive S-BSD terms? Sorry for the length. This license stuff can be complicated. -- On a weirdly unrelated coincidence, thanks for smtpd.py. I just hacked it into smtp-to-maildir for a private hosted webmail installation. We were migrating code/data to some new machines and smtpd.py seemed simpler than fighting with qmail-installation or configuring postfix to accept everything (something it doesn't seem designed to do). But that absolutely shouldn't stop any other third party archiver from being Mailman 3 compatible. As I said, like the Python standard library, it's both a blessing and a curse. :) As for the features it doesn't have from your list: Editing would be easy to add because it's sqlite (deciding on the auth system is probably more of an issue than the editing). Anti-Crawl code is really an issue of configuration for cheap in-memory state-management. NNTP is well. that would be a big job that I doubt will be bitten off by something as small as a list archiver. Why can't we kill off Gmane while we're killing off Gmail, and *Groups? :). What is the REST UI used by? CSLA supports RSS. When it comes to a more involved REST UI, what software would be hitting it? I don't think I'll understand your other API/REST points until I see an answer to this. I'm a list owner and someone requests that a post containing private information be taken down. As a drive-by archive user, I want to request that a message get sent to me so that I can reply to it in my mail reader as if I had received the original. I run a question/answer forum that gateways a list, and I want to +1 really helpful messages, or give some extra kudos to really helpful users. -Barry ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives:
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
I brought up a quick CSLA injest of some mailman-dev posts to show off some basic features. I'm having a little trouble with the swish-e code, so the features that depend on text-indexing arn't working at the moment (search, author search). The current UI uses user-cookies to choose between a 2-pane frames mode and a noframes mode. I set the frames-mode as the default, which is inspired by the old google-groups usenet multi-message tree. Use prefs to check out the noframes mode, which is more like pipermail. Take a peek more for general capabilities than UI details (unless you see something you really like). I'm planning to change the UI to be more like the gmail-esq conversation style UIs I've been using since then... (where most/all of a thread's messages are rendered into a single page) I like the outgoing google-groups UI pretty well (not the new AJAXy one). 1) basic home for a list with bymonth and top-author metrics, RSS. Extended top-author page. http://dj1.willowmail.com/csla/Mailman-Developers http://dj1.willowmail.com/csla/Mailman-Developers/top_authors?date=2012-03 2) Prefs, including timezone, read-status key, browse-style.. (not sure if the read state is working on my install) http://dj1.willowmail.com/csla/Mailman-Developers/prefs 3) threadlist for the month (frames or noframes in prefs) http://dj1.willowmail.com/csla/Mailman-Developers/browse/month/2012-03/107 4) single-thread-list display (frames or noframes in prefs) http://dj1.willowmail.com/csla/Mailman-Developers/threads?start=1 5) here is a good example of a thread with redundant text eliding http://dj1.willowmail.com/csla/Mailman-Developers/browse_frm/thread/4/221 ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
On Mar 27, 2012 5:32 AM, Stephen J. Turnbull step...@xemacs.org wrote: If the archiver/web UI is going to be distributed *with* Mailman, Barry would probably prefer Storm + Django because that's what Mailman/Protorius (core and admin web UI, resp.) are using. But I imagine that's negotiable as long as everything is free software. I'll take a look at the Mailman2/3 code and see what the pattern looks like. Storm-ORM looks like it allows any primary key so that's good. From a quick glance it looks very similar to the Clearsilver-odb-orm in syntax and function. My primary beef with django-orm (last time I looked at it) was it requiring a unique-id primary key which is pretty poor for multi-row fetch performance design. Clearsilver templating has a slightly more aggressive enforcement of 'no code in template' than django, because it has an explicit intermediate dataset called HDF (it doesn't allow access to arbitrary python lists, structs, classes, and methods). This allows it to work from any language (not just python), and makes the binding from the ORM a bit more explicit. However, it also means it's a c-module, and I can see the advantage to keeping c-compile dependencies to a minimum. AIUI, the current philosophy is that Using SMTP for an included archiver would require it be a long running server instead of merely a handler. Sure, but you're already running a sufficient server, namely your MTA. Gotcha. I thought you were saying Mailman wanted the archiver to be a long-running server accepting SMTP/LMTP on a socket. I normally manage the archive queue with the MTA, and have it launch the archive-injestor for each message.. (or for higher volume applications, long-running injester on a Maildir).. which sounds like what you're saying.. After I finish some quick cleanup and UI changes to CSLA I want to do, I'll take a look at MM2/3 and see if I have better questions once I'm informed. Thanks for the great overview. ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
I'm writing to find out the state of and philosophy surrounding pipermail in mailman, to see if there is a productive way to provide some code/development-time to that part of mailman. I know there are several decent third-party archivers out there, but many of the mailing list archives I browse regularly are using pipermail because it's the mailman default one less thing to install, administer, and upgrade. Unfortunately, pipermail doesn't do a good job formatting messages for html.. (messages with no line-breaks are the most annoying problem I regularly run into) I've written code for this a number of times (eGroups, Yahoo Groups, Google Groups). I also released an open-source python/clearsilver/sqlite based archiver with redundant text-eliding, a few different thread views, and search... ( http://www.clearsilver.net/archive/ ) which is hardly used both because I don't try to popularize it, and because many sites just leave the default (pipermail). If there is some code (and time) I can contribute to mailman/pipermail, I'd like to do so. I'm writing this message to take a temperature and find out what, if any, contributions would be appreciated the mailman development team. I could imagine answers like: a) pipermail is fine... if you want to fix a bug or two submit a patch, but we don't want to improve it b) we're ditching pipermail entirely... in the future sites will have to choose an install an external archiver c) we'd love pipermail to be improved... but we still want it to be simple, static-html, and dependency free d) we'd love a dynamic-ui replacement for pipermail... as long as it uses the same cgi/templating model as mailman ui Thoughts? Is there any help I can offer up here? ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
On Mon, Mar 26, 2012 at 5:11 PM, Toshio Kuratomi a.bad...@gmail.com wrote: We'd love to have work done on the archier! I know that we're ditching pipermail entirely and that archivers are becoming separate from the core mailman. What I don't know is whether mailman3 will eventually have a standard archiver which lives outside of the core mailman but is recommended for installation along with it. I see.. that sounds like option-b. I highly recommend reconsidering this and including a standard archiver with mailman. If the number of sites that use pipermail is any indication, I think failing to include something will basically mean lots of lists without any archives. At PyCon a few weeks ago, I demoed hyperkitty which showed some of the things that a next generation archiver could do. I recommend you take a closer look at ClearsilverListArchivehttp://www.clearsilver.net/archive/, it's written in Python, Clearsilver, SQLite.. is real open-source (BSD License), and hits most of the features on your ModernArchiving wishlist plus a bunch you didn't (author pages, redundant text elimination, cookie preferences. As for the features it doesn't have from your list: Editing would be easy to add because it's sqlite (deciding on the auth system is probably more of an issue than the editing). Anti-Crawl code is really an issue of configuration for cheap in-memory state-management. NNTP is well. that would be a big job that I doubt will be bitten off by something as small as a list archiver. Sadly I can't point to any lists using it at the moment, because, well, it's hidden under a rock. I'll injest an archive of the mailman list so you can take it for a spin. As I was talking about hyperkitty we touched briefly on what I think is one of the central conundrums about having only unofficial third party archivers: how to have a consistent programatic interface available over http. What is the REST UI used by? CSLA supports RSS. When it comes to a more involved REST UI, what software would be hitting it? I don't think I'll understand your other API/REST points until I see an answer to this. Grackle is another archiver for mailman that doesn't have the UI bells and whistles of hyperkitty but it does make an effort to expose a REST UI to the world. I think that's a beautiful thing. But I don't like that a site that wanted both would need to run two archivers that were saving mail into two sets of storage. I think here you are entering into a catch-22. If you have a single storage system, then you have a single storage schema, which means you have a single set of things you can do fast and most other things become impractical (because they would require synchronizing state). I'm quite sure, for example, that the Grackle schema is not the same as the CSLA schema, and that many CSLA features would be impossible with the Grackle REST API. (short of just using it to suck everything down, but then you're just duplicating). Why is message duplication an issue? ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)
So if you have already developed some stuff, I certainly would love to see you put it on the table as a candidate for the Mailman 3 default archive web UI. Yes. My CSLA code is on the table for MM2 or 3. It's python, and it's BSD. More than that, if you folks want it, I'm happy to retool/rewrite it as needed to make it suitable. (it has lots of great features, but i havn't touched it in a long time and it could be better) I'll put up a demo of the mailman list in CSLA tomorrow so folks can check it out. I also have some experience with this space, and by that I mean mountainous-experience. I coded/managed/architected several versions of eGroups/Yahoo Groups and Google Groups, including the listprocessor, schema design, and a few different python mail-to-html formatters. I also wrote my own personal gmail-like web-mailer in python, which someday I'll get around to open-sourcing. Can you share something about dependency philosophy (besides licensing) in Mailman? Currently CSLA uses Clearsilver templating (BSD c-module) and sqlite (public domain) storage.. both of which can be included directly as source to make installation easy, but they are both c-modules. I'm quite fond of the Clearsilver orm-to-templates toolchain, which is in use for lots of big sites. However, I also recognize it's not that popular in the python commmunity, mostly because we don't promote it, but also because the template processor c-module (for performance) instead of pure python. If it's necessary I could convert CSLA to use django templates, though they have a nasty design/performance problem with the orm that I think would prevent me from using it (unless I fixed it). I'm not a fan of javascript client-side rendering because of the generally poor performance, poor mobile compatibility, and lack of benefit for this kind of application. Doing a *better* job of formatting HTML (than pipermail) is easy. Doing a *good* job is going to be relatively hard, though. Isn't that the truth. Back at Egroups there was a time where it seems like we used to tweak the formatter almost daily. Deciding when preformatting is necessary vs harmful is a real pain. My CSLA python code has a fairly stable msg-to-html formatter that I've been happy with for a long time. It does very aggressive automatic repetitive text eliding for thread-display, so you can read many messages in a thread in a single page without massive quote pileup. View-source is provided for the unfortunate case where it does something bad. AIUI, the current philosophy is that (1) the communication from Mailman to the archivers will be via LMTP/SMTP, including a Mailman-specific header to identify the message's permalink (currently the SHA1 hash of the message ID in BASE32 format, IIRC); and Using SMTP for an included archiver would require it be a long running server instead of merely a handler. Are we talking about third-party-site archivers here? (2) the summary, search, and retrieval UI will be a separate application from Mailman per se, which will communicate with Mailman via the REST API for authentication and user configuration purposes. That makes sense... CSLA doesn't currently have any concept server-auth. The only stateful features it has are view-preferences and read-state, neither of which are important enough to require a password. It uses a password-less system which uses cookies for prefs and a 'read state userid' which a user can manually set if they want. I like it, because it doesn't require login to get some basic browsing prefs and features. Hooking up an auth system would be necessary for some of the editing ideas in the document I read, or to allow online posting. ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] templating, search, and stuff...
On Fri, Apr 05, 2002 at 10:19:09AM -0500, Barry A. Warsaw wrote: DJ == David Jeske [EMAIL PROTECTED] writes: DJ Also, in fishing through the TODO list, I notice that adding DJ some kind of templating system is still on there. Improving the templating system (for MM3) is more than just picking one and going with it. The real issue is that folks want to put all kinds of web skins on Mailman, to integrate them into their own way of managing their sites. First off, it's no skin off my back, I don't run or administer mailman, I just interact with them on several mailing lists so I'm happy to help out. However, if you look at Clearsilver, you'll find that it was designed to handle exactly this kind of skinning. We used it for cobranding and OEM deals back at eGroups. While integrating into Yahoo, we ran the old eGroups version of the UI in Japanese for six months while running every other language in the new Yahoo UI, on the same codebase. It really excels at this. The majority of the skinning of the UI to the Yahoo look and feel took one engineer less than 10 days. Some people don't care about the integration, and for them we can pick a Good One and use that as the default. But they've got to be able to chuck it, design their own web interface, and integrate it easily into Mailman. For that we need APIs, a clean architecture, and component-like system. If you read my paper on dataset driven templating, you'll find that this is exactly what Clearsilver provides, in an extremely fast and easy to use package. There are other template systems that do this also, but for integrating with Python, Clearsilver is the fastest and most mature way to go. http://www.clearsilver.net/docs/apples_to_oranges.hdf http://www.clearsilver.net/docs/man_templates.hdf The dataset (like a simpler form of XML) is your API. It sits between the application logic and the page template, and allows you to easily skin the application. There are some advantages to using XML/XSLT instead, namely because you can write an XML schema and typecheck the XML output. However, considering Mailman is written in Python, I figure you all understand the development ease gained by using an untyped system over a typed one for small-ish projects like this. Anyhow, I'm not here to get into an off-topic templating thread. I just saw that Mailman still didn't have templates and I figured I'd offer up some assistance. -- David Jeske (N9LCA) + http://www.chat.net/~jeske/ + [EMAIL PROTECTED] ___ Mailman-Developers mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-developers
