Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Pierre-Yves Chibon]

On Mon, 2012-03-26 at 17:11 -0700, Toshio Kuratomi wrote:
> Grackle is another archiver for mailman that doesn't
> have the UI bells and whistles of hyperkitty but it does make an
> effort to expose a REST UI to the world.  I think that's a beautiful
> thing.  

I started a small thing on hyperkitty there:
http://mm3test.fedoraproject.org/api/

Pierre
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[David Champion]

* On 29 Mar 2012, Stephen J. Turnbull wrote: 
> On Thu, Mar 29, 2012 at 4:04 PM, David Jeske  wrote:
> >> - Merging of forums, archives, newsgroups, and IMAP.
> >
> > You like to bite off the big ones eh? NNTP, then IMAP.
> >
> > Does anyone even use fat-mail-clients anymore?
> 
> Yes, and yes.  But I don't think anybody has really thought carefully
> about these desiderata yet, they're just frequently suggested on
> mailman-users and sometimes mailman-developers.

+1

For what it's worth: I bridged MM archives to IMAP once using UW imapd
and some scripts for proxying Mailman membership data into something
that imapd could use for access control.  When I presented it to this
list it was generally identified as something nobody (w|sh)ould ever
want.  I don't care particularly about the details of what happened in
2002 or whenever it was, but I conclude that the much bigger problem
than building this ship is choosing a destination.  It needs to start
with clear goals, because the wrong implementation is easy.

-- 
David Champion • d...@uchicago.edu • IT Services • University of Chicago
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Stephen J. Turnbull]

On Thu, Mar 29, 2012 at 4:04 PM, David Jeske  wrote:
>> - Merging of forums, archives, newsgroups, and IMAP.
>
> You like to bite off the big ones eh? NNTP, then IMAP.
>
> Does anyone even use fat-mail-clients anymore?

Yes, and yes.  But I don't think anybody has really thought carefully
about these desiderata yet, they're just frequently suggested on
mailman-users and sometimes mailman-developers.
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[David Jeske]

> - Stable URLs, RFC 5064 + X-Message-ID-Hash.
> See the above links.  If you can implement the
> `IArchiver.permalink()` method and ensure that even if
>  completely wiped and regenerated from the
> underlying raw messages, your URLs
>  will remain stable, I think you will have won. :)

This is really trivial to implement.

HOWEVER, it also has two notable problems.

First, if the sender-selected-messageID is trusted to make a unique-id and
permalink, then you are trusting senders, which IMO is a bad idea. What
happens when senders intentionally duplicate a message-id? It's pretty hard
to do anything smart when the listprocessor doesn't track messageids.

Second, when message-id uniqueness breaks, it's often not an isolated
instance, but a pattern of software doing bad things.

I prefer generating a trustably unique message IDs, even if those are only
used internally. I've used partial-content hashes in the past, but they
have their tradeoffs. They are hard to abuse, since matching a content hash
tends to mean supplying the same content, which kinda prevents abuse.
However, if the content is being changed at all (intentionally or not),
they breakdown.

---

I like your ideas about plug-in text-match-formatters. We have something
like that in the willowmail system, and it's pretty useful. (auto-linking
fedex tracking numbers in email is one example)

> - Merging of forums, archives, newsgroups, and IMAP.

You like to bite off the big ones eh? NNTP, then IMAP.

Does anyone even use fat-mail-clients anymore? Even the IMAP clients I've
used in recent years have been web-based. which is kinda already
unified with web-based archives since they are both in the browser. :)

Seriously though, what is the goal here?

I suppose I've had one thought over the years, which is that sometimes it
seems weird that I have all these mailing lists delivering into my mail
client. In theory my mail client could be 'faking' the whole thing with
NNTP to a known archive location.

However, that only works when online. If you have to do delivery to my
client, who cares if it's SMTP/NNTP/IMAP? I say pick the simplest one
(SMTP), which works with all mail clients, no changes, and allows you to
keep your messages when the archive goes down.

As for mail-client integration features... I like mail-client push-button
unsubscribe. I like the idea of making it trivial for mail clients to
recognize mailing lists posts and auto-configure filter-to-folder for them.
Neither of these require NNTP/IMAP in the archive though.
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[David Jeske]

On Mar 27, 2012 8:47 PM, "Barry Warsaw"  wrote:
>  but that we're pretty far away from being
> able to bless an archiver right now.

I just want to be clear, that I'm obviously not asking for anyone to bless
a specific archiver and roll it into the distribution.

I'm trying to get some idea what requirements an  archiver would need to
meet for you folks to jump for joy and roll it into the MM2 and MM3
distros  -- because my goal is to (with respect and fondness) exterminate
pipermail for the betterment of all. Along these lines, I've been thinking
that in addition to a dynamic rendering system (like CSLA), it might be
worth it to include a static-html generation model a bit more like
pipermail ... as some sites may prefer it and it's easy to do.

If there is a compatible set of goals I'll gladly offer / retool rewrite my
stuff to make folks happy.

.. if requirement #1 is GPL, then sadly I'll have to rethink my
extermination plan, because GPL is not a jeske-compatible-license. :)

> Why can't we kill off Gmane while we're killing off
> Gmail, and *Groups? :).

Is that the primary niche for gmane? Folks who want to read mailing lists
via NNTP clients? What is the motivation for NNTP support? I don't think it
would be too bad to support NNTP clients, it's NNTP sync that is more
involved. I just don't know why anyone would want to use an NNTP client
anymore.

As for killing off things, don't get me started. I'm very unhappy with the
new over-ajax Groups UI, and the fact that they seem to have broke the
faceted directory browser. The old html-with-javascript-tweaks is so much
faster and lighter on all browsers (especially mobile).

> >What is the REST UI used by? CSLA supports RSS. When it comes to a more
> >involved REST UI, what software would be hitting it? I don't think I'll
> >understand your other API/REST points until I see an answer to this.
>
> I'm a list owner and someone requests that a post containing private
> information be taken down.

I would expect the user/list-owner go to the
new-builtin-super-cool-mm-archive-web-ui to request/approve-takedown..
which would have CGIs and UI to directly modify it's data. The only REST
API I see is for validating the user-cookie, checking his auth-permissinos,
etc.

Is that what you are thinking, or am I missing some part of your thinking?
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Barry Warsaw]

On Mar 26, 2012, at 06:07 PM, David Jeske wrote:

>I highly recommend reconsidering this and including a standard archiver
>with mailman. If the number of sites that use pipermail is any indication,
>I think failing to include something will basically mean lots of lists
>without any archives.

I think you're right.  People want a turnkey solution - download one thing,
run one command, and you've got everything you need.  Of course, with mm2, you
*don't* though because Pipermail never provided searching for example.

There are lots of questions to ask about how we in the Mailman project would
provide that kind of turnkey solution that includes our best-of-breed
services.  OTOH, it's also powerful to provide choice and not require any
specific bits.

My gut tells me that we're on the right track with Postorius, but that we're
pretty far away from being able to bless an archiver right now.  It's
definitely not something I want to hold up the 3.0 final release for, so we
have to find the right way to manage our users expectations.

Understand though that we're constrained in other ways when we start thinking
about bundling or officially blessing certain components.  Mailman is a GNU
project and the core's copyright has been owned by the FSF for over a decade.
We require copyright assignments to the FSF.  Postorius falls under the same
administrative structure, so there's no problem calling it the official GNU
Mailman web ui.  A bless, bundled archiver will have to probably adhere to the
same constraints, or at least we'd have to have that conversation with the FSF
about it.

But that absolutely shouldn't stop any other third party archiver from being
Mailman 3 compatible.

As I said, like the Python standard library, it's both a blessing and a
curse. :)

>As for the features it doesn't have from your list: Editing would be easy
>to add because it's sqlite (deciding on the auth system is probably more of
>an issue than the editing). Anti-Crawl code is really an issue of
>configuration for cheap in-memory state-management. NNTP is well. that
>would be a big job that I doubt will be bitten off by something as "small"
>as a list archiver.

Why can't we kill off Gmane while we're killing off Gmail, and *Groups? :).

>What is the REST UI used by? CSLA supports RSS. When it comes to a more
>involved REST UI, what software would be hitting it? I don't think I'll
>understand your other API/REST points until I see an answer to this.

I'm a list owner and someone requests that a post containing private
information be taken down.

As a drive-by archive user, I want to request that a message get sent to me so
that I can reply to it in my mail reader as if I had received the original.

I run a question/answer forum that gateways a list, and I want to +1 really
helpful messages, or give some extra kudos to really helpful users.

-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Barry Warsaw]

On Mar 26, 2012, at 05:11 PM, Toshio Kuratomi wrote:

>We'd love to have work done on the archier!  I know that we're ditching
>pipermail entirely and that archivers are becoming separate from the core
>mailman.  What I don't know is whether mailman3 will eventually have
>a standard archiver which lives outside of the core mailman but is
>recommended for installation along with it.

Yeah, who knows? :)

>At PyCon a few weeks ago, I demoed hyperkitty which showed some of the
>things that a next generation archiver could do.  hyperkitty is continuing
>to be developed.  As I was talking about hyperkitty we touched briefly on
>what I think is one of the central conundrums about having only unofficial
>third party archivers:  how to have a consistent programatic interface
>available over http.  Grackle is another archiver for mailman that doesn't
>have the UI bells and whistles of hyperkitty but it does make an effort to
>expose a REST UI to the world.  I think that's a beautiful thing.  But
>I don't like that a site that wanted both would need to run two archivers
>that were saving mail into two sets of storage.

Really excellent points Toshio.

>I think there's several ways we could go about this.
>
>* We could create a standard REST API that archivers were free but
>  encouraged  to implement.
>* We could expand the python API that archivers needed to expose and then
>  implement the REST API inside of mailman Core (or a re-envisioned,
>  lightweight Grackle).
>* We could promote a standard archiver much as we're going to promote
>  posterius as the standard admin front-end and that archiver would provide
>  a standard REST API that others could then emulate.

And very good suggestions too.  I'm not sure what the best thing to do right
now, but I've long thought that the core needs a basic "message store" (as
you'll see in the IMessageStore interface, which probably sucks ;).  It's
possible that the prototype archiver morphs into this thing and that as we
expose the IMessageStore to the core's REST API, we'll start to define what we
need from an archiver.  I agree that having such an API in front of the
archiver is a truly beautiful thing.

>(One thing I notice about grackle now is that it's AGPL... that's going to
>be unpleasant for some sites to run.  Perhaps we can change that or get some
>changes added to the AGPL.)

It may be difficult to change.  Canonical's default license on FLOSS it
releases is AGPL.  If we can make a good case for wanting something different,
it may be possible to change.  On a separate track, at Pycon you made a
persuasive argument about the AGPL's flaws, and since that's an official FSF
license, it would be good if "someone" would explore addressing those problems
with them.  That probably won't be me .

-Barry


signature.asc
Description: PGP signature
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Barry Warsaw]

On Mar 27, 2012, at 08:53 AM, David Jeske wrote:

>Storm-ORM looks like it allows any primary key so that's good. From a quick
>glance it looks very similar to the Clearsilver-odb-orm in syntax and
>function. My primary beef with django-orm (last time I looked at it) was it
>requiring a unique-id primary key which is pretty poor for multi-row fetch
>performance design.

Here's my take on ORMs.

Early on mm3 went through phases with both SQLObject and SQLAlchemy, which I'd
say are the other two most popular Python ORMs that aren't specifically tied
to a CMS.  It's been a few years now, but both had lots of problems that
manifested as unacceptably slow velocity for making changes to Mailman.

What I like about Storm is that it's so lightweight.  When something goes
wrong (e.g. not being able to store flufl.enum.Enums in the database or UTC
timezone-aware datetimes into SQLite), I've always found it pretty trivial to
dig into Storm, find the problem, maybe fix it, or at least easily work around
it.  The well-written, thin-ish layer of Python code makes up for the lack of
comprehensive documentation.  UTSL.

Storm is also heavily used within Canonical so I know who to beg with or
scream at if I need something :).  Seriously though, it powers Launchpad among
other tech, so it's pretty well battle-tested.

But I'm the first to admit I'm not a database expert, so while I can make
stuff work in Storm, it may not be optimal.  Two examples: I've gotten
donations to make some of the queries more efficient and I'll always welcome
those types of contributions.  Second, I know that some of the constraints I
have expressed in the SQLite schema can't be expressed in a more strict
database such as PostgreSQL.  It would be nice for someone to dig in and fix
those.

The other thing to keep in mind is that within the core, I've tried very hard
to again, use formal interfaces between components so that it could almost not
matter.  This hasn't been field tested so maybe I failed, but the idea is that
you could potentially rip out the model and database layers and provide
alternative implementations of IDatabase and other interfaces, and the rest of
Mailman should Just Work.  I try to be very careful not to break those
abstractions, and I think the Zope Component Architecture is the big win
here.

>Clearsilver templating has a slightly more aggressive enforcement of 'no
>code in template' than django, because it has an explicit intermediate
>dataset called HDF (it doesn't allow access to arbitrary python lists,
>structs, classes, and methods). This allows it to work from any language
>(not just python), and makes the binding from the ORM a bit more explicit.
>However, it also means it's a c-module, and I can see the advantage to
>keeping c-compile dependencies to a minimum.

While I want to keep the Mailman core as pure Python, and so far have seen no
reason to think it can't be, we already have some C dependencies in our
stack.  Some of the libraries that Mailman depends on have C extensions that
are either required or improve performance.  This already means that to deploy
a mm3 server you currently need a C compiler (and the "development" bit of
Python, e.g. Python.h).

Now, eventually you'll probably get this all from your distro, so won't have
to worry about it, but that's the state of affairs atm.

>After I finish some quick cleanup and UI changes to CSLA I want to do, I'll
>take a look at MM2/3 and see if I have better questions once I'm informed.

mm2 and mm3 will differ greatly in the parts you're going to care about.  My
suggestion is to forget about mm2 and concentrate on mm3.

Cheers,
-Barry

___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Barry Warsaw]

On Mar 27, 2012, at 09:32 PM, Stephen J. Turnbull wrote:

>Well, for the official poop you'll have to wait for Barry, but AFAICS
>archivers aren't restricted to Storm + RESTish (which is what Mailman itself
>uses) because they're separate applications.  If the archiver/web UI is going
>to be distributed *with* Mailman, Barry would probably prefer Storm + Django
>because that's what Mailman/Protorius (core and admin web UI, resp.) are
>using.  But I imagine that's negotiable as long as everything is free
>software.
>
>I don't think the database backend much matters, but Mailman and Django
>are both happy to use sqlite, I believe.

What Stephen says above is pretty accurate.  See also my previous follow ups
in this thread.

The core engine uses Storm as its ORM, which I think officially supports
SQLite, PostgreSQL, and MySQL, though it's probably compatible with others out
of the box or easily so.

Mailman itself currently only supports SQLite out of the box because that
comes with Python, and we have donated support for PostgreSQL.  I'd love for
someone to donate MySQL support; I think it would be easy for someone with
MySQL experience, there isn't much that would need to be added to Mailman,
there are good examples now, and I would of course be happy to help.

The biggest problem with multiple database support is ensuring we're getting
good testing with them.  I do almost all my tests with SQLite and we don't
have a buildbot/jenkins farm yet to test all combinations (and Python 2.6/2.7
:).

>> I'm not a fan of javascript client-side rendering because of the generally
>> poor performance, poor mobile compatibility, and lack of benefit for this
>> kind of application.
>
>Not my pidgin, you'll need to talk to Toshio/Barry about that.
>>> AIUI, the current philosophy is that

As a third party archiver, I can't tell you what to do, but my personal
preference would be that JS is fine if it enhances the user experience, but
that it be possible to perform basic archiver interactions without it.

Cheers,
-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Barry Warsaw]

On Mar 27, 2012, at 01:29 PM, Terri Oda wrote:

>So Postorius (the webUI) has a sketch of an auth system using BrowserID at
>the moment.  BrowserID is convenient 'cause it proves you have ownership of a
>given email address, but we should have OpenID working soon once we've got
>the code to confirm that a given OpenID can be associated with an email
>address.
>
>We should do a little thinking about how to make sure that the archives
>system can make use of the webui authentication.  In theory, you could just
>use the same browserID/etc. and perform authentication again to provide a
>single sign on with the same tokens, but we can probably do something nicer
>by sharing the webui django accounts.

Definitely.

One thing the engine has to expose is the ability to associate multiple email
addresses with a "user".  The core supports this as a concept, but we may not
have what we need exposed in the REST API yet.  It's also something I want to
expose in the email commands interface.

-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Barry Warsaw]

On Mar 26, 2012, at 10:37 PM, David Jeske wrote:

>Yes. My CSLA code is on the table for MM2 or 3. It's python, and it's BSD.
>More than that, if you folks want it, I'm happy to retool/rewrite it as
>needed to make it suitable. (it has lots of great features, but i havn't
>touched it in a long time and it could be better) I'll put up a demo of the
>mailman list in CSLA tomorrow so folks can check it out.
>
>I also have "some experience" with this space, and by that I mean
>mountainous-experience. I coded/managed/architected several versions of
>eGroups/Yahoo Groups and Google Groups, including the listprocessor, schema
>design, and a few different python mail-to-html formatters. I also wrote my
>own personal gmail-like web-mailer in python, which someday I'll get around
>to open-sourcing.

See, stuff like this is *awesome*.

>Can you share something about dependency philosophy (besides licensing) in
>Mailman?

I really want to promote loose dependencies between major components, with
well-defined formal interfaces for them to interoperate.  Think "service
oriented architecture".  Unlike mm2, which tightly bundled the web ui and
archiver with the core engine, I think we can split these out into separate
components and yet ensure they'll all work together nicely.  Then we can
decide what might fall under the "GNU Mailman" umbrella, what will be our
default services, how we're going to bundle and release them, etc.  But others
will be able to make their own choices.

>Currently CSLA uses Clearsilver templating (BSD c-module) and sqlite
>(public domain) storage.. both of which can be included directly as source
>to make installation easy, but they are both c-modules.
>
>I'm quite fond of the Clearsilver orm-to-templates toolchain, which is in
>use for lots of big sites. However, I also recognize it's not that popular
>in the python commmunity, mostly because we don't promote it, but also
>because the template processor c-module (for performance) instead of pure
>python.

In a way, it's like the Python library ecosystem.  Most third party libraries
are independently developed and released, but pretty much work with (a
specific version of) Python.  Occasionally, some are borged into the Python
stdlib, which is both a blessing and a curse.

So in some sense, CSLA needn't become *the* Mailman archiver, but it should
definitely be *a* Mailman archiver.  Then you can make all the engineering and
design decisions you prefer, but with the confidence that it will Just Work
with Mailman 3.

(This of course also means you don't have to worry about boring stuff like
licensing, FSF copyright assignments, etc. etc.)

>Using SMTP for an included archiver would require it be a long running
>server instead of merely a handler.
>
>Are we talking about third-party-site archivers here?

In some sense, yes.  Eventually, something will percolate to the top, have
technological decisions that we're comfortable with maintaining, compatible
licensing, etc. and then we can bring it under the Mailman umbrella.  But hey,
don't wait for that!  Make it work with the mm3 engine now and release early
and often.

(I clarified in a previous message that SMTP into the archiver is not a
requirement.  The implementation should be flexible enough to handle just
about any input mechanism you need.)

>CSLA doesn't currently have any concept server-auth. The only stateful
>features it has are view-preferences and read-state, neither of which are
>important enough to require a password. It uses a password-less system
>which uses cookies for prefs and a 'read state userid' which a user can
>manually set if they want. I like it, because it doesn't require login to
>get some basic browsing prefs and features.
>
>Hooking up an auth system would be necessary for some of the editing ideas
>in the document I read, or to allow online posting.

Or to support private archives, which is very definitely a feature goal IMO.
Even a 99% public site will likely have a few closed or limited access lists,
so I think an archiver will need to support use case.

-Barry

___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Barry Warsaw]

On Mar 27, 2012, at 10:34 AM, Stephen J. Turnbull wrote:

>Pipermail is going the way of the dodo, yes, but there will be something
>bundled with Mailman, I'm pretty sure.

fsvo "bundled" :)

>(1) the communication from Mailman to the archivers will be via
>LMTP/SMTP, including a Mailman-specific header to identify the
>message's permalink (currently the SHA1 hash of the message ID
>in BASE32 format, IIRC); and

Kind of.  Communication *into* Mailman is via LMTP, but sending a message out
of mm3 into the archiver can be hidden behind the IArchiver interface.  The
prototype archiver, just opens a maildir and adds the new message to that.
The mailarchive implementation drops the message into the outgoing queue, but
with a recipient as specified in the configuration file.  The mhonarc archiver
calls a subprocess and pipes the message's bytes to that process's stdin.  The
moral is that whatever you need to do to get the message into the archiver,
you can do it.

You're right about the algorithm for calculating the X-Message-Id-Hash.  Full
details are on the wiki.  The intent here is to be able to calculate the URL
to the message in the archiver without actually having to communicate with the
archiver.

>(2) the summary, search, and retrieval UI will be a separate
>application from Mailman per se, which will communicate
>with Mailman via the REST API for authentication and user
>configuration purposes.

Yep.

I've also been thinking; we have an IMessageStore interface which currently
isn't much hooked up in any way.  It's possible that a specific archiver could
satisfy the IMessageStore API as well, and that eventually we could implement
things like a mail command that, given a Message-ID (or its hash) would send
you back the message from the store.

>Some ideas that have *not* been elaborated as "philosophy", but
>which I think are consistent with various desiderata and Barry's
>general approach and specific statements are
>
>(3) an archiver Handler for the pipeline will be provided, which
>will store posts in maildir format and do nothing else; and

I've been thinking that the prototype archiver could be this thing.  Well, it
already essentially *is* that thing!  So, it's not in a handler, but gets
invoked from the ArchiveRunner.

>(4) a simple default summary/search/retrieval application will
>be bundled with (but have a separate development cycle from)
>Mailman.

+1

>Even if that's not the current philosophy  that's the
>direction I would propose.
>
>So if you have already developed some stuff, I certainly would
>love to see you put it on the table as a candidate for the
>Mailman 3 default archive web UI.

Definitely.

My intent with mm3 is to provide an enabling platform for archiver
developers.  I think there's lots of opportunity for experimentation here, so
let's make sure we get the API between mm3 and a generic archiver right.
We'll see what pans out and then we can choose a preferred default that we can
bless and bundle in some kind of sumo release.

Cheers,
-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Barry Warsaw]

On Mar 26, 2012, at 03:20 PM, David Jeske wrote:

>I'm writing to find out the state of and philosophy surrounding pipermail
>in mailman, to see if there is a productive way to provide some
>code/development-time to that part of mailman.

That's awesome.  Pipermail (in its current state) is ancient, and it shows.
It's biggest flaw IMO is the lack of "stable URL" support, meaning if you
regenerate the archive from the underlying mbox file, you'll likely break all
your links.  Pipermail has lots of other problems, but this one for example is
why it was ditched for Launchpad.

I think the time is ripe for new archivers, and Mailman 3's philosophy is to
provide a framework in which developers can easily experiment with new
archiver technology, integrated easily into Mailman.

I personally think it would be best to spend effort on one of the many
archiver projects being worked on, and mentioned here, rather than trying to
improve Pipermail.  One possible connection though would be to consider how a
site would migrate from a Pipermail-based archiver to one of the new ones.  Do
you keep the old URLs alive but just not add anything new to Pipermail?  Do
you provide a mapping from Pipermail URLs to new-archiver URLs?  etc.

>I've written code for this a number of times (eGroups, Yahoo Groups, Google
>Groups). I also released an open-source python/clearsilver/sqlite based
>archiver with redundant text-eliding, a few different thread views, and
>search...  ( http://www.clearsilver.net/archive/ ) which is hardly used
>both because I don't try to popularize it, and because many sites just
>leave the default (pipermail).

Neat.  Perhaps you'd like to contribute an implementation of the IArchiver
interface in Mailman 3 that would send posted messages to ClearSilver?

Here's the current interface definition:

http://tinyurl.com/cup7amw

and some example implementations:

http://tinyurl.com/cynyfou

>a) pipermail is fine... if you want to fix a bug or two submit a patch, but
>we don't want to improve it

Note that before I ripped Pipermail out of the Mailman 3 trunk branch, I
created a project on Launchpad and pushed a semi-sanitized branch:

https://code.launchpad.net/pipermail

I don't personally plan to touch this, but if someone is really motivated to
hack on Pipermail specifically, I'd happily give you access.

>b) we're ditching pipermail entirely... in the future sites will have to
>choose an install an external archiver

Yes, but remember, you can choose one or more archivers in Mailman 3.  So it
would be easy to archive to something local, and The Mail Archive, and Gmane,
etc.

>c) we'd love pipermail to be improved... but we still want it to be simple,
>static-html, and dependency free
>d) we'd love a dynamic-ui replacement for pipermail... as long as it uses
>the same cgi/templating model as mailman ui

Because Postorius (the official, but not required MM3 web ui) is Django-based,
I think it should be pretty flexible for customizing the look and feel.  I
won't dictate what a new archiver should look like, but some principles that I
personally think should be followed include:

- Modern web technologies, with flexible templating, so that sites can
  customize the look and feel as needed.

- JavaScript not required: an archiver should be navigable with a text-based
  simple browser, but that also doesn't mean it has to be feature-equivalent
  if you *do* have JavaScript.  I think "progressive enhancement" is the right
  term?  I.e. make it awesome for today's web, but usable in older browsers,
  screen readers (for accessibility), etc.

- Dynamic generation of pages with caching.  I'd love to see an enhanceable
  approach to the actual HTML generation.  Let's say for example, that I
  suddenly want to recognize and hyperlink "bug numbers" so they point to my
  tracker.  I should be able to drop in some extension to do that.  Or maybe
  the spammers have cracked my email obfuscation algorithm.  I should be able
  to drop in a replacement, invalidate the cache, and all my new page would
  automatically get the new obfuscation algorithm.

- Separate, dynamic support for take down notices.  You posted your personal
  information and have asked the site's postmasters to take that down (either
  the full message or the personal information parts of it).  It should be
  really simple for the site admins to do this, while possibly retaining the
  original message in a publicly inaccessible location for forensic purposes.

- Support for private archivers, so configurable authentication would be
  necessary.

- Merging of forums, archives, newsgroups, and IMAP.

- A REST API for querying information about the site, its lists, individual
  archived messages, metrics, etc.  Maybe even some control (e.g. take downs)
  for users with the proper permissions.

- Stable URLs, RFC 5064 + X-Message-ID-Hash.  See the above links.  If you can
  implement the `IArchiver.permalink()` method and ensure that even if
  completely wiped and

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Terri Oda]

On 03/26/2012 11:37 PM, David Jeske wrote:

CSLA doesn't currently have any concept server-auth. The only stateful
features it has are view-preferences and read-state, neither of which are
important enough to require a password. It uses a password-less system
which uses cookies for prefs and a 'read state userid' which a user can
manually set if they want. I like it, because it doesn't require login to
get some basic browsing prefs and features.

Hooking up an auth system would be necessary for some of the editing ideas
in the document I read, or to allow online posting.


So Postorius (the webUI) has a sketch of an auth system using BrowserID 
at the moment.  BrowserID is convenient 'cause it proves you have 
ownership of a given email address, but we should have OpenID working 
soon once we've got the code to confirm that a given OpenID can be 
associated with an email address.


We should do a little thinking about how to make sure that the archives 
system can make use of the webui authentication.  In theory, you could 
just use the same browserID/etc. and perform authentication again to 
provide a single sign on with the same tokens, but we can probably do 
something nicer by sharing the webui django accounts.


 Terri

___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[David Jeske]

On Mar 27, 2012 5:32 AM, "Stephen J. Turnbull"  wrote:
> If the archiver/web UI is going to be distributed
> *with* Mailman, Barry would probably prefer
> Storm + Django because that's what
> Mailman/Protorius (core and admin web UI, resp.) are
> using.  But I imagine that's negotiable as long as
> everything is free software.

I'll take a look at the Mailman2/3 code and see what the pattern looks
like.

Storm-ORM looks like it allows any primary key so that's good. From a quick
glance it looks very similar to the Clearsilver-odb-orm in syntax and
function. My primary beef with django-orm (last time I looked at it) was it
requiring a unique-id primary key which is pretty poor for multi-row fetch
performance design.

Clearsilver templating has a slightly more aggressive enforcement of 'no
code in template' than django, because it has an explicit intermediate
dataset called HDF (it doesn't allow access to arbitrary python lists,
structs, classes, and methods). This allows it to work from any language
(not just python), and makes the binding from the ORM a bit more explicit.
However, it also means it's a c-module, and I can see the advantage to
keeping c-compile dependencies to a minimum.

>>> AIUI, the current philosophy is that
>>>
>> Using SMTP for an included archiver would require
>> it be a long running server instead of merely
>> a handler.
>
> Sure, but you're already running a sufficient server,
> namely your MTA.

Gotcha. I thought you were saying Mailman wanted the archiver to be a
long-running server accepting SMTP/LMTP on a socket.

I normally manage the archive queue with the MTA, and have it launch the
archive-injestor for each message.. (or for higher volume applications,
long-running injester on a Maildir).. which sounds like what you're saying..

After I finish some quick cleanup and UI changes to CSLA I want to do, I'll
take a look at MM2/3 and see if I have better questions once I'm informed.

Thanks for the great overview.
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Stephen J. Turnbull]

On Tue, Mar 27, 2012 at 2:37 PM, David Jeske  wrote:

> Can you share something about dependency philosophy (besides licensing) in
> Mailman?

Well, for the official poop you'll have to wait for Barry, but AFAICS archivers
aren't restricted to Storm + RESTish (which is what Mailman itself uses)
because they're separate applications.  If the archiver/web UI is going to be
distributed *with* Mailman, Barry would probably prefer Storm + Django
because that's what Mailman/Protorius (core and admin web UI, resp.) are
using.  But I imagine that's negotiable as long as everything is free software.

I don't think the database backend much matters, but Mailman and Django
are both happy to use sqlite, I believe.

> I'm not a fan of javascript client-side rendering because of the generally
> poor performance, poor mobile compatibility, and lack of benefit for this
> kind of application.

Not my pidgin, you'll need to talk to Toshio/Barry about that.
>> AIUI, the current philosophy is that
>>
>> (1) the communication from Mailman to the archivers
>> will be via LMTP/SMTP, including a Mailman-specific
>> header to identify the message's permalink (currently
>> the SHA1 hash of the message ID in BASE32
>> format, IIRC); and
>
> Using SMTP for an included archiver would require it be a long running
> server instead of merely a handler.

Sure, but you're already running a sufficient server, namely
your MTA.  Of course that may not be efficient.  If an mbox or
maildir storage is sufficient, any decent MTA/MDA can do that
for you.  If not, I don't think that writing a separate handler is
a big deal; I'm just saying that AIUI the configurable Handler
provided with Mailman will certainly know how to do LMTP/
SMTP.

> Are we talking about third-party-site archivers here?

That is the motivation for choosing ?MTP as the default
transport to archivers, yes.

Steve
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[David Jeske]

I brought up a quick CSLA injest of some mailman-dev posts to show off some
basic features. I'm having a little trouble with the swish-e code, so the
features that depend on text-indexing arn't working at the moment (search,
author search).  The current UI uses user-cookies to choose between a
2-pane frames mode and a noframes mode. I set the frames-mode as the
default, which is inspired by the old google-groups usenet multi-message
tree. Use "prefs" to check out the noframes mode, which is more like
pipermail.

Take a peek more for general capabilities than UI details (unless you see
something you really like). I'm planning to change the UI to be more like
the gmail-esq conversation style UIs I've been using since then... (where
most/all of a thread's messages are rendered into a single page) I like the
outgoing google-groups UI pretty well  (not the new AJAXy one).

1) basic "home" for a list with bymonth and top-author metrics, RSS.
Extended top-author page.

http://dj1.willowmail.com/csla/Mailman-Developers
http://dj1.willowmail.com/csla/Mailman-Developers/top_authors?date=2012-03

2) Prefs, including timezone, read-status key, browse-style.. (not sure if
the read state is working on my install)

http://dj1.willowmail.com/csla/Mailman-Developers/prefs

3) threadlist for the month (frames or noframes in prefs)

http://dj1.willowmail.com/csla/Mailman-Developers/browse/month/2012-03/107

4) single-thread-list display (frames or noframes in prefs)

http://dj1.willowmail.com/csla/Mailman-Developers/threads?start=1

5) here is a good example of a thread with redundant text eliding

http://dj1.willowmail.com/csla/Mailman-Developers/browse_frm/thread/4/221
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[David Jeske]

> So if you have already developed some stuff, I
> certainly would love to see you put it on the table
> as a candidate for the Mailman 3 default
> archive web UI.

Yes. My CSLA code is on the table for MM2 or 3. It's python, and it's BSD.
More than that, if you folks want it, I'm happy to retool/rewrite it as
needed to make it suitable. (it has lots of great features, but i havn't
touched it in a long time and it could be better) I'll put up a demo of the
mailman list in CSLA tomorrow so folks can check it out.

I also have "some experience" with this space, and by that I mean
mountainous-experience. I coded/managed/architected several versions of
eGroups/Yahoo Groups and Google Groups, including the listprocessor, schema
design, and a few different python mail-to-html formatters. I also wrote my
own personal gmail-like web-mailer in python, which someday I'll get around
to open-sourcing.

Can you share something about dependency philosophy (besides licensing) in
Mailman?

Currently CSLA uses Clearsilver templating (BSD c-module) and sqlite
(public domain) storage.. both of which can be included directly as source
to make installation easy, but they are both c-modules.

I'm quite fond of the Clearsilver orm-to-templates toolchain, which is in
use for lots of big sites. However, I also recognize it's not that popular
in the python commmunity, mostly because we don't promote it, but also
because the template processor c-module (for performance) instead of pure
python.

If it's necessary I could convert CSLA to use django templates, though they
have a nasty design/performance problem with the orm that I think would
prevent me from using it (unless I fixed it).

I'm not a fan of javascript client-side rendering because of the generally
poor performance, poor mobile compatibility, and lack of benefit for this
kind of application.

> Doing a *better* job of formatting HTML (than pipermail) is easy.
> Doing a  *good* job is going to be relatively hard, though.

Isn't that the truth.

Back at Egroups there was a time where it seems like we used to tweak the
formatter almost daily. Deciding when preformatting is necessary vs harmful
is a real pain.

My CSLA python code has a fairly stable msg-to-html formatter that I've
been happy with for a long time. It does very aggressive automatic
repetitive text eliding for thread-display, so you can read many messages
in a thread in a single page without massive quote pileup. View-source is
provided for the unfortunate case where it does something bad.

> AIUI, the current philosophy is that
>
> (1) the communication from Mailman to the archivers
> will be via LMTP/SMTP, including a Mailman-specific
> header to identify the message's permalink (currently
> the SHA1 hash of the message ID in BASE32
> format, IIRC); and

Using SMTP for an included archiver would require it be a long running
server instead of merely a handler.

Are we talking about third-party-site archivers here?

> (2) the summary, search, and retrieval UI will be a
> separate application from Mailman per se, which will
> communicate with Mailman via the REST API for
> authentication and user configuration purposes.

That makes sense...

CSLA doesn't currently have any concept server-auth. The only stateful
features it has are view-preferences and read-state, neither of which are
important enough to require a password. It uses a password-less system
which uses cookies for prefs and a 'read state userid' which a user can
manually set if they want. I like it, because it doesn't require login to
get some basic browsing prefs and features.

Hooking up an auth system would be necessary for some of the editing ideas
in the document I read, or to allow online posting.
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Stephen J. Turnbull]

On Tue, Mar 27, 2012 at 9:11 AM, Toshio Kuratomi  wrote:
> On Mon, Mar 26, 2012 at 03:20:05PM -0700, David Jeske wrote:

> As I was talking about hyperkitty we touched briefly on
> what I think is one of the central conundrums about having only unofficial
> third party archivers:  how to have a consistent programatic interface
> available over http.

I don't understand why this is an issue.  Third party archives
(no "r"!) are going to want to have their own copies of the
posts, and will be heavily duplicated in any case (if only because
they store stuff on RAID and backup regularly :-).  Communicating
with them via SMTP will be fine, and LMTP provides a reasonable
way to handle local archives efficiently -- you can even just use
a virtual user and a standard MDA for the purpose.

If a third party wants to provide archiving service, but allow the
list to supply authentication and user configuration information,
that's another kettle of fish, but that's not a question of the
archiver itself, that's going to be part of the Mailman 3 REST
API per se.

> I don't like that a site that wanted both would need to run two archivers
> that were saving mail into two sets of storage.

Agreed, but the storage issue is easy to solve for the
Mailman site itself.  Just provide a simple Handler to stuff
posts into a maildir format mail folder (as noted above, this
can communicate with a standard MDA via LMTP, and
"require" that all third party archive UI software support that
format.  Maybe if space is that big an issue, provide a
maildir-in-zipfile backend.  If they want to do something
else (eg, access an IMAP store or stuff it into a PostgreSQL
database) they can provide their own handler for that ...
surely this is trivial?

> (One thing I notice about grackle now is that it's AGPL... that's going to
> be unpleasant for some sites to run.  Perhaps we can change that or get some
> changes added to the AGPL.)

Yes, let's stay away from copyleft, period, to get a standard
archiver that commercial sites and developers will be
comfortable with extending.  I know people are disgusted with
Plesk and especially cPanel, but (a) GPL hasn't stopped those
folks keeping their sources away from general public, and even
with AGPL, *we* would have to keep track of *their* releases
to get our hands on their changes in any timely fashion -- which
half the time we don't want anyway! -- and (b) what we're doing
is all about UI in some sense.  Even the pipeline architecture of
core Mailman is about allowing users (the script-able but not
always program-able list and site admins) to easily make
changes to their lists' configurations.  UI design is necessarily
visible, and it's unlikely to be that much of a challenge to
reproduce their changes.  The hard part will be getting the
internal design past Barry, anyway (which is one reason why
some of the more frequently-requested features provided by
cPanel Mailman, like duplicate list names across virtual servers,
haven't been added in the past).
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Stephen J. Turnbull]

On Tue, Mar 27, 2012 at 7:20 AM, David Jeske  wrote:
> I'm writing to find out the state of and philosophy surrounding pipermail
> in mailman,

There's been a lot of discussion of this over the years; you should spend
some time in the mailman-developers and mailman-users archives
looking for it, because neither the needs nor Barry's philosophy have
changed much over that time period.  It would be really nice if somebody
would summarize it in Python-PEP fashion, but that's above and beyond
the call of duty from your point of view.  (Ie, it would be useful, but you
may prefer coding etc, and that's fine too I believe, though I don't speak
for Barry, although I occasionally get lucky channeling him ;-).

> Unfortunately, pipermail doesn't do a good job formatting
> messages for html.. (messages with no line-breaks are the most
> annoying problem I regularly run into)

Doing a *better* job of formatting HTML (than pipermail) is easy.
Doing a  *good* job is going to be relatively hard, though.

> If there is some code (and time) I can contribute to mailman/pipermail, I'd
> like to do so.

A better pipermail probably would be a good thing, as transition to
Mailman 3 is likely to take several years, and Mailman 2 is likely to
continue to be used for several years.  However, IMO you should
focus on low-hanging fruit, as there are now much better alternatives
to the technology used in pipermail, to the point where rewriting from
scratch (or adapting a 3rd-party product) for Mailman 3 makes sense.

> a) pipermail is fine... if you want to fix a bug or two submit a patch, but
> we don't want to improve it

Pipermail is fine for EOLing Mailman 2.  Improving it would be good, but
only if archive formats don't need upgrading and at minimum expense
of effort, IMO.

> b) we're ditching pipermail entirely... in the future sites will have to
> choose an install an external archiver

Pipermail is going the way of the dodo, yes, but there will be something
bundled with Mailman, I'm pretty sure.

> d) we'd love a dynamic-ui replacement for pipermail... as long as it uses
> the same cgi/templating model as mailman ui

A dynamic UI replacement seems to be a very good idea to me.
Sites running mailman at all will be running a dynamic UI for the
admin, so I see no good reason to stick to static for the archives.

AIUI, the current philosophy is that

(1) the communication from Mailman to the archivers will be via
LMTP/SMTP, including a Mailman-specific header to identify the
message's permalink (currently the SHA1 hash of the message ID
in BASE32 format, IIRC); and

(2) the summary, search, and retrieval UI will be a separate
application from Mailman per se, which will communicate
with Mailman via the REST API for authentication and user
configuration purposes.

Some ideas that have *not* been elaborated as "philosophy", but
which I think are consistent with various desiderata and Barry's
general approach and specific statements are

(3) an archiver Handler for the pipeline will be provided, which
will store posts in maildir format and do nothing else; and

(4) a simple default summary/search/retrieval application will
be bundled with (but have a separate development cycle from)
Mailman.

Even if that's not the current philosophy  that's the
direction I would propose.

So if you have already developed some stuff, I certainly would
love to see you put it on the table as a candidate for the
Mailman 3 default archive web UI.
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[David Jeske]

On Mon, Mar 26, 2012 at 5:11 PM, Toshio Kuratomi  wrote:

> We'd love to have work done on the archier!  I know that we're ditching
> pipermail entirely and that archivers are becoming separate from the core
> mailman.  What I don't know is whether mailman3 will eventually have
> a standard archiver which lives outside of the core mailman but is
> recommended for installation along with it.
>

I see.. that sounds like option-b.

I highly recommend reconsidering this and including a standard archiver
with mailman. If the number of sites that use pipermail is any indication,
I think failing to include something will basically mean lots of lists
without any archives.


> At PyCon a few weeks ago, I demoed hyperkitty which showed some of the
> things that a next generation archiver could do.


I recommend you take a closer look at
ClearsilverListArchive,
it's written in Python, Clearsilver, SQLite.. is "real open-source" (BSD
License), and hits most of the features on your  ModernArchiving wishlist
plus a bunch you didn't (author pages, redundant text elimination, cookie
preferences.

As for the features it doesn't have from your list: Editing would be easy
to add because it's sqlite (deciding on the auth system is probably more of
an issue than the editing). Anti-Crawl code is really an issue of
configuration for cheap in-memory state-management. NNTP is well. that
would be a big job that I doubt will be bitten off by something as "small"
as a list archiver.

Sadly I can't point to any lists using it at the moment, because, well,
it's hidden under a rock. I'll injest an archive of the mailman list so you
can take it for a spin.


> As I was talking about hyperkitty we touched briefly on what I think is
> one of the central conundrums about having only unofficial third party
> archivers:  how to have a consistent programatic interface available over
> http.


What is the REST UI used by? CSLA supports RSS. When it comes to a more
involved REST UI, what software would be hitting it? I don't think I'll
understand your other API/REST points until I see an answer to this.


> Grackle is another archiver for mailman that doesn't have the UI bells and
> whistles of hyperkitty but it does make an effort to expose a REST UI to
> the world.  I think that's a beautiful thing.  But I don't like that a site
> that wanted both would need to run two archivers
> that were saving mail into two sets of storage.
>

I think here you are entering into a catch-22. If you have a single storage
system, then you have a single storage schema, which means you have a
single set of things you can do fast and most other things become
impractical (because they would require synchronizing state).

I'm quite sure, for example, that the Grackle schema is not the same as the
CSLA schema, and that many CSLA features would be impossible with the
Grackle REST API. (short of just using it to suck everything down, but then
you're just duplicating).

Why is message duplication an issue?
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] any interest in a new built-in web-archive? (i.e. pipermail replacement)

[Toshio Kuratomi]

On Mon, Mar 26, 2012 at 03:20:05PM -0700, David Jeske wrote:
> I'm writing to find out the state of and philosophy surrounding pipermail
> in mailman, to see if there is a productive way to provide some
> code/development-time to that part of mailman.
> 
> I know there are several decent third-party archivers out there, but many
> of the mailing list archives I browse regularly are using pipermail because
> it's the mailman default one less thing to install, administer, and
> upgrade.  Unfortunately, pipermail doesn't do a good job formatting
> messages for html.. (messages with no line-breaks are the most annoying
> problem I regularly run into)
> 
> I've written code for this a number of times (eGroups, Yahoo Groups, Google
> Groups). I also released an open-source python/clearsilver/sqlite based
> archiver with redundant text-eliding, a few different thread views, and
> search...  ( http://www.clearsilver.net/archive/ ) which is hardly used
> both because I don't try to popularize it, and because many sites just
> leave the default (pipermail).
> 
> If there is some code (and time) I can contribute to mailman/pipermail, I'd
> like to do so. I'm writing this message to "take a temperature" and find
> out what, if any, contributions would be appreciated the mailman
> development team. I could imagine answers like:
> 
> a) pipermail is fine... if you want to fix a bug or two submit a patch, but
> we don't want to improve it
> b) we're ditching pipermail entirely... in the future sites will have to
> choose an install an external archiver
> c) we'd love pipermail to be improved... but we still want it to be simple,
> static-html, and dependency free
> d) we'd love a dynamic-ui replacement for pipermail... as long as it uses
> the same cgi/templating model as mailman ui
> 
> Thoughts? Is there any help I can offer up here?
>
We'd love to have work done on the archier!  I know that we're ditching
pipermail entirely and that archivers are becoming separate from the core
mailman.  What I don't know is whether mailman3 will eventually have
a standard archiver which lives outside of the core mailman but is
recommended for installation along with it.

At PyCon a few weeks ago, I demoed hyperkitty which showed some of the
things that a next generation archiver could do.  hyperkitty is continuing
to be developed.  As I was talking about hyperkitty we touched briefly on
what I think is one of the central conundrums about having only unofficial
third party archivers:  how to have a consistent programatic interface
available over http.  Grackle is another archiver for mailman that doesn't
have the UI bells and whistles of hyperkitty but it does make an effort to
expose a REST UI to the world.  I think that's a beautiful thing.  But
I don't like that a site that wanted both would need to run two archivers
that were saving mail into two sets of storage.

I think there's several ways we could go about this.

* We could create a standard REST API that archivers were free but
  encouraged  to implement.
* We could expand the python API that archivers needed to expose and then
  implement the REST API inside of mailman Core (or a re-envisioned,
  lightweight Grackle).
* We could promote a standard archiver much as we're going to promote
  posterius as the standard admin front-end and that archiver would provide
  a standard REST API that others could then emulate.


hyperkitty:
Project page: https://fedorahosted.org/hyperkitty/
Code: http://bzr.fedorahosted.org/bzr/hyperkitty/
Demo: http://mm3test.fedoraproject.org/2

grackle: https://launchpad.net/grackle

(One thing I notice about grackle now is that it's AGPL... that's going to
be unpleasant for some sites to run.  Perhaps we can change that or get some
changes added to the AGPL.)

-Toshio


pgp3wLf2vz2Es.pgp
Description: PGP signature
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9