Re: [Mailman-Developers] Python 3

[Florian Fuchs]

Am 06.01.2015 um 13:48 schrieb Barry Warsaw:
 On Jan 06, 2015, at 01:37 PM, Aurelien Bompard wrote:
 
 I must also consider how much work it would be to just port HyperKitty 
 KittyStore to Python3.  All things considered, it may very well be faster and
 more reliable.
 
 Of course, I encourage any porting to Python 3 just on principle :) and
 certainly if I can help, let me know.  But you may still want to consider a
 more distributed communication protocol between core and HK.  What if a site
 wanted to run them on different VMs for example?

Another thing we shouldn't forget is that Hyperkitty or Postorius might
not be the only apps living in a Django project. Some people might
choose to integrate them into their long-running Django sites, with lots
of Py2-app dependencies.

Of course it would be easier for us to just port our apps to Py3. But as
far as I remember integratability was the main argument in favor of
keeping our Django apps bilingual when we discussed the topic at Pycon
last year. But then, hey, one *could* also argue we ignore these issues
in order to make a point.


Florian

___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Aurelien Bompard]

I must also consider how much work it would be to just port HyperKitty 
KittyStore to Python3.
All things considered, it may very well be faster and more reliable.
A.
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

On Jan 06, 2015, at 01:37 PM, Aurelien Bompard wrote:

I must also consider how much work it would be to just port HyperKitty 
KittyStore to Python3.  All things considered, it may very well be faster and
more reliable.

Of course, I encourage any porting to Python 3 just on principle :) and
certainly if I can help, let me know.  But you may still want to consider a
more distributed communication protocol between core and HK.  What if a site
wanted to run them on different VMs for example?

Cheers,
-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Aurelien Bompard]

  What if a site wanted to run them on different VMs for example?


Yeah, but certainly there should be a limit to that. What if a site wanted
to run each mailman runner on a different VM for example?
Oh well, I guess there's no other way. I hope it's not going to be a mess
to configure, I'd like to avoid having the admin set HK's base URL in 3
different places.

Another thing we shouldn't forget is that Hyperkitty or Postorius might
 not be the only apps living in a Django project. Some people might
 choose to integrate them into their long-running Django sites, with lots
 of Py2-app dependencies.


Very good point.

Thanks.
Aurélien
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

On Jan 06, 2015, at 02:55 PM, Aurelien Bompard wrote:

  What if a site wanted to run them on different VMs for example?


Yeah, but certainly there should be a limit to that. What if a site wanted to
run each mailman runner on a different VM for example?

There's already some support for that, in a way.  The runners with queues
already employ an algorithm to safely split the queue space up into 2^n
separate runners without the need for locking.  They do of course have to have
access to the same directory (e.g. over NFS).  Also, now that MM3 is backed by
SQL, the use of a client/server db like PostgreSQL eliminates another
co-location bottleneck.  I wouldn't say this latter is thoroughly fleshed out,
and I'm not even sure there's a compelling use case, but it may indeed be
possible.

Oh well, I guess there's no other way. I hope it's not going to be a mess to
configure, I'd like to avoid having the admin set HK's base URL in 3
different places.

Agreed.  Configuration complexity among all the various components is
definitely something to be wary of.

Cheers,
-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Florian Fuchs]

Am 29.12.2014 um 00:26 schrieb Barry Warsaw:
 On Dec 28, 2014, at 01:51 PM, Aurelien Bompard wrote:
 
 As I mentioned, I think LMTP *could* work, but REST (inside HK) could
 work too.  Aurelien, what do you think?

 I'd go with REST, it seems more flexible and we already have nice libraries
 for it.
 
 REST will be very nice, and I think with something like Falcon, pretty easy to
 implement.  You probably don't need the object-based (i.e. restish-like)
 traversal stuff, but if so, I'd be happy to split that out into a separately
 library that HK could use (and keep it bilingual).  Keep it simple, but think
 through the resource tree and leave open some opportunities for expanding the
 API later, both for use by MM3 and perhaps other scripting applications.

Are you planning to implement that in a way similar to the IArchiver
interface? Like, say, having a REST-based equivalent for each of the
IArchiver's `list_url`, `permalink` and `archive_message` methods?

I think it would be nice if it worked more or less the same as the an
IArchiver class, not only on the implementation side but also as far as
the configuration is concerned. Although allowing multiple, potentially
slow-answering, REST-based archivers could be a trap performance-wise
(if those requests are not made concurrently).

Florian
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Florian Fuchs]

Am 29.12.2014 um 00:27 schrieb Barry Warsaw:
 On Dec 27, 2014, at 08:41 AM, Florian Fuchs wrote:
 
 How about a third option, a generic pub/sub or event archiver,
 implemented in py3? It would meet all the above criteria (archivers on other
 systems, no dependency on py3 -- or Python for that matter).

 We could start supporting one backend (zeromq for instance) and maybe
 add more later.
 
 This would be pretty cool.  Would you be interested in writing an IArchiver
 implementation for that?

Sure. I don't know how much interest there is out there in something
like this. But I think I like the idea enough to just do it anyway. :-)


Florian
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Stephen J. Turnbull]

Barry Warsaw writes:

  What if I do A/B releases for 3.0b5?

Up to you; I have no objection.

But I'm only going to be looking at the Python 3 branch.  I've already
ported my (work internal) Django site to Python 3 (which was mostly
trivial, I had more trouble with the Django upgrade itself).


___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

Here's a crazy, probably stupid idea.  I want to do one more release before
the end of the year, probably tomorrow.

What if I do A/B releases for 3.0b5?

A would be current snapshot of trunk, i.e. MM3 running on Python 2.7.

B would be current snapshot of the py3 branch, i.e. MM3 running on Python 3.4.

I will hold of on the merge until 2015.

The advantage of this is that folks would get a chance to download the
tarballs of their choice and give it a try.  I'd encourage people to try the
Python 3 (i.e. B) release.

I think people are more engaged with an actual release tarball than with bzr
branches, so this could provide some good feedback.  I think right now the two
branches are functionally equivalent.

Cheers,
-Barry


pgp9FNnamxayC.pgp
Description: OpenPGP digital signature
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Aurelien Bompard]

 This was a design mistake, I think.


Yeah, I see that now. But at that time, if I had known that I had two more
years (and counting) before MM3 was released, I'd probably have made
different choices.
I hope I'm not making the same mistake again.

 As I mentioned, I think LMTP *could* work, but REST (inside HK) could
 work too.  Aurelien, what do you think?

I'd go with REST, it seems more flexible and we already have nice libraries
for it.

A.
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Tanstaafl]

On 12/26/2014 4:25 PM, Barry Warsaw ba...@list.org wrote:
 On Dec 26, 2014, at 08:48 PM, Geoff Shang wrote:
 FWIW, Debian Jessy (aka Testing/Frozen?) has Python 3.4.2.

 Yep, Jessie will have 3.4, and Ubuntu has had it since Trusty Tahr (14.04
 LTS).  I don't know about other distros.

First, Jessie is not even released yet, wheezy is the current stable,
and still has a long lifecycle remaining.

Second, I (for one, but there are many other debian admins maintaining
servers who feel the same) will not be upgrading my wheezy install to
jessie due to the ongoing systemd debacle.

Locking them out would be a mistake, imo.
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

On Dec 27, 2014, at 03:57 PM, Stephen J. Turnbull wrote:

By the way, I would say to adopt modern IETF practice here and drop
the X- (in practice collisions are rare while the annoyance of
fixing platforms to use the standardized name is frequent), and
include the algorithm in the name.  Eg, Message-ID-MD5 or
Hashed-Message-ID-MD5.  Or we could use the List-* namespace.

We should do this while we still can. :-)  If you want I can try to
write an RFC to make it official.

I like the idea of putting this information in a List-* header, and I'll take
you up on the RFC offer.  Are you thinking about trying to push this through
the IETF to make it official?

The spec currently lives on the wiki:

http://wiki.list.org/display/DEV/Stable+URLs

MM3 and HK should both be implementing this now, and I think mail-archive.com
does too.

If we change the header name, I'd want to keep X-Message-ID-Hash for the MM3
final release, but deprecate it.  I.e. MM3 would write both headers.

As for what the List-* header would be, well, if you wanted to include the
algorithm name, to be completely accurate it would have to be something like
List-Base32-Encoded-SHA1-Hash-Of-The-Message-ID.  Yuck ;)

The value of this header both serves to uniquely identify the message in a
more regular format, and to serve as the final path component in the
Archived-At (RFC 5064) header.  So the following names come to mind:

List-Message-ID
List-Archive-ID
List-Archived-At-ID

suggestions welcome.

The only reason I can think of is that you want to check that the permalink
isn't already occupied (that's the only thing HyperKitty proper knows that
can't be computed the same way in the IArchiver as in HyperKitty proper
AFAICS)

Right.  However, when this was discussed several years ago, the
mail-archive.com guys did some extensive data analysis on their vast
collection of email.  You'd have to go spelunking in the -developers archives
for details, but I recall that the collision rate was so small as to be
effectively negligible, even more so if you ignore spam.  And if the
X-Message-ID-Hash collides, then the Message-ID will collide, and it's likely
that any archiver would drop the message anyway.

My own preference is for a permalink that can be computed from the
originator header data (author, recipients, date, message ID, subject)
by anyone with access to the message, and that means you need the
archive server to be able to deal gracefully with collisions.  (In
practice message IDs are not perfect UUIDs, although they're very
close, and some messages don't have them or have different ones
assigned by mediating hosts at arrival at multiple recipients.)

Right, we hash (pun intended :) all this out years ago.  We can ignore
collisions, and we can do the entire calculation on the server side, using
Message-ID as the sole input.  I think the only issue that's worth reopening
is the name of the header.

Cheers,
-Barry


pgpilrya8Pypk.pgp
Description: OpenPGP digital signature
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

On Dec 28, 2014, at 09:23 AM, Tanstaafl wrote:

Second, I (for one, but there are many other debian admins maintaining
servers who feel the same) will not be upgrading my wheezy install to
jessie due to the ongoing systemd debacle.

I *definitely* don't want to infect this list with a debate about systemd, but
as I understand it, while systemd will be the default in Jessie, it will still
be possibly to run it with other init systems.

Sticking to the issue at hand: whether we should try to support older Python 3
versions in order to support older Linux distro versions.  The mechanics of
setting up automated tests for those two versions is tricky, but I took a look
at the test suite for the py3 branch against Python 3.3 and 3.2.

Python 3.3 is no problem; the test suite passes fully.  This doesn't help for
Wheezy though.

Python 3.2 is more problematic.  There are quite a few things that aren't
supported, some of which would be easy to fix (e.g. residual uses of u''
strings), some which could be compat'd around (e.g. missing unittest.mock
library and contextlib.ExitStack classes), and some which are probably
blockers.  The biggest immediate problem seems to be that Falcon doesn't
support Python 3.2.  Its minimum requirement is 3.3.

But here's the thing: let's say you want to run MM3 on Wheezy, or some other
Linux distro that doesn't have Python 3.4.  It's unlikely that MM3 will be
backported into those distro's backports channels, so you'll have to install
it from some alternative location, like from source, a PPA, or an unofficial
channel.  The same probably goes for the newer versions of some MM3
dependencies.  If you're going to install MM3 that way anyway, is it that much
more effort to install Python 3.4 that way too?

Cheers,
-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

On Dec 28, 2014, at 01:51 PM, Aurelien Bompard wrote:

 As I mentioned, I think LMTP *could* work, but REST (inside HK) could
 work too.  Aurelien, what do you think?

I'd go with REST, it seems more flexible and we already have nice libraries
for it.

REST will be very nice, and I think with something like Falcon, pretty easy to
implement.  You probably don't need the object-based (i.e. restish-like)
traversal stuff, but if so, I'd be happy to split that out into a separately
library that HK could use (and keep it bilingual).  Keep it simple, but think
through the resource tree and leave open some opportunities for expanding the
API later, both for use by MM3 and perhaps other scripting applications.

Cheers,
-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

On Dec 27, 2014, at 08:41 AM, Florian Fuchs wrote:

How about a third option, a generic pub/sub or event archiver,
implemented in py3? It would meet all the above criteria (archivers on other
systems, no dependency on py3 -- or Python for that matter).

We could start supporting one backend (zeromq for instance) and maybe
add more later.

This would be pretty cool.  Would you be interested in writing an IArchiver
implementation for that?

Cheers,
-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Stephen J. Turnbull]

Barry Warsaw writes:

  I like the idea of putting this information in a List-* header, and
  I'll take you up on the RFC offer.

OK.

  Are you thinking about trying to push this through the IETF to make
  it official?

Yes.  It will depend on how much resistance I get, but having it
already implemented and used in Mailman will certainly help.  On the
other hand, there may be resistance on the basis that RFC 5064 already
does everything that is really needed.

  The spec currently lives on the wiki:
  
  http://wiki.list.org/display/DEV/Stable+URLs

Yes, I'm a little bit familiar with that spec. :-)

  If we change the header name, I'd want to keep X-Message-ID-Hash
  for the MM3 final release, but deprecate it.  I.e. MM3 would write
  both headers.

I'll ask some of the IETF guys what they think about that.  But if you
put it in a public release, you're screwing the same kind of people
Tanstaafl was talking about.  Beta testers (and I mean beta testers,
ie, people who have put the code in production even though it's not
considered a public release) have signed up for this kind of
annoyance.  Random ancient Debian sysadmins haven't.

Of course we don't want to abuse our beta testers if we can avoid it,
but I think if we don't want to maintain dual headers indefinitely,
the public release is the time to get rid of the X- version.

  As for what the List-* header would be, well, if you wanted to
  include the algorithm name, to be completely accurate it would have
  to be something like
  List-Base32-Encoded-SHA1-Hash-Of-The-Message-ID.  Yuck ;)

We'd have to think somewhat carefully about how strong a hash we want
to use if we don't specify algorithm in the field name.  I'm not
particularly concerned with how many bytes the header takes up.
Future users can just deal with the implied BASE32 vs. BASE85 or
whatever.  However, if somebody thinks they need a stronger hash than
we chose, we'll have interoperability problems for people who receive
the message off-list.

  The value of this header both serves to uniquely identify the
  message in a more regular format, and to serve as the final path
  component in the Archived-At (RFC 5064) header.  So the following
  names come to mind:
  
  List-Message-ID
  List-Archive-ID
  List-Archived-At-ID
  
  suggestions welcome.

The last two are too easily confused with Archived-At.

  Right.  However, when this was discussed several years ago, the
  mail-archive.com guys did some extensive data analysis on their
  vast collection of email.  You'd have to go spelunking in the
  -developers archives for details, but I recall that the collision
  rate was so small as to be effectively negligible,

Yes.  The problem is that there are people out there with MUAs that
provide bogus Message-IDs (Kyle Jones's VM used to do that), and for
those people all messages after the first get dropped.

Note that if the server does indeed ignore the possibility of
collisions on Message-ID, then there is no need (AFAICS) for the
thin IArchiver to communicate with the archiver proper.  I don't
see how it hurts to provide for the possibility of an archiver that
does check content.

  Right, we hash (pun intended :) all this out years ago.  We can
  ignore collisions, and we can do the entire calculation on the
  server side, using Message-ID as the sole input.  I think the only
  issue that's worth reopening is the name of the header.

Well, that's true for *us*.  The folks at the IETF don't have a habit
of leaving well enough alone, though. ;-)



___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Aurelien Bompard]

 One of my top priorities has been to port Mailman 3 (core) to Python 3.
 This
 work is now complete, and ready to be merged into trunk.  No doubt bugs
 still
 lurk, but at least the entire test suite is passing.


Congrats !


* A few database columns have changed from LargeBinary to Unicode.  I have
 not
   yet figured out how to add the Alembic schema migrations to handle this,
 but
   I'll do so before the release.


This should be pretty easy, something like the following command should
work:
alembic -c src/mailman/config/alembic.cfg revision --autogenerate -m
python3 port

[...]  then I'll merge this to trunk and do a 3.0b5 release as a
 Python 3-only application.  Mailman 3 will not be bilingual so Python 2
 support will be dropped.


Wow. I am very surprised.
So we went from Python3 compatibility is not on the blocker list for
Mailman 3.0 to Mailman 3.0 will only work on Python 3.
That's quite a change.
This means I must now port HyperKitty and Kittystore to Python3, check that
we don't use Py2-only libraries, etc.
And mailman.client must be ported too, since it does import stuff from
mailman for testing purposes. I think Postorious is safe, but that must be
verified (I believe it imports the testing framework from mailman.client
and thus must be ported too).

And I thought the 3.0 release was almost there. I must say this is rather
discouraging.

Aurélien
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Pierre-Yves Chibon]

On Fri, Dec 26, 2014 at 04:38:08PM +0100, Aurelien Bompard wrote:
 [...]  then I'll merge this to trunk and do a 3.0b5 release as a
  Python 3-only application.  Mailman 3 will not be bilingual so Python 2
  support will be dropped.
 
 
 Wow. I am very surprised.
 So we went from Python3 compatibility is not on the blocker list for
 Mailman 3.0 to Mailman 3.0 will only work on Python 3.
 That's quite a change.
 This means I must now port HyperKitty and Kittystore to Python3, check that
 we don't use Py2-only libraries, etc.
 And mailman.client must be ported too, since it does import stuff from
 mailman for testing purposes. I think Postorious is safe, but that must be
 verified (I believe it imports the testing framework from mailman.client
 and thus must be ported too).

This also means that mailman 3 will pretty much only not work on server-oriented
distro (except probably Debian that likely ships a python 3 version, although
not being involved in Debian, I do not know for sure).
For the record, neither RHEL7 nor SL7 nor CentOS7 currently have a python3
stack...

 And I thought the 3.0 release was almost there. I must say this is rather
 discouraging.

Same for me.

Pierre
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

On Dec 26, 2014, at 04:38 PM, Aurelien Bompard wrote:

This should be pretty easy, something like the following command should work:
alembic -c src/mailman/config/alembic.cfg revision --autogenerate -m python3
port

I'm having some trouble with this because the system alembic doesn't have
access to the mailman source tree, which it needs (you get import errors).  I
tried to run alembic out of the .tox virtualenv, but that fails because the
mailman config system needs to be initialized.

Early on in the SA branch, there was an alembic subcommand to the mailman
command.  I don't think I want that, but I think we need something that calls
into alembic programmatically, but with sys.path set up correctly, and the
mailman system properly initialized.  Maybe a 'withlist' script will do the
trick.  Anyway, I'll spend some time working on this and follow up here if
needed.


[...]  then I'll merge this to trunk and do a 3.0b5 release as a  Python
3-only application.  Mailman 3 will not be bilingual so Python 2  support
will be dropped.

This means I must now port HyperKitty and Kittystore to Python3

Oh gosh, no!  I'm sorry I was imprecise, but I meant only that Mailman 3
*core* would be Python 3.  I certainly would never make you guys do all that
busy work without consulting you first.  So for the record, HyperKitty and
Postorious can absolutely port on their own time frame, and that need not at
all block a release of the full suite.  One of the advantages of accessing the
core through the REST API is that it doesn't matter what clients like
HyperKitty and Postorious are written in.

verified (I believe it imports the testing framework from mailman.client and
thus must be ported too).

We've always said that mailman.client should be bilingual.  It should be
usable in either Python 2 or Python 3 scripts.  I took a quick look at the
current mailman.client trunk, and yes its test suite will have to be fixed,
but I will take it upon myself to do that.  If that's the only hangup, then
I'm not worried.

And I thought the 3.0 release was almost there. I must say this is rather
discouraging.

Hopefully the above provides some clarity and dis-disencouragement ;).

Cheers,
-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

On Dec 26, 2014, at 08:48 PM, Geoff Shang wrote:

On Fri, 26 Dec 2014, Pierre-Yves Chibon wrote:

 This also means that mailman 3 will pretty much only not work on
 server-oriented distro (except probably Debian that likely ships a python 3
 version, although not being involved in Debian, I do not know for sure).

It looks like the latest Fedora has Python 3.4, though I'm not sure what
version is available in the latest RedHat.  Don't RH/Fedora users have
something called Software Collections that can provide Python 3.4 even on
older versions?

Debian Wheezy(aka Stable) ships Python3, but it's Python 3.2.3, and IIRC,
Barry said Mailman3 now needs Python 3.4 in order to be fully supported.

FWIW, Debian Jessy (aka Testing/Frozen?) has Python 3.4.2.

Yep, Jessie will have 3.4, and Ubuntu has had it since Trusty Tahr (14.04
LTS).  I don't know about other distros.

There's always an intricate version dance we play, but I really think 3.4 is
the best base Python version to support.  I would be open to merge proposals
that help make it possible to run MM3 on older versions of Python 3 (but for
sure nothing before 3.2), however everything that I run already has 3.4.
Python 3.4 is 9 months old now and has seen two point releases, so while it
make take some time for some distros to catch up, I think the release is
plenty mature to use as a baseline.

Cheers,
-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Andrew Stuart]

Is there anything I can do to help with these? If there’s a task list maybe I 
can work out what I could do.

This means I must now port HyperKitty and Kittystore to Python3, check that
we don't use Py2-only libraries, etc.
And mailman.client must be ported too, since it does import stuff from
mailman for testing purposes. I think Postorious is safe, but that must be
verified (I believe it imports the testing framework from mailman.client
and thus must be ported too).


On 27 Dec 2014, at 2:38 am, Aurelien Bompard aurel...@bompard.org wrote:

 One of my top priorities has been to port Mailman 3 (core) to Python 3.
 This
 work is now complete, and ready to be merged into trunk.  No doubt bugs
 still
 lurk, but at least the entire test suite is passing.


Congrats !


* A few database columns have changed from LargeBinary to Unicode.  I have
 not
  yet figured out how to add the Alembic schema migrations to handle this,
 but
  I'll do so before the release.
 

This should be pretty easy, something like the following command should
work:
alembic -c src/mailman/config/alembic.cfg revision --autogenerate -m
python3 port

[...]  then I'll merge this to trunk and do a 3.0b5 release as a
 Python 3-only application.  Mailman 3 will not be bilingual so Python 2
 support will be dropped.
 

Wow. I am very surprised.
So we went from Python3 compatibility is not on the blocker list for
Mailman 3.0 to Mailman 3.0 will only work on Python 3.
That's quite a change.
This means I must now port HyperKitty and Kittystore to Python3, check that
we don't use Py2-only libraries, etc.
And mailman.client must be ported too, since it does import stuff from
mailman for testing purposes. I think Postorious is safe, but that must be
verified (I believe it imports the testing framework from mailman.client
and thus must be ported too).

And I thought the 3.0 release was almost there. I must say this is rather
discouraging.

Aurélien
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40supercoders.com.au

Security Policy: http://wiki.list.org/x/QIA9

___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

On Dec 27, 2014, at 08:26 AM, Andrew Stuart wrote:

Is there anything I can do to help with these? If there’s a task list maybe I
can work out what I could do.

Hi Andrew, if you're interested in helping port Postorious and HyperKitty,
I'll defer to Aurelien and Florian.

Cheers,
-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Andrew Stuart]

If it is released to version 1.0 with support for Python 2 then it will need to 
support Python 2 for the forseeable future. It’ll be deployed all over the 
place with Python 2 - when would the Python 2 be removed ever after 1.0? David 
Murray has made substantial changes to email in Py3 - supporting those in both 
2 and 3 could be an open question. A Python 3-only version 1.0 has the longer 
term gain  of never having to write mixed 2/3 code after 1.0. 



On 27 Dec 2014, at 5:48 am, Geoff Shang ge...@quitelikely.com wrote:

On Fri, 26 Dec 2014, Pierre-Yves Chibon wrote:

 This also means that mailman 3 will pretty much only not work on 
 server-oriented
 distro (except probably Debian that likely ships a python 3 version, although
 not being involved in Debian, I do not know for sure).

Debian Wheezy(aka Stable) ships Python3, but it's Python 3.2.3, and IIRC, Barry 
said Mailman3 now needs Python 3.4 in order to be fully supported.

FWIW, Debian Jessy (aka Testing/Frozen?) has Python 3.4.2.

HTH,
Geoff.

___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40supercoders.com.au

Security Policy: http://wiki.list.org/x/QIA9

___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Aurelien Bompard]

 I'm having some trouble with this because the system alembic doesn't have
 access to the mailman source tree, which it needs (you get import
 errors).  I
 tried to run alembic out of the .tox virtualenv, but that fails because the
 mailman config system needs to be initialized.


Right, I'm having issues too with the config file not being initalized.
I'll have a closer look.

Oh gosh, no!  I'm sorry I was imprecise, but I meant only that Mailman 3
 *core* would be Python 3.  I certainly would never make you guys do all
 that
 busy work without consulting you first.  So for the record, HyperKitty and
 Postorious can absolutely port on their own time frame, and that need not
 at
 all block a release of the full suite.  One of the advantages of accessing
 the
 core through the REST API is that it doesn't matter what clients like
 HyperKitty and Postorious are written in.


I'm not exclusively using the REST API though. I'm importing a couple
interfaces, mostly the archiver interface. I'm also using the custom
Message class a lot in the tests. And I'm importing a couple database types
for compatibility in my own schemas (Enum and UUID)
But I think the main problem is the import of mailman's config object in
the class that implements the IArchiver interface. I don't believe there's
another way to get the configuration.
And now that I think of it, the archiver interface will be imported by
Mailman core, and will thus run with a Python3 interpreter. As a result,
all of KittyStore must at least be Python3 compatible.

A.
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

On Dec 27, 2014, at 08:21 AM, Andrew Stuart wrote:

If it is released to version 1.0 with support for Python 2 then it will need
to support Python 2 for the forseeable future. It’ll be deployed all over the
place with Python 2 - when would the Python 2 be removed ever after 1.0?
David Murray has made substantial changes to email in Py3 - supporting those
in both 2 and 3 could be an open question. A Python 3-only version 1.0 has
the longer term gain of never having to write mixed 2/3 code after 1.0.

Exactly.  I'm highly motivated to not release the *core* under Python 2 for
the reasons you state above.  Personally speaking, I've been so immersed in
Python 3 for almost all my other projects (both work and play) that continuing
to hack in Python 2 seems like I'm taking down my dinner with pointed
sticks. ;)

After finishing the port, I think it's impractical to try to have the core
support both Python 2 and 3.  There's *a lot* of great new stuff in the Python
3 email package, and while I'm still mostly using the legacy API, I did have
to use some of the new stuff to do Subject prefix munging correctly.  I expect
that as time goes on, we'll need to adopt more of the new email APIs in order
to fix bugs, add features, and do things more efficiently.  That's not even
talking about so much of the other great new stdlib stuff available in Python
3.

Cheers,
-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

On Dec 26, 2014, at 10:32 PM, Aurelien Bompard wrote:

Right, I'm having issues too with the config file not being initalized.
I'll have a closer look.

Awesome, thanks.

I'm not exclusively using the REST API though. I'm importing a couple
interfaces, mostly the archiver interface. I'm also using the custom Message
class a lot in the tests. And I'm importing a couple database types for
compatibility in my own schemas (Enum and UUID)

I wonder if it would be possible to refactor some of this code into separate
libraries, which would reduce the surface area for bilingual support.  The
config object is probably the tricky part here, but also the Message class has
a lot less in it now that we don't have to worry about all that unicode stuff.
Really, it's essentially just the .sender and .senders properties and some
backward compatible pickle support (unnecessary atm for MM3 and probably not
much needed going forward, unless the stdlib Message object in some future
Python grows internal attributes that break unpickling - I'll certainly test
that against what will be Python 3.5).

But I think the main problem is the import of mailman's config object in the
class that implements the IArchiver interface. I don't believe there's
another way to get the configuration.

But there could be!  I took a very quick look at HK (but not KittyStore) and
it doesn't look like you need much.  What if I added a REST API to access the
core's system configuration settings?  They would be read-only, and I would
only start with the minimum of what HK needs.  I think it's probably dangerous
in the long term to expect the core's internal config API to remain stable, so
accessing that through REST would provide the guarantees you need, in addition
to decoupling HK from the core.

And now that I think of it, the archiver interface will be imported by
Mailman core, and will thus run with a Python3 interpreter. As a result, all
of KittyStore must at least be Python3 compatible.

Yep, I missed that.  Is it possible to feed HK messages out-of-process?
E.g. via the command line?  If so, then we write HK's IArchiver implementation
to make those out of process calls, and we can even pull that bit into the
core.

I would be happy to help with that.

Cheers,
-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Aurelien Bompard]

 But there could be!  I took a very quick look at HK (but not KittyStore)
 and
 it doesn't look like you need much.  What if I added a REST API to access
 the
 core's system configuration settings?


That would work.


 Yep, I missed that.  Is it possible to feed HK messages out-of-process?
 E.g. via the command line?


Not now. Would an LMTP server be appropriate? If so, could Mailman's
internal LMTP server be split off into a Py2-compatible library?
What protocol would be best suited in your opinion?

A.
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Stephen J. Turnbull]

Aurelien Bompard writes:
  Barry writes:

   One of the advantages of accessing the core through the REST API
   is that it doesn't matter what clients like HyperKitty and
   Postorious are written in.

  I'm not exclusively using the REST API though. I'm importing a
  couple interfaces, mostly the archiver interface.

This was a design mistake, I think.  An IArchiver really needs to be
in core for two reasons: first, it needs to generate a permalink and
attach it to the message as distributed.  Second, it needs to
associate that permalink with the message so the real archiver
process will do set that link correctly.  For a production-quality
archiver, that's *all* it should do.  The archiver itself should be a
separate process, receiving the message and permalink data by IPC.

@Barry: maybe rename IArchiver to IPermalinker? ;-)

Remember, Mailman core is going to be distributing Internet mail.
Except in the case where 100.00% of the users are on one host, that's
going to be the bottleneck on message processing.  Archiving simply
does not need to be fast.  The archiver can implement LMTP even,
although that would be overkill if we didn't already have an LMTP
server.  The simplest approach would be to simply put the
Archive-Permalink header in the message and stream it to the archiver
which would parse it out.

  I'm also using the custom Message class a lot in the tests.

Can't avoid that with Python 2, I guess, but using Message will be
*so* much less painful with Python 3.

  But I think the main problem is the import of mailman's config
  object in the class that implements the IArchiver interface. I
  don't believe there's another way to get the configuration.

If you need that configuration (which, come to think of it, you
probably do, at least parts of it), you could have a private protocol
for communicating it as metadata (a message header or as metadata in
the stream).

  And now that I think of it, the archiver interface will be imported
  by Mailman core, and will thus run with a Python3 interpreter. As a
  result, all of KittyStore must at least be Python3 compatible.

In the long run (ie, when nobody who's anybody uses Python 2 at all) I
think everybody would be happier if you refactor to keep KittyStore at
arm's length from Mailman core.

___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

On Dec 26, 2014, at 11:02 PM, Aurelien Bompard wrote:

 But there could be!  I took a very quick look at HK (but not KittyStore)
 and it doesn't look like you need much.  What if I added a REST API to
 access the core's system configuration settings?

That would work.

It turned out to be almost ridiculously easy, so that probably means it's a
good idea wink.  Quoting from the documentation:


System configuration


The entire system configuration is available through the REST API.  You can
get a list of all defined sections.

 dump_json('http://localhost:9001/3.0/system/configuration')
http_etag: ...
sections: ['antispam', 'archiver.mail_archive', 'archiver.master', ...

You can also get all the values for a particular section.

 dump_json('http://localhost:9001/3.0/system/configuration/mailman')
default_language: en
email_commands_max_lines: 10
filtered_messages_are_preservable: no
http_etag: ...
layout: testing
noreply_address: noreply
pending_request_life: 3d
post_hook:
pre_hook:
sender_headers: from from_ reply-to sender
site_owner: nore...@example.com

Dotted section names work too, for example, to get the French language
settings section.

 dump_json('http://localhost:9001/3.0/system/configuration/language.fr')
charset: iso-8859-1
description: French
enabled: yes
http_etag: ...


-end quote-

Of course, these configuration resources are read-only.  Also, I am
deprecating the `/api/system` resource path; for backward compatibility this
will still return the system versions, but you'll notice that the self_link
now points to `api/system/versions` which is the new canonical resource
path.  I'll remove the old path when the API version gets rev'd.

This makes a good point though: I'm not outright discouraging applications
from using core as a library, and in fact that was part of the point of making
everything importable via the `mailman` package.  But I'm far from ready to
make any guarantees of stability for the internal API, including interfaces.
The REST API however I *am* making guarantees about, as evidenced by using an
API version at the root of the resource tree.

Note that configuration values are verbatim strings.  You might need to
convert the value to some typed data, e.g. [mailman]pending_request_lifetime
has a value of '3d' which is shorthand for a timedelta.  Clients must convert
those data types themselves.  Similarly for ints, booleans, etc.  Multiline or
whitespace separated strings values must be .split() by the client.

But you *do* have access to all configuration settings now, although only on
the py3 branch atm.

Cheers,
-Barry


pgpOhlPV6Y0aG.pgp
Description: OpenPGP digital signature
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

On Dec 27, 2014, at 12:48 PM, Stephen J. Turnbull wrote:

Let's please take some care to think about this.  ISTM this is like
the Nth time we've bumped into you need to be core to access config
data.

No more!  See my other follow up.  And there's no restriction on values.
Sometimes generic is easier than specific (shouldn't that be a Zen of Python?
:).

Hopefully Aurelien can confirm that this will be enough for HK, but of course
Postorious can use it now too.

Cheers,
-Barry


pgpuNGldLwvBE.pgp
Description: OpenPGP digital signature
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Barry Warsaw]

On Dec 27, 2014, at 12:42 PM, Stephen J. Turnbull wrote:

Remember, Mailman core is going to be distributing Internet mail.
Except in the case where 100.00% of the users are on one host, that's
going to be the bottleneck on message processing.  Archiving simply
does not need to be fast.  The archiver can implement LMTP even,
although that would be overkill if we didn't already have an LMTP
server.  The simplest approach would be to simply put the
Archive-Permalink header in the message and stream it to the archiver
which would parse it out.

Remember too that in MM3, messages only get fed to the registered IArchiver
interfaces by a separate archive runner.  So they aren't a bottleneck for
delivery to the user, but on heavily trafficked sites, they can potential
consume a lot of resources if the archiver is local and relatively
inefficient.

I tend to agree that a good design for any archiver is to be able to accept
messages over an IPC channel.  A site may for example want to run the core on
one system and HK on another system (e.g. separate VMs perhaps).  This would
only really be possible if the core can feed HK messages over a configurable
IPC.  As I mentioned, I think LMTP *could* work, but REST (inside HK) could
work too.  Aurelien, what do you think?

  I'm also using the custom Message class a lot in the tests.

Can't avoid that with Python 2, I guess, but using Message will be
*so* much less painful with Python 3.

See my other follow up.  I don't think there's much in the py3 branche's
Message class that is going to be super helpful to HK.  Even the core *could*
use the standard email.message.Message class with a couple of utility
functions, but it's just a little easier to add a few properties and methods
in the subclasses.

In the long run (ie, when nobody who's anybody uses Python 2 at all) I
think everybody would be happier if you refactor to keep KittyStore at
arm's length from Mailman core.

Agreed, with of course the caveat that we'll need a thin HK IArchiver
implementation in the core to generate the permalink and communicate with HK
over IPC.  Generally we want the permalink to be able to be generated without
direction communication with the archiver (see the motivation for
X-Message-ID-Hash), but if the core *has* to talk to HK to generate the
permalink, then I don't think an LMTP channel will work.  In that case REST or
some homegrown protocol may be the answer.

Cheers,
-Barry


pgpB8qUmdPh_A.pgp
Description: OpenPGP digital signature
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Stephen J. Turnbull]

Barry Warsaw writes:

  No more!  See my other follow up.  And there's no restriction on
  values.  Sometimes generic is easier than specific (shouldn't that
  be a Zen of Python?  :).

Uh-oh.  Looks like somebody is using EggNog programming style!  Or are
you just high on life? ;-)

___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Stephen J. Turnbull]

Barry Warsaw writes:

  Remember too that in MM3, messages only get fed to the registered
  IArchiver interfaces by a separate archive runner.  So they aren't
  a bottleneck for delivery to the user, but on heavily trafficked
  sites, they can potential consume a lot of resources if the
  archiver is local and relatively inefficient.

I'm talking about total load on the server host, not load on the
Mailman subsystem.  So I don't think the Mailman-to-archive function
will consume many resources compared to delivery to subscribers if
there are any remote users at all.  A local archiver communicates at
CPU-to-disk speed basically once or maybe twice as I understand it.
The MTA resources for queuing alone will exceed and probably overwhelm
this.  Then there are the multiple Mailman queues, etc, etc.

Of course the *other* side of the archiver (the client access to the
message store) can be extremely resource consuming.  I'm just saying
that in the grand scheme of message distribution (including to the
archiver), the efficiency of a local archiver is not going to be a
bottleneck.

  In the long run (ie, when nobody who's anybody uses Python 2 at
  all) I think everybody would be happier if you refactor to keep
  KittyStore at arm's length from Mailman core.
  
  Agreed, with of course the caveat that we'll need a thin HK
  IArchiver implementation in the core to generate the permalink and
  communicate with HK over IPC.  Generally we want the permalink to
  be able to be generated without direction communication with the
  archiver (see the motivation for X-Message-ID-Hash),

By the way, I would say to adopt modern IETF practice here and drop
the X- (in practice collisions are rare while the annoyance of
fixing platforms to use the standardized name is frequent), and
include the algorithm in the name.  Eg, Message-ID-MD5 or
Hashed-Message-ID-MD5.  Or we could use the List-* namespace.

We should do this while we still can. :-)  If you want I can try to
write an RFC to make it official.

  but if the core *has* to talk to HK to generate the permalink,

I personally don't think that is a good idea, but see below.
 
  then I don't think an LMTP channel will work.

The only reason I can think of is that you want to check that the
permalink isn't already occupied (that's the only thing HyperKitty
proper knows that can't be computed the same way in the IArchiver as
in HyperKitty proper AFAICS), and that can be implemented as follows

MailmanLHLO mailman-host
HyperKitty 250 OK
MailmanMAIL FROM Mailman@mailman-host
HyperKitty 250 OK
MailmanRCPT TO permalink-variable-part@archiver-host
HyperKitty 553 Permalink already occupied
MailmanRCPT TO new-permalink-variable-part@archiver-host
HyperKitty 250 OK
MailmanDATA
HyperKitty 354 Go for it!

and so on.  I don't think this even violates the spirit of the LMTP
protocol, but it certainly conforms to the letter as long as permalink
variable parts are valid email localparts.  (One could quibble about
which 5xx response to give.  AFAICS only 551 user not local is out.)

My own preference is for a permalink that can be computed from the
originator header data (author, recipients, date, message ID, subject)
by anyone with access to the message, and that means you need the
archive server to be able to deal gracefully with collisions.  (In
practice message IDs are not perfect UUIDs, although they're very
close, and some messages don't have them or have different ones
assigned by mediating hosts at arrival at multiple recipients.)

Steve
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] Python 3

[Florian Fuchs]

Am 27.12.2014 um 05:18 schrieb Barry Warsaw:
 I tend to agree that a good design for any archiver is to be able to accept
 messages over an IPC channel.  A site may for example want to run the core on
 one system and HK on another system (e.g. separate VMs perhaps).  This would
 only really be possible if the core can feed HK messages over a configurable
 IPC.  As I mentioned, I think LMTP *could* work, but REST (inside HK) could
 work too.  Aurelien, what do you think?

How about a third option, a generic pub/sub or event archiver,
implemented in py3? It would meet all the above criteria (archivers on
other systems, no dependency on py3 -- or Python for that matter).

We could start supporting one backend (zeromq for instance) and maybe
add more later.


Flo
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

[Mailman-Developers] Python 3

[Barry Warsaw]

One of my top priorities has been to port Mailman 3 (core) to Python 3.  This
work is now complete, and ready to be merged into trunk.  No doubt bugs still
lurk, but at least the entire test suite is passing.  My intent is to merge
this to trunk and do another beta release before the end of the year.

If you want to take a look at the branch, try this:

$ bzr branch lp:~barry/mailman/py3

The diff is available online via the merge proposal:

https://code.launchpad.net/~barry/mailman/py3/+merge/245313

It's a big diff.

$ bzr diff . --old ../3.0 | diffstat -s
 166 files changed, 1881 insertions(+), 1538 deletions(-)

Python 3.4 will be the minimum required version.  (It make work with earlier
Python 3's but I will not officially support anything earlier.)

There are a few things to keep an eye out for, especially if you are currently
experimenting with Mailman 3 in production:

* A few database columns have changed from LargeBinary to Unicode.  I have not
  yet figured out how to add the Alembic schema migrations to handle this, but
  I'll do so before the release.

* The internal format of the requests table has changed.  While the column
  type is a string, the internal format was a sort of home baked pickle-like
  format.  This turned out to be incompatible with Python 3, due to some
  assumptions made about the format.  I've changed this to using a JSON
  representation, which is safe and platform independent.  This means that any
  held requests (subscriptions, unsubscriptions, held posts) must be cleared
  before switching to the Python 3 code.  I don't think it's worth trying to
  migrate this internal format.

* The new email API in Python 3 is really wonderful, but I am mostly not yet
  taking advantage of it.  The one area that required a significant rewrite is
  in the Subject prefix munging code.  Some Python 2 behavior is different in
  Python 3 and the old algorithms had to be rewritten.  I've added a lot of
  tests, but there are probably bugs in the new code, especially when various
  mixed charsets appear in the Subject header.

Please do test, review, grouse, compliment, curse, rejoice, and enjoy.   Over
the holidays I'll spend some time cleaning a few things up (especially the
documentation), then I'll merge this to trunk and do a 3.0b5 release as a
Python 3-only application.  Mailman 3 will not be bilingual so Python 2
support will be dropped.

Cheers,
-Barry


pgp9nV_cNvaOP.pgp
Description: OpenPGP digital signature
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

[Mailman-Developers] Python 3

[Stephen J. Turnbull]

Congratulations!  Wow, Merry Christmas[1] to us!

Barry Warsaw writes:

  One of my top priorities has been to port Mailman 3 (core) to Python 3.  This
  work is now complete, and ready to be merged into trunk.

Footnotes: 
[1]  Season to taste.

___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] python 3

[Stephen J. Turnbull]

On Sat, Mar 24, 2012 at 12:15 AM, Barry Warsaw ba...@list.org wrote:
 Not all of that will be needed.  For example, I do want to eventually get rid
 of zc.buildout, zope.testing, and zope.testrunner, so some of those
 dependencies will go away as a result of that.

But presumably new dependencies will be added to support those
functions, and while Python 3 support is an important desideratum,
for replacements, I don't think that it's a good idea to let it override
quality in the implementation of dependencies.
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] python 3

[Barry Warsaw]

On Mar 24, 2012, at 03:41 PM, Stephen J. Turnbull wrote:

On Sat, Mar 24, 2012 at 12:15 AM, Barry Warsaw ba...@list.org wrote:
 Not all of that will be needed.  For example, I do want to eventually get rid
 of zc.buildout, zope.testing, and zope.testrunner, so some of those
 dependencies will go away as a result of that.

But presumably new dependencies will be added to support those
functions, and while Python 3 support is an important desideratum,
for replacements, I don't think that it's a good idea to let it override
quality in the implementation of dependencies.

Agreed.  However, In these cases, there are already compelling technical
reasons to find replacements.

-Barry


signature.asc
Description: PGP signature
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

[Mailman-Developers] python 3

[Andrea Crotti]

Someone told me that I should poke Barry about Python3 support, so here
I am ;)

It's clearly something that can be done really soon, but I think it's
worth to start thinking about it.

In particular I noticed that lazr.config have been untouched since 2009,
which makes me think that is less likely that they will be ported to
Python 3.

Is it a valid assumption or there is a porting plan?

Are there any other blocking factors in general from libraries or other
issues??
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] python 3

[Barry Warsaw]

On Mar 23, 2012, at 08:19 PM, Andrea Crotti wrote:

Someone told me that I should poke Barry about Python3 support, so here
I am ;)

It's clearly something that can be done really soon, but I think it's
worth to start thinking about it.

Probably you meant s/can/can't/ ... but still, it may not be so dire.  It
won't happen for Mailman 3.0 but I think it should definitely be a goal for
Mailman 3.1 (which will *not* take another 4 years :).

I've tried to be very careful in writing Mailman 3 so that *its* code should
be relatively easy to port to Python 3.  No doubt there will be gotchas, but I
am definitely keeping that in mind as I go.

In particular I noticed that lazr.config have been untouched since 2009,
which makes me think that is less likely that they will be ported to
Python 3.

Is it a valid assumption or there is a porting plan?

Are there any other blocking factors in general from libraries or other
issues??

Really, we need to get our dependency stack ported.  I'm committed to making
Mailman 3 a Python 3 application as soon as it's possible.  Martin von Loewis
was working on porting Storm at Pycon, which obviously is a critical
component, but I don't think he quite finished.

We need restish.io ported, along with its dependency stack, as well as several
of the still-unported Zope libraries, and *their* dependency stacks.  By
looking in buildout's eggs directory, you can get a good sense of what is
currently required:

flufl.bounce-2.1-py2.7.egg/
flufl.enum-3.3.1-py2.7.egg/
flufl.i18n-1.1-py2.7.egg/
flufl.lock-2.2-py2.7.egg/
flufl.password-1.2-py2.7.egg/
httplib2-0.7.4-py2.7.egg/
lazr.config-1.1.3-py2.7.egg/
lazr.delegates-1.2.0-py2.7.egg/
lazr.smtptest-1.3-py2.7.egg/
mimeparse-0.1.3-py2.7.egg/
restish-0.12.1-py2.7.egg/
setuptools-0.6c12dev_r88846-py2.7.egg/
six-1.1.0-py2.7.egg/
storm-0.19-py2.7-linux-x86_64.egg/
WebOb-1.2b3-py2.7.egg/
z3c.recipe.scripts-1.0.1-py2.7.egg/
z3c.recipe.tag-0.4.1-py2.7.egg/
zc.buildout-1.5.2-py2.7.egg/
zc.recipe.egg-1.3.2-py2.7.egg/
zc.recipe.testrunner-1.4.0-py2.7.egg/
zope.component-3.12.0-py2.7.egg/
zope.configuration-3.8.0-py2.7.egg/
zope.event-3.5.1-py2.7.egg/
zope.exceptions-3.6.1-py2.7.egg/
zope.i18nmessageid-3.6.1-py2.7-linux-x86_64.egg/
zope.interface-3.8.0-py2.7-linux-x86_64.egg/
zope.schema-4.0.1-py2.7.egg/
zope.testing-3.10.3-py2.7.egg/
zope.testrunner-4.0.4-py2.7.egg/

The flufl packages are of course ported. :)

Not all of that will be needed.  For example, I do want to eventually get rid
of zc.buildout, zope.testing, and zope.testrunner, so some of those
dependencies will go away as a result of that.

The best way to help is to work with those upstreams to get official Python 3
support in their libraries.  If you're looking for guidance, start with storm,
WebOb, restish, lazr.*, and the unported zope.* libraries (minus .testing and
.testrunner, and possibly .i18nmessageid). 

The email6 package availability will probably help us a lot too.  Help
R. David Murray on that package, or just pay him money to finish it. :)

Cheers,
-Barry
___
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9