[Mailman-Users] AES encryption and Resent-Message-ID

[Lindsay Haisley]

Here are a few tidbits pursuant to putting an encrypted copy of a list
post recipient in a "Resent-Message-ID" header, as Stephen Turnbull
suggested.  There are four parts:

1.  A patch to SMTPDirect.py

2.  A secret key entry in mm_cfg.py

3.  A utility, ~mailman/bin/aes_genkey, to manage key generation

4.  A handler module to do encryption, decryption and key generation -
AEScrypt.py

Here's the patch to SMTPDirect.py (mm 2.1.15):

--- SMTPDirect.py.orig  2012-06-17 17:16:25.0 -0500
+++ SMTPDirect.py   2012-06-18 23:29:58.0 -0500
@@ -43,6 +43,7 @@
 from email.Utils import formataddr
 from email.Header import Header
 from email.Charset import Charset
+import AEScrypt
 
 DOT = '.'
 
@@ -307,6 +308,11 @@
  'host'   : DOT.join(rdomain),
  }
 envsender = '%s@%s' % ((mm_cfg.VERP_FORMAT % d), DOT.join(bdomain))
+try:
+skey = AEScrypt.encrypt(recip)
+msgcopy["Resent-Message-ID"] = skey + "@" + DOT.join(bdomain)
+except:
+pass
 if mlist.personalize == 2:
 # When fully personalizing, we want the To address to point to the
 # recipient, not to the mailing list


mm_cfg.py requires an AES key in AES_SECRET_KEY.  Without this, the
Resent-Message-ID header isn't inserted in outgoing posts and everything
works as it does without this stuff.

The AES key can be generated with aes_genkey which lives in ~mailman/bin
and works like other scripts in this directory.  Running it with -a
appends AES_SECRET_KEY to mm_cfg.py with an appropriate comment.

~mailman/bin/aes_genkey
---
#! /usr/bin/python
"""Generate an AES secret key on stdout for inclusion in mm_cfg.py as
AES_SECRET_KEY.

Usage: %(PROGRAM)s [options]

Where:
-a append AES secret key to mm_cfg.py

-h / --help
Print help and exit.
"""

import sys
import getopt
import os
import paths
from Mailman import mm_cfg
from Mailman.Handlers import AEScrypt
from Mailman.i18n import _

def usage(code, msg=''):
if code:
fd = sys.stderr
else:
fd = sys.stdout
print >> fd, _(__doc__)
if msg:
print >> fd, msg
sys.exit(code)

def main():
try:
opts, args = getopt.getopt(sys.argv[1:], 'ha', ['help'])
except getopt.error, msg:
usage(1, msg)

for opt, arg in opts:
if opt in ('-h', '--help'):
usage(0)
if opt in ('-a',):
try:
f = mm_cfg.AES_SECRET_KEY
print "AES secret key already in mm_cfg.py"
return(0)
except:
mm = open(os.getenv("HOME") + "/Mailman/mm_cfg.py", "a")
ktxt = """
# Experimental address encryption key.  To renew this key,
# delete AES_SECRET_KEY and run 'aes_keygen -a' and restart
# Mailman.
AES_SECRET_KEY = '%s'
""" % (AEScrypt.genkey(),)
mm.write(ktxt)
mm.close()
print "AES secret key added to mm_cfg.py"
return(0)

print AEScrypt.genkey()

if __name__ == '__main__':
sys.exit(main())


The final part is the encryption/decryption module, AEScrypt.py  For
this to work the python-crypto ("Crypto") package must be installed.

~mailman/Mailman/Handlers/AEScrypt.py
-
from Crypto.Cipher import AES
from Crypto.Util import randpool
from Mailman import mm_cfg
import base64

block_size = 16
key_size = 32
mode = AES.MODE_CBC
try:
key_string = mm_cfg.AES_SECRET_KEY  
except:
pass

def genkey():
key_bytes = randpool.RandomPool(512).get_bytes(key_size)
key_string = base64.urlsafe_b64encode(str(key_bytes))
return key_string   

def encrypt(plain_text):
pad = block_size - len(plain_text) % block_size
data = plain_text + pad * chr(pad)
iv_bytes = randpool.RandomPool(512).get_bytes(block_size)
encrypted_bytes = iv_bytes + 
AES.new(base64.urlsafe_b64decode(key_string), mode, iv_bytes).encrypt(data)
return base64.urlsafe_b64encode(str(encrypted_bytes))

def decrypt(cypher_text):
key_bytes = base64.urlsafe_b64decode(key_string)
encrypted_bytes = base64.urlsafe_b64decode(cypher_text)
iv_bytes = encrypted_bytes[:block_size]
encrypted_bytes = encrypted_bytes[block_size:]
plain_text = AES.new(key_bytes, mode, iv_bytes).decrypt(encrypted_bytes)
pad = ord(plain_text[-1])
return plain_text[:-pad]


The Resent-Message-ID header has the domain name of the server host
appended to it and this will need to be stripped before decrypting the
address string.  Something like 'crypt, dn = full_header.split("@")'
will pull the encrypted address from the header.  A withlist script can
easily extract the plain text content of the encrypted string.

I hope this helps someone.

-- 
Lindsay Haisley   | "Real programmers use butterflies"
FMP Computer Services 

Re: [Mailman-Users] AOL redacts user addresses even with VERP and full personalization enabled

[Lindsay Haisley]

On Mon, 2012-06-18 at 17:58 -0500, Lindsay Haisley wrote:
> FWIW, pursuant to Stephen's comments re. using encryption rather than
> hashing for passing recipient addresses in headers, I've attached a
> short Python script which puts short strings of data, such as an email
> address, into an AES cipher.

It looks as if the attachment got stripped.  Here's the script, based on
information at
http://www.codekoala.com/blog/2009/aes-encryption-python-using-pycrypto/


class AEScrypt:
from Crypto.Cipher import AES
from Crypto.Util import randpool
import base64

block_size = 16
key_size = 32
mode = AES.MODE_CBC

def genkey(self):
key_bytes = 
self.randpool.RandomPool(512).get_bytes(self.key_size)
key_string = self.base64.urlsafe_b64encode(str(key_bytes))
return key_string   

def encrypt(self, plain_text, key_string):
pad = self.block_size - len(plain_text) % self.block_size
data = plain_text + pad * chr(pad)
iv_bytes = 
self.randpool.RandomPool(512).get_bytes(self.block_size)
encrypted_bytes = iv_bytes + 
self.AES.new(self.base64.urlsafe_b64decode(key_string), 
self.mode, iv_bytes).encrypt(data)
return self.base64.urlsafe_b64encode(str(encrypted_bytes))

def decrypt(self, cypher_text, key_string):
key_bytes = self.base64.urlsafe_b64decode(key_string)
encrypted_bytes = self.base64.urlsafe_b64decode(cypher_text)
iv_bytes = encrypted_bytes[:self.block_size]
encrypted_bytes = encrypted_bytes[self.block_size:]
plain_text = self.AES.new(key_bytes, self.mode, 
iv_bytes).decrypt(encrypted_bytes)
pad = ord(plain_text[-1])
return plain_text[:-pad]

-- 
Lindsay Haisley   | "In an open world, who needs  
FMP Computer Services |Windows or Gates"
512-259-1190  |
http://www.fmp.com|

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] AOL redacts user addresses even with VERP and full personalization enabled

[Lindsay Haisley]

On Mon, 2012-06-18 at 14:59 -0400, David wrote:
> In terms of privacy, as list admins we already have the member's
> information. All we are doing in this case is helping that member stop
> receiving messages they obviously no longer wish to receive. This is
> clearly not an invasion of privacy (especially with a properly
> encrypted implementation). It is a service to the individual (and to
> the entire list membership and even the Internet as a whole, I think).

Dave, you're spot-on in this assessment, and this is the way I run my
business.  Unfortunately, the Internet is no longer the kinder, gentler
network it was 15, or even 10 years ago.  In terms of an effective and
progressive attitude toward customer service and satisfaction, AOL's
position is 180 degrees counterintuitive and makes NO sense whatsoever.
It only makes sense in terms of butt-covering!  In that context, it's
totally logical.  AOL has for years, perhaps always, been infamous for
the lousy quality of their email service.

FWIW, pursuant to Stephen's comments re. using encryption rather than
hashing for passing recipient addresses in headers, I've attached a
short Python script which puts short strings of data, such as an email
address, into an AES cipher.  This could be folded into the Mailman
handlers and AES_SECRET_KEY could be put into mm_cfg.py.  Hacks to
SMTPDirect.py to incorporate an encrypted cipher of the recipient
address could make use if it.  I believe all the Python modules it uses
are standard issue with the distribution.

-- 
Lindsay Haisley   | SUPPORT NETWORK NEUTRALITY
FMP Computer Services | --
512-259-1190  | Boycott Yahoo, RoadRunner, AOL
http://www.fmp.com| and Verison
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Personalisation: how long does it take?

[Terry Earley]

We are using postfix on the same server as our MTA.

Terry

On Mon, Jun 18, 2012 at 4:32 PM, Andrew Hodgson wrote:

> Terry Earley wrote:
>
> >I should also mention that this test was with DKIM signing and the number
> of members was between 2100 and 2200.
>
> Thanks, something is wrong with the setup or the VPS I use is not running
> efficiently.  What MTA are you using? I am on Exim4.
>
> Andrew.
>
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Personalisation: how long does it take?

[Brad Knowles]

On Jun 18, 2012, at 3:32 PM, Andrew Hodgson wrote:

> Thanks, something is wrong with the setup or the VPS I use is not running 
> efficiently.  What MTA are you using? I am on Exim4.

Postfix has some out-of-the-box settings that make it particularly well suited 
for the role of being an MTA for a mailing list manager like Mailman.  With 
work, you can probably get a configuration for Exim that will perform nearly as 
fast for the same size of list, but Exim is designed primarily for other 
purposes than maximum performance.  As the list grows larger, the performance 
gap between a well-tuned postfix configuration and a well-tuned Exim 
configuration is likely to grow.

On the other hand, some people will find it easier to administer the Exim 
configuration than they would postfix.


If ten minutes to send out your list is actually acceptable for you, then don't 
worry that someone else can get their entire list sent out in 120 seconds, even 
if they're the same size and they're doing all the same crypto operations.  You 
need what is good enough for you, and you don't need to be comparing yourself 
to someone else.

--
Brad Knowles 
LinkedIn Profile: 

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Personalisation: how long does it take?

[Andrew Hodgson]

Terry Earley wrote:

>I should also mention that this test was with DKIM signing and the number of 
>members was between 2100 and 2200.

Thanks, something is wrong with the setup or the VPS I use is not running 
efficiently.  What MTA are you using? I am on Exim4.

Andrew. 
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] AOL redacts user addresses even with VERP and full personalization enabled

[Thomas Hochstein]

Lindsay Haisley schrieb:

> So what would be the implications of hacking an extra header into
> outgoing posts on lists for which personalization is enabled, say
> "X-Subdata", with said header containing a hash of the subscriber
> address to which the post is directed?

AOL ist actually recommending something like that (or adding the user
name somewhere in a way not looking like a mail addeess) - which makes
the whole thing even more absurd.

-thh
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Personalisation: how long does it take?

[Terry Earley]

I should also mention that this test was with DKIM signing and the number
of members was between 2100 and 2200.

Terry

On Mon, Jun 18, 2012 at 2:24 PM, Terry Earley  wrote:

> Our own experience sending to 2100 users is that without personalization a
> message went out in 2 seconds. When we applied personalization and VERP,
> that test message went out in 120 seconds, which was acceptable to us.
>
> Obviously, there are many factors affecting speed of delivery.
> Personalization is just one. What you did was best. that is to test with
> your own environment.
>
> Terry Earley
>
> FitEyes
>
>
>
> On Mon, Jun 18, 2012 at 1:32 PM, Andrew Hodgson 
> wrote:
>
>> Hi,
>>
>> Just wondering how long in the real world it takes to get all messages
>> from a personalised list sent out and processed by the MTA on the same
>> machine as Mailman?  List is around 750 members.  I did run this once and
>> it took over 10 minutes to get all the messages delivered through the MTA,
>> I am partially wondering whether that is because I am DKIM signing each
>> message?  If I use standard delivery the processing time is a lot shorter
>> (around a minute).
>>
>> Just wondering,
>> Andrew.
>> --
>> Mailman-Users mailing list Mailman-Users@python.org
>> http://mail.python.org/mailman/listinfo/mailman-users
>> Mailman FAQ: http://wiki.list.org/x/AgA3
>> Security Policy: http://wiki.list.org/x/QIA9
>> Searchable Archives:
>> http://www.mail-archive.com/mailman-users%40python.org/
>> Unsubscribe:
>> http://mail.python.org/mailman/options/mailman-users/terry%40fiteyes.com
>>
>
>
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] Personalisation: how long does it take?

[Terry Earley]

Our own experience sending to 2100 users is that without personalization a
message went out in 2 seconds. When we applied personalization and VERP,
that test message went out in 120 seconds, which was acceptable to us.

Obviously, there are many factors affecting speed of delivery.
Personalization is just one. What you did was best. that is to test with
your own environment.

Terry Earley

FitEyes


On Mon, Jun 18, 2012 at 1:32 PM, Andrew Hodgson wrote:

> Hi,
>
> Just wondering how long in the real world it takes to get all messages
> from a personalised list sent out and processed by the MTA on the same
> machine as Mailman?  List is around 750 members.  I did run this once and
> it took over 10 minutes to get all the messages delivered through the MTA,
> I am partially wondering whether that is because I am DKIM signing each
> message?  If I use standard delivery the processing time is a lot shorter
> (around a minute).
>
> Just wondering,
> Andrew.
> --
> Mailman-Users mailing list Mailman-Users@python.org
> http://mail.python.org/mailman/listinfo/mailman-users
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Security Policy: http://wiki.list.org/x/QIA9
> Searchable Archives:
> http://www.mail-archive.com/mailman-users%40python.org/
> Unsubscribe:
> http://mail.python.org/mailman/options/mailman-users/terry%40fiteyes.com
>
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] AOL redacts user addresses even with VERP and full personalization enabled

[Brad Knowles]

On Jun 18, 2012, at 12:06 PM, Larry Stone wrote:

> And the problem that I'm trying to fix is that their user has violated MY TOS 
> regarding reporting list mail (that they subscribed to) as spam. That AOL 
> sent their Feedback Loop message to me is therefore part of the violation of 
> my terms. So whose terms ends up governing when they're in conflict?

When you sign up for the feedback loop, you do so under the TOS of the feedback 
loop.  If their user violates your TOS by reporting your list traffic as spam, 
that doesn't change the TOS of the feedback loop that you signed up for.

Two lefts make a U-turn, not a right.  ;-)

> Personally, I'm not going to worry about it. I'll use them as best I can to 
> unsubscribe and server ban the offending subscriber. As I said, that AOL user 
> has violated my terms and I am entitled to deal with that violation. If AOL 
> were to ever call me on it, I'll worry about that then.

On that subject, I agree with you.

--
Brad Knowles 
LinkedIn Profile: 

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

[Mailman-Users] Personalisation: how long does it take?

[Andrew Hodgson]

Hi,

Just wondering how long in the real world it takes to get all messages from a 
personalised list sent out and processed by the MTA on the same machine as 
Mailman?  List is around 750 members.  I did run this once and it took over 10 
minutes to get all the messages delivered through the MTA, I am partially 
wondering whether that is because I am DKIM signing each message?  If I use 
standard delivery the processing time is a lot shorter (around a minute).

Just wondering,
Andrew.
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] AOL redacts user addresses even with VERP and full personalization enabled

[Lindsay Haisley]

On Mon, 2012-06-18 at 12:05 -0700, Brad Knowles wrote:
> Uh, trust me -- you really don't want to get into the discussion of
> creating new SMTP protocol enhancements.  I was on the DRUMS WG.  You
> really, really don't want to go there.
> 
VERP is not an SMTP protocol, but a MTA property supported by many
modern MTAs such as Courier.  It relies on the fact that MTAs which
support it treat user-somed...@example.com as an email address extension
of u...@example.com.

Pardon me if I'm missing something here.

-- 
Lindsay Haisley   | "Never expect the people who caused a problem
FMP Computer Services |  to solve it." - Albert Einstein
512-259-1190  |
http://www.fmp.com|

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] AOL redacts user addresses even with VERP and full personalization enabled

[Larry Stone]

On Sun, 17 Jun 2012, Brad Knowles wrote:

In fact, when you sign up for the AOL Feedback Loop (as I did years ago 
for the lists hosted at python.org), the instructions explicitly state 
that you may not use any information they give you to determine who the 
affected user is -- they're simply telling you that you have a problem 
that you need to fix on your end to keep spam from being generated in 
the first place, and it is not relevant which AOL user is complaining.


And the problem that I'm trying to fix is that their user has violated MY 
TOS regarding reporting list mail (that they subscribed to) as spam. That 
AOL sent their Feedback Loop message to me is therefore part of the 
violation of my terms. So whose terms ends up governing when they're in 
conflict?


Personally, I'm not going to worry about it. I'll use them as best I can 
to unsubscribe and server ban the offending subscriber. As I said, that 
AOL user has violated my terms and I am entitled to deal with that 
violation. If AOL were to ever call me on it, I'll worry about that then.


-- Larry Stone
   lston...@stonejongleux.com
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] AOL redacts user addresses even with VERP and full personalization enabled

[David]

On Mon, Jun 18, 2012 at 2:44 PM, Lindsay Haisley wrote:

> On Mon, 2012-06-18 at 13:04 -0400, Tanstaafl wrote:
> > On 2012-06-18 12:22 PM, Lindsay Haisley  wrote:
> > > Doing this as a custom hack helps.  If this were implemented as a
> > > Mailman standard option then word might indeed get back to them about
> > > it.  Using Resent-Message-ID as a header name is a clever idea.
> >
> > I'd also argue that since this is not AOL specific but is a generic way
> > for a mail system admin to control his own server, and AOL cannot
> > dictate what you add to your own headers on your own messages, why not
> > make it part of mailman official, with appropriate warnings about some
> > brain-dead (probably unenforcable and possibly even illegal) limitations
> > by certain clueless providers?
>
> I agree.  Stephen Turnbull points out that using reversible encryption
> with a secret key would be more secure from the point of view of
> restricting 3rd party knowledge of the unencrypted/unhashed data.  A
> secret key could be kept per-list or per-site.  The ability to securely
> track recipient information (or any information) across a list
> distribution, or across a non-delivery bounce might be very useful.
>
> It might be very convenient to have what one might call EVERP, where the
> recipient address is encrypted into the envelope sender address, as an
> alternative choice to Mailman's VERP implementation.
>

This whole thread is a good and interesting discussion. Anything along
these lines sounds like a great suggestion  to me.

In terms of privacy, as list admins we already have the member's
information. All we are doing in this case is helping that member stop
receiving messages they obviously no longer wish to receive. This is
clearly not an invasion of privacy (especially with a properly encrypted
implementation). It is a service to the individual (and to the entire list
membership and even the Internet as a whole, I think).

Originally, this seemed appropriate as a personal project. But the more I
think about this, the more clear it seems that a feature that allows a list
admin to stop sending emails to members who no longer want that email is a
very good feature to include in Mailman. It can help ensure that Mailman is
used in a way that causes the least amount of grief for everyone across the
Internet, right?
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] AOL redacts user addresses even with VERP and full personalization enabled

[Lindsay Haisley]

On Mon, 2012-06-18 at 13:04 -0400, Tanstaafl wrote:
> On 2012-06-18 12:22 PM, Lindsay Haisley  wrote:
> > Doing this as a custom hack helps.  If this were implemented as a
> > Mailman standard option then word might indeed get back to them about
> > it.  Using Resent-Message-ID as a header name is a clever idea.
> 
> I'd also argue that since this is not AOL specific but is a generic way 
> for a mail system admin to control his own server, and AOL cannot 
> dictate what you add to your own headers on your own messages, why not 
> make it part of mailman official, with appropriate warnings about some 
> brain-dead (probably unenforcable and possibly even illegal) limitations 
> by certain clueless providers?

I agree.  Stephen Turnbull points out that using reversible encryption
with a secret key would be more secure from the point of view of
restricting 3rd party knowledge of the unencrypted/unhashed data.  A
secret key could be kept per-list or per-site.  The ability to securely
track recipient information (or any information) across a list
distribution, or across a non-delivery bounce might be very useful.

It might be very convenient to have what one might call EVERP, where the
recipient address is encrypted into the envelope sender address, as an
alternative choice to Mailman's VERP implementation.

-- 
Lindsay Haisley   |  "Humor will get you through times of no humor
FMP Computer Services |  better than no humor will get you through
512-259-1190  | times of humor."
http://www.fmp.com|- Butch Hancock

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] AOL redacts user addresses even with VERP and full personalization enabled

[Lindsay Haisley]

On Tue, 2012-06-19 at 02:11 +0900, Stephen J. Turnbull wrote:
> Lindsay Haisley writes:
>  > Why would, say, hashlib.md5(recip).hexdigest() be any more or less
>  > detectable than a reversible encryption?
> 
> Because once the idea becomes public, anybody can check the nonesense
> strings in your headers to see if any of them hash to the user's id.
> That's a lot more difficult if you use encryption based on a secret
> key.

Very true, and a good point.  A little research turned up
http://www.codekoala.com/blog/2009/aes-encryption-python-using-pycrypto/
which is a good discussion of using AES encryption in Python.  The
Crypto module seems to be standard issue with Python - no special
libraries required.

>  > IMHO, AOL's days on this planet are numbered.  They'll go the way of
>  > Compuserve :)
> 
> Yeah, I hope so.  Unfortunately, where I live, NiftyServe still exists
> and its customers still put raw Shift JIS in their headers
> occasionally.  I'm not going to bet on AOL's timely demise.

It took a major meteor hit to wipe out the dinosaurs!

>  > I've seen Email Feedback Reports come in on posts that went out six
>  > months prior.  Parsing Message IDs out of this many MBs of back mail
>  > logs, most of them compressed, would be hugely expensive of processing
>  > time.
> 
> Seriously?  How many feedback reports do you get per second?  Yes, it
> would be a little costly, but presumably they give something like a
> date, you can narrow it down to a few MB I would guess.

Wlll ...  The average number of feedback reports / second received
on my servers is pretty managable, actually ;)  I prefer the idea of
using Resent-Message-ID and and AES encryption on the recipient address
rather than mucking with log files.  It would be nice to put this into
the Mailman structure in such a way that I could retrieve, or access the
secret key, or at least perform encryption and decryption from a
withlist script.

-- 
Lindsay Haisley   | "The difference between a duck is because
FMP Computer Services |one leg is both the same"
512-259-1190  | - Anonymous
http://www.fmp.com|

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] AOL redacts user addresses even with VERP and full personalization enabled

[Stephen J. Turnbull]

Lindsay Haisley writes:

 > Good suggestion.  I assume that Mailman never inserts
 > "Resent-Message-ID" into posts, is that correct?

Currently it doesn't, it seems, but there have been proposals to make
it do so (related to DKIM IIRC).  However, if and when it does, it
wouldn't hurt to add your obfuscated user id to it.

 > I'd rather not mess with "Message-ID" which provides a traceable
 > path to the original sender.

Right.  My comment about "content" was for the case where the list
owner is the only (or main) original sender.

 > Why would, say, hashlib.md5(recip).hexdigest() be any more or less
 > detectable than a reversible encryption?

Because once the idea becomes public, anybody can check the nonesense
strings in your headers to see if any of them hash to the user's id.
That's a lot more difficult if you use encryption based on a secret
key.

 > IMHO, AOL's days on this planet are numbered.  They'll go the way of
 > Compuserve :)

Yeah, I hope so.  Unfortunately, where I live, NiftyServe still exists
and its customers still put raw Shift JIS in their headers
occasionally.  I'm not going to bet on AOL's timely demise.

 > I've seen Email Feedback Reports come in on posts that went out six
 > months prior.  Parsing Message IDs out of this many MBs of back mail
 > logs, most of them compressed, would be hugely expensive of processing
 > time.

Seriously?  How many feedback reports do you get per second?  Yes, it
would be a little costly, but presumably they give something like a
date, you can narrow it down to a few MB I would guess.

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] AOL redacts user addresses even with VERP and full personalization enabled

[Lindsay Haisley]

On Mon, 2012-06-18 at 10:01 -0700, Brad Knowles wrote:
> > IMHO, AOL's days on this planet are numbered.  They'll go the way of
> > Compuserve :)
> 
> You mean that they'll get bought -- by AOL?  ;-)
> 
The irony is not lost :)  The snake eats itself tail-first until it
disappears.

They'll probably get bought by Google!  Didn't TW dump them recently?

-- 
Lindsay Haisley   |  "We are all broken toasters, but we still
FMP Computer Services |manage to make toast"
512-259-1190  |
http://www.fmp.com|- Cheryl Dehut

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] AOL redacts user addresses even with VERP and full personalization enabled

[Tanstaafl]

On 2012-06-18 12:22 PM, Lindsay Haisley  wrote:

Doing this as a custom hack helps.  If this were implemented as a
Mailman standard option then word might indeed get back to them about
it.  Using Resent-Message-ID as a header name is a clever idea.


I'd also argue that since this is not AOL specific but is a generic way 
for a mail system admin to control his own server, and AOL cannot 
dictate what you add to your own headers on your own messages, why not 
make it part of mailman official, with appropriate warnings about some 
brain-dead (probably unenforcable and possibly even illegal) limitations 
by certain clueless providers?

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] AOL redacts user addresses even with VERP and full personalization enabled

[Lindsay Haisley]

On Mon, 2012-06-18 at 17:03 +0900, Stephen J. Turnbull wrote:
> Lindsay Haisley writes:
> 
>  > So what would be the implications of hacking an extra header into
>  > outgoing posts on lists for which personalization is enabled, say
>  > "X-Subdata", with said header containing a hash of the subscriber
>  > address to which the post is directed?
> 
> I would use Resent-Message-ID, unless the content of posts is such
> that you can get away with munging Message-ID itself.

Good suggestion.  I assume that Mailman never inserts
"Resent-Message-ID" into posts, is that correct?  I'd rather not mess
with "Message-ID" which provides a traceable path to the original
sender.

> I would also use a
> reversible encryption rather than a hash.  (Not so much because it's
> reversible, but rather because it's undetectable except insofar as
> it's different from standard Mailman.)

Suggestions, Stephen?  Why would, say, hashlib.md5(recip).hexdigest() be
any more or less detectable than a reversible encryption?

>  > This would, in theory, mostly satisfy AOL's privacy concern
> 
> I really don't think so.  It might satisfy *your* privacy concerns,
> but their "privacy" concern is absolute.

I don't give a rat's behinder about privacy on this issue, only that _I_
be able to identify the complaining recipient, based on having the
subscriber lists available, and that AOL and their minions _not_ be able
to do so.

> That's not to say you shouldn't do it, but if they catch on, they'll
> start redacting those headers, too, and quite possibly boot you from
> their feedback loop.

They've been letting VERPed subscriber addresses through their rather
scattershot redaction process for years.  I've been parsing them out of
the Sender header for about as long and automatically unsubscribing
these addresses from Mailman lists.  I could easily ignore them and stay
under AOL's radar, but I consider it a service to my customers to help
them keep their lists free of subscribers who don't want the traffic, no
matter how clueless they may be.

Doing this as a custom hack helps.  If this were implemented as a
Mailman standard option then word might indeed get back to them about
it.  Using Resent-Message-ID as a header name is a clever idea.

> As Brad points out, they simply don't care if their members get the
> mail that they want.  Or at least, they don't care about that anywhere
> near as much as they care that their members don't get mail that they
> don't want!

IMHO, AOL's days on this planet are numbered.  They'll go the way of
Compuserve :)

>  > Hacking the message ID out of mail logs to identify the subscriber seems
>  > somewhat chancier and more difficult, since mail logs roll over and
>  > eventually disappear from the system.
> 
> If you say so, but *that is under your control*.  I'd much rather make
> the effort to make my logs dependable, than depend on any cooperation
> from AOL.

I've seen Email Feedback Reports come in on posts that went out six
months prior.  Parsing Message IDs out of this many MBs of back mail
logs, most of them compressed, would be hugely expensive of processing
time.  I don't depend on cooperation from AOL, just stupidity, which
seems to be pretty dependable :)  On the other hand, the process of
dealing with these reports only happens a few times a month, at most.  

-- 
Lindsay Haisley   | "Real programmers use butterflies"
FMP Computer Services |
512-259-1190  |   - xkcd
http://www.fmp.com|

--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] AOL redacts user addresses even with VERP and full personalization enabled

[Stephen J. Turnbull]

Lindsay Haisley writes:

 > So what would be the implications of hacking an extra header into
 > outgoing posts on lists for which personalization is enabled, say
 > "X-Subdata", with said header containing a hash of the subscriber
 > address to which the post is directed?

I would use Resent-Message-ID, unless the content of posts is such
that you can get away with munging Message-ID itself.  That is a
standardized header that Mailman uses anyway.  I would also use a
reversible encryption rather than a hash.  (Not so much because it's
reversible, but rather because it's undetectable except insofar as
it's different from standard Mailman.)

 > This would, in theory, mostly satisfy AOL's privacy concern

I really don't think so.  It might satisfy *your* privacy concerns,
but their "privacy" concern is absolute.  (I doubt that their basic
motive is to protect their customers' privacy, especially given Brad's
statements, but I see no reason not to take them at their word that
*any* attempt to identify customers is a violation of their feedback
loop user agreement.)

That's not to say you shouldn't do it, but if they catch on, they'll
start redacting those headers, too, and quite possibly boot you from
their feedback loop.

As Brad points out, they simply don't care if their members get the
mail that they want.  Or at least, they don't care about that anywhere
near as much as they care that their members don't get mail that they
don't want!

 > Hacking the message ID out of mail logs to identify the subscriber seems
 > somewhat chancier and more difficult, since mail logs roll over and
 > eventually disappear from the system.

If you say so, but *that is under your control*.  I'd much rather make
the effort to make my logs dependable, than depend on any cooperation
from AOL.
--
Mailman-Users mailing list Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org