[This mail is intentionally hand wrapped..]
On Freitag, 10. Juni 2016 17:54:34 Hans-Peter Jansen wrote:
> On Donnerstag, 9. Juni 2016 18:37:54 Hans-Peter Jansen wrote:
> >
> > Let's see, how this goes.
>
> Hmm, compat32 and Python3 start to get in my way in no funny ways.
Okay, got it working now. It turned out to be a problem in the logging module.
Unfortunately, it's not reproducible outside postfix mail filter setup.
http://www.postfix.org/FILTER_README.html
I will try to explain this in short words, but you will not believe me, as I
have a hard time to believe this myself..
When a python3 process is used in a simple mail filter setup, as described in
the FILTER_README.html document, it is executed in a stripped down environment.
As relevant parts, just LANG=C and PATH=/bin;/usr/bin is set.
The filter reads the mail from stdin, and calls sendmail for passing it on,
again, with the mail on stdin..
Now, it takes a mail with some "higher" encoding (I'm using a utf-8 encoded
subject containing german umlauts), and an attempt to log the subject line
(which has to be a log file for obvious reasons). Now take a save seat, this
attempt of unicode logging results in a manipulation of the execution frame,
the execution precedes a few instructions "below". Yes, I'm not kidding, no
escaped surrogates output, no error message, just no logging of the offending
line, and this "esoteric" behavior. Be assured, that if I would still have
enough hair on my head, I would have teared it off completely by now.
This is reproducible for Python 3.4.4 on openSUSE 13.2/x86_64 here.
For the brave, who want to reproduce/investigate this issue, I'm attaching
everything necessary. All others should stop reading now. Thank you.
Still with me, here we go: [$: root prompt]
A working postfix setup is implied. Stop all usual processing (fetchmail, ...)
$ useradd --gid mail mfilter
$ cat >> /etc/postfix/master.cf << EOF
mfilter unix - n n - 1 pipe
flags=Rq user=mfilter argv=/path/to/mail_filter_test.py -f ${sender} --
${recipient}
EOF
$ systemctl restart postfix
$ sendmail -f your@email.address your@email.address < umlaut-subject-2.mail
$ less +F /tmp/mail_filter_test.log
Defective output:
2016-06-12 15:36:28,540 [mail_filter_test] DEBUG: parse message
2016-06-12 15:36:28,543 [mail_filter_test] DEBUG: call ['/usr/sbin/sendmail',
'-G', '-i', '-f', 'your@email.address', '--', 'your@email.address']
Output with SILLY_BEHAVIOR = 0:
2016-06-12 15:37:50,887 [mail_filter_test] DEBUG: parse message
2016-06-12 15:37:50,889 [mail_filter_test] DEBUG: subject: Wie deaktiviere oder
lösche ich meine SprachBox IP der Telekom?
2016-06-12 15:37:50,890 [mail_filter_test] DEBUG: call ['/usr/sbin/sendmail',
'-G', '-i', '-f', 'your@email.address', '--', 'your@email.address']
Note, that the subject line is missing. In my real filter, it left the
current execution frame, and execution continued one or two level up the
stack.
This reminds me at my assembler times (680x0 power, long ago!), where I used
such tricks like modifying the program counter in "very special arrangements".
I don't think, this is an adequate outcome of the attached code, do you?
I'm directing this here, while I know, this is quite off-topic in the result.
OTOH, Stephen and David discussed the LANG=C issues with Victor, and this
is one example, where this is very relevant to Python3 fitness to act as
such a filter. Apart from fixing the logging, I'm encoding all paths and file
names with a configurable encoding before calling the OS. Butt ugly, but
feasible.
I hope, that at least one of you is able to reproduce this, before we decide,
how to precede. I hope, I don't offend anybody here with that approach.
Please speak up, if I should go away and search for another tree to bark at.
Thanks,
Pete#!/usr/bin/env python3
import sys
import getopt
import logging
import subprocess
import email
import email.policy
import email.header
import email.generator
logfile = '/tmp/mail_filter_test.log'
logformat = '%(asctime)s [%(name)s] %(levelname)5s: %(message)s'
encoding = 'utf-8'
# setup logging
log = logging.getLogger('mail_filter_test')
EX_TEMPFAIL = 75 # queue and retry
SILLY_BEHAVIOR = 1
if SILLY_BEHAVIOR:
logging.basicConfig(
level = logging.DEBUG,
format = logformat,
filename = logfile,
)
else:
log.setLevel(logging.DEBUG)
filelog = logging.FileHandler(logfile, encoding = encoding)
filelog.setLevel(logging.DEBUG)
filelog.setFormatter(logging.Formatter(logformat))
log.addHandler(filelog)
def decode_header(value):
if value is not None:
value = str(email.header.make_header(email.header.decode_header(value)))
return value
def mail_filter(sender, recipients):
log.debug('parse message')
msg = email.message_from_binary_file(sys.stdin.buffer, policy = email.policy.compat32)
if not sender:
sender = msg.get('from')
if not recipients:
recipients = msg.get_all('to')
subject = decode_header(msg.get('subject'))
log.debug('subject: %s', subject)
sendmail = ['/usr/sbin/sendmail', '-G', '-i', '-f', sender, '--'] + recipients
log.debug('call %s', sendmail)
p = subprocess.Popen(sendmail, stdin = subprocess.PIPE)
email.generator.BytesGenerator(p.stdin).flatten(msg)
p.stdin.close()
return p.wait()
if __name__ == '__main__':
try:
optlist, recipients = getopt.getopt(sys.argv[1:], 'f:')
except getopt.error as msg:
print(msg, file = sys.stderr, flush = True)
sys.exit(EX_TEMPFAIL)
sender = None
for opt, par in optlist:
if opt in ('-f', '--from'):
sender = par
sys.exit(mail_filter(sender, recipients))
https://www.telekom.de/hilfe/festnetz-internet-tv/telefonieren-einstellungen/sprachbox/sprachbox-ip/sprachbox-ip-deaktivieren-oder-loeschen?samChecked=true
_______________________________________________
Email-SIG mailing list
Email-SIG@python.org
Your options:
https://mail.python.org/mailman/options/email-sig/archive%40mail-archive.com
