Re: [PATCH v6 1/2] parsemail: Convert to a management command

[Thomas Monjalon]

Hi,

2016-09-20 00:08, Stephen Finucane:
> On 20 Sep 01:22, Daniel Axtens wrote:
> > So, umm, I went ahead and had a crack at this.
> > 
> > It turns out this is hideously difficult to get right. But this plus my
> > other patch to fix Thomas' problem should have things working on Py2 and
> > Py3 with this series.

Thanks for taking care.

[...]
> I don't have the offending patch on hand, but isn't the issue with the
> headers. If so, would something like the below do (I haven't tested
> it - there could be typos).
[...]

Please would it be possible to fix this bug in the stable branch also?
Thanks a lot
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH v6 1/2] parsemail: Convert to a management command

[Stephen Finucane]

On 20 Sep 01:22, Daniel Axtens wrote:
> So, umm, I went ahead and had a crack at this.
> 
> It turns out this is hideously difficult to get right. But this plus my
> other patch to fix Thomas' problem should have things working on Py2 and
> Py3 with this series.
> 
> It's a bit of a work in progress: I need to close the file at the end
> of the function, the logging needs to be added again, etc.
> 
> Tests to come.
> 
> Stephen: I can do this up into a proper patch if you like or you can
> fold it into your series.
> 
> Regards,
> Daniel

I don't have the offending patch on hand, but isn't the issue with the
headers. If so, would something like the below do (I haven't tested
it - there could be typos).

I'll review this if not.

Stephen

diff --git a/patchwork/parser.py b/patchwork/parser.py
index 1805df8..7917e97 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -21,7 +21,8 @@
 
 import codecs
 import datetime
-from email.header import Header, decode_header
+from email.header import decode_header
+from email.header import make_header
 from email.utils import parsedate_tz, mktime_tz
 from fnmatch import fnmatch
 from functools import reduce
@@ -155,10 +156,10 @@ def find_date(mail):
 
 
 def find_headers(mail):
-return reduce(operator.__concat__,
-  ['%s: %s\n' % (k, Header(v, header_name=k,
-   continuation_ws='\t').encode())
-   for (k, v) in list(mail.items())])
+headers = {key: decode_header(value) for key, value in list(mail.items())}
+return '\n'.join(['%s: %s' % (key, make_header(value[0], header_name=key,
+   continuation_wd='\t'))
+  for key, value in headers])
 
 
 def find_references(mail):
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH v6 1/2] parsemail: Convert to a management command

[Daniel Axtens]

So, umm, I went ahead and had a crack at this.

It turns out this is hideously difficult to get right. But this plus my
other patch to fix Thomas' problem should have things working on Py2 and
Py3 with this series.

It's a bit of a work in progress: I need to close the file at the end
of the function, the logging needs to be added again, etc.

Tests to come.

Stephen: I can do this up into a proper patch if you like or you can
fold it into your series.

Regards,
Daniel

diff --git a/patchwork/management/commands/parsemail.py 
b/patchwork/management/commands/parsemail.py
index a60e2ad11f06..2b957473167f 100644
--- a/patchwork/management/commands/parsemail.py
+++ b/patchwork/management/commands/parsemail.py
@@ -18,10 +18,11 @@
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 
 import argparse
-from email import message_from_file
+import email
 import logging
 from optparse import make_option
 import sys
+import codecs
 
 import django
 from django.core.management import base
@@ -48,9 +49,9 @@ class Command(base.BaseCommand):
 parser.add_argument(
 'infile',
 nargs='?',
-type=argparse.FileType('r'),
-default=sys.stdin,
-help='input mbox file (a filename or stdin)')
+type=str,
+default=None,
+help='input mbox file (stdin if not provided)')
 parser.add_argument(
 '--list-id',
 help='mailing list ID. If not supplied, this will be '
@@ -59,17 +60,19 @@ class Command(base.BaseCommand):
 def handle(self, *args, **options):
 infile = args[0] if args else options['infile']
 
-# Attempt to parse the path if provided, and fallback to stdin if not
-if infile and isinstance(infile, six.string_types):  # Django < 1.8
-logger.info('Parsing mail loaded by filename')
-with open(infile, 'r+') as file_:
-mail = message_from_file(file_)
+if six.PY3:
+if infile:
+file_ = open(infile, 'r', encoding='utf-8', errors='replace')
+else:
+file_ = codecs.getreader('utf-8')(sys.stdin.buffer, 
errors='replace')
 else:
-if infile == sys.stdin:
-logger.info('Parsing mail loaded from stdin')
-else:  # Djano >= 1.8
-logger.info('Parsing mail loaded by filename')
-mail = message_from_file(infile)
+if infile:
+file_ = open(infile, 'r')
+else:
+file_ = codecs.getreader('utf-8')(sys.stdin, errors='replace')
+
+mail = email.message_from_file(file_)
+
 try:
 result = parse_mail(mail, options['list_id'])
 if result:


signature.asc
Description: PGP signature
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH v6 1/2] parsemail: Convert to a management command

[Stephen Finucane]

Management comands allow applications to register their own actions with
'manage.py'. This provides some advantages, like automatically
configuring Django (removing the need for 'django.setup' calls) and
removing the need to set the PYTHON_PATH. The 'parsemail' script is a
natural fit for this type of application. Migrate 'parsemail' to a
management command.

This includes some extensive work on logging configuration, as logging
is moved from code into settings. In addition, it removes a lot of the
customizable logging previously introduced in the parsemail command, in
favour of modifications to the settings files.

Signed-off-by: Stephen Finucane 
Partial-bug: #17
Closes-bug: #15
---
v6:
- Rework tests
- Remove unneeded 'stdin' parameter
v5:
- Resolve some Argparse vs. OptParse issues
v4:
- Add unit tests
- Add support for Django 1.10
v3:
- Allow passing of additional arguments
v2:
- Remove unnecessary import
- Don't raise exception
- Add note for parsemail script return code
---
 patchwork/bin/parsemail.py | 114 -
 patchwork/bin/parsemail.sh |   8 +-
 patchwork/management/commands/parsemail.py |  81 
 patchwork/parser.py|  12 +--
 patchwork/settings/base.py |  60 +--
 patchwork/tests/__init__.py|  23 ++
 patchwork/tests/test_management.py |  80 
 patchwork/tests/test_parser.py |   4 +-
 patchwork/tests/utils.py   |   2 +-
 9 files changed, 254 insertions(+), 130 deletions(-)
 delete mode 100755 patchwork/bin/parsemail.py
 create mode 100644 patchwork/management/commands/parsemail.py
 create mode 100644 patchwork/tests/test_management.py

diff --git a/patchwork/bin/parsemail.py b/patchwork/bin/parsemail.py
deleted file mode 100755
index b35b1f6..000
--- a/patchwork/bin/parsemail.py
+++ /dev/null
@@ -1,114 +0,0 @@
-#!/usr/bin/env python
-#
-# Patchwork - automated patch tracking system
-# Copyright (C) 2008 Jeremy Kerr 
-#
-# This file is part of the Patchwork package.
-#
-# Patchwork is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
-#
-# Patchwork is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Patchwork; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
-
-from __future__ import absolute_import
-
-import argparse
-from email import message_from_file
-import logging
-import sys
-
-import django
-from django.conf import settings
-from django.utils.log import AdminEmailHandler
-
-from patchwork.parser import parse_mail
-
-LOGGER = logging.getLogger(__name__)
-
-VERBOSITY_LEVELS = {
-'debug': logging.DEBUG,
-'info': logging.INFO,
-'warning': logging.WARNING,
-'error': logging.ERROR,
-'critical': logging.CRITICAL
-}
-
-
-extra_error_message = '''
-== Mail
-
-%(mail)s
-
-
-== Traceback
-
-'''
-
-
-def setup_error_handler():
-"""Configure error handler.
-
-Ensure emails are send to settings.ADMINS when errors are
-encountered.
-"""
-if settings.DEBUG:
-return
-
-mail_handler = AdminEmailHandler()
-mail_handler.setLevel(logging.ERROR)
-mail_handler.setFormatter(logging.Formatter(extra_error_message))
-
-logger = logging.getLogger('patchwork')
-logger.addHandler(mail_handler)
-
-return logger
-
-
-def main(args):
-django.setup()
-logger = setup_error_handler()
-parser = argparse.ArgumentParser()
-
-def list_logging_levels():
-"""Give a summary of all available logging levels."""
-return sorted(list(VERBOSITY_LEVELS.keys()),
-  key=lambda x: VERBOSITY_LEVELS[x])
-
-parser.add_argument('infile', nargs='?', type=argparse.FileType('r'),
-default=sys.stdin, help='input mbox file (a filename '
-'or stdin)')
-
-group = parser.add_argument_group('Mail parsing configuration')
-group.add_argument('--list-id', help='mailing list ID. If not supplied '
-   'this will be extracted from the mail headers.')
-group.add_argument('--verbosity', choices=list_logging_levels(),
-   help='debug level', default='info')
-
-args = vars(parser.parse_args())
-
-logging.basicConfig(level=VERBOSITY_LEVELS[args['verbosity']])
-
-mail = message_from_file(args['infile'])
-try:
-result = parse_mail(mail, args['list_id'])
-if