[PATCH v6 1/2] parsemail: Convert to a management command

[Stephen Finucane]

Management comands allow applications to register their own actions with
'manage.py'. This provides some advantages, like automatically
configuring Django (removing the need for 'django.setup' calls) and
removing the need to set the PYTHON_PATH. The 'parsemail' script is a
natural fit for this type of application. Migrate 'parsemail' to a
management command.

This includes some extensive work on logging configuration, as logging
is moved from code into settings. In addition, it removes a lot of the
customizable logging previously introduced in the parsemail command, in
favour of modifications to the settings files.

Signed-off-by: Stephen Finucane 
Partial-bug: #17
Closes-bug: #15
---
v6:
- Rework tests
- Remove unneeded 'stdin' parameter
v5:
- Resolve some Argparse vs. OptParse issues
v4:
- Add unit tests
- Add support for Django 1.10
v3:
- Allow passing of additional arguments
v2:
- Remove unnecessary import
- Don't raise exception
- Add note for parsemail script return code
---
 patchwork/bin/parsemail.py | 114 -
 patchwork/bin/parsemail.sh |   8 +-
 patchwork/management/commands/parsemail.py |  81 
 patchwork/parser.py|  12 +--
 patchwork/settings/base.py |  60 +--
 patchwork/tests/__init__.py|  23 ++
 patchwork/tests/test_management.py |  80 
 patchwork/tests/test_parser.py |   4 +-
 patchwork/tests/utils.py   |   2 +-
 9 files changed, 254 insertions(+), 130 deletions(-)
 delete mode 100755 patchwork/bin/parsemail.py
 create mode 100644 patchwork/management/commands/parsemail.py
 create mode 100644 patchwork/tests/test_management.py

diff --git a/patchwork/bin/parsemail.py b/patchwork/bin/parsemail.py
deleted file mode 100755
index b35b1f6..000
--- a/patchwork/bin/parsemail.py
+++ /dev/null
@@ -1,114 +0,0 @@
-#!/usr/bin/env python
-#
-# Patchwork - automated patch tracking system
-# Copyright (C) 2008 Jeremy Kerr 
-#
-# This file is part of the Patchwork package.
-#
-# Patchwork is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
-#
-# Patchwork is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Patchwork; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
-
-from __future__ import absolute_import
-
-import argparse
-from email import message_from_file
-import logging
-import sys
-
-import django
-from django.conf import settings
-from django.utils.log import AdminEmailHandler
-
-from patchwork.parser import parse_mail
-
-LOGGER = logging.getLogger(__name__)
-
-VERBOSITY_LEVELS = {
-'debug': logging.DEBUG,
-'info': logging.INFO,
-'warning': logging.WARNING,
-'error': logging.ERROR,
-'critical': logging.CRITICAL
-}
-
-
-extra_error_message = '''
-== Mail
-
-%(mail)s
-
-
-== Traceback
-
-'''
-
-
-def setup_error_handler():
-"""Configure error handler.
-
-Ensure emails are send to settings.ADMINS when errors are
-encountered.
-"""
-if settings.DEBUG:
-return
-
-mail_handler = AdminEmailHandler()
-mail_handler.setLevel(logging.ERROR)
-mail_handler.setFormatter(logging.Formatter(extra_error_message))
-
-logger = logging.getLogger('patchwork')
-logger.addHandler(mail_handler)
-
-return logger
-
-
-def main(args):
-django.setup()
-logger = setup_error_handler()
-parser = argparse.ArgumentParser()
-
-def list_logging_levels():
-"""Give a summary of all available logging levels."""
-return sorted(list(VERBOSITY_LEVELS.keys()),
-  key=lambda x: VERBOSITY_LEVELS[x])
-
-parser.add_argument('infile', nargs='?', type=argparse.FileType('r'),
-default=sys.stdin, help='input mbox file (a filename '
-'or stdin)')
-
-group = parser.add_argument_group('Mail parsing configuration')
-group.add_argument('--list-id', help='mailing list ID. If not supplied '
-   'this will be extracted from the mail headers.')
-group.add_argument('--verbosity', choices=list_logging_levels(),
-   help='debug level', default='info')
-
-args = vars(parser.parse_args())
-
-logging.basicConfig(level=VERBOSITY_LEVELS[args['verbosity']])
-
-mail = message_from_file(args['infile'])
-try:
-result = parse_mail(mail, args['list_id'])
-if 

[PATCH v6 0/2] Convert 'parse' scripts to management commands

[Stephen Finucane]

The parsemail and parsearchive scripts are currently the main way of
loading content into Patchwork. However, running them requires modifying
the PYTHONPATH environment variable, along with calls to configure
Django before doing anything.

Django describes management commands as a good option for standalone
scripts, which make it a good fit for these two scripts. Migrate these
to management commands, moving the main bulk of the parsing code into a
dedicated parsing module. As part of this, we fix some bugs and
refactor some of the cruftier code.

This sets us up nicely for adding an '/upload' endpoint at some point in
the future, which could allow Patchwork to be deployed on PaaS systems.
These systems don't have access to the underlying environment, meaning
something like a Unix mailbox might not be available. An API may well be
the only way to submit content in this area.

Change since v5:
- Rework unit tests to cover additional cases
- Remove some unnecessary Django<1.8-handling code

Stephen Finucane (2):
  parsemail: Convert to a management command
  parsearchive: Convert to a management command

 docs/development.md   |   8 +-
 patchwork/bin/parsearchive.py | 106 
 patchwork/bin/parsemail.py| 114 --
 patchwork/bin/parsemail.sh|   8 +-
 patchwork/management/commands/parsearchive.py | 106 
 patchwork/management/commands/parsemail.py|  81 ++
 patchwork/parser.py   |  12 +--
 patchwork/settings/base.py|  60 --
 patchwork/tests/__init__.py   |  23 ++
 patchwork/tests/test_management.py| 112 +
 patchwork/tests/test_parser.py|   4 +-
 patchwork/tests/utils.py  |   2 +-
 12 files changed, 396 insertions(+), 240 deletions(-)
 delete mode 100755 patchwork/bin/parsearchive.py
 delete mode 100755 patchwork/bin/parsemail.py
 create mode 100644 patchwork/management/commands/parsearchive.py
 create mode 100644 patchwork/management/commands/parsemail.py
 create mode 100644 patchwork/tests/test_management.py

-- 
2.7.4

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH v6 2/2] parsearchive: Convert to a management command

[Stephen Finucane]

As with parsemail, parsearchive makes more sense as a management
command. Make it so.

As with the conversion of the 'parsemail' tool, this removes
customisable logging as it's not necessary.

Signed-off-by: Stephen Finucane 
Closes-bug: #17
---
v6:
- Add additional unit tests
v4:
- Add unit tests
- Add support for Django 1.10
---
 docs/development.md   |   8 +-
 patchwork/bin/parsearchive.py | 106 --
 patchwork/management/commands/parsearchive.py | 106 ++
 patchwork/tests/test_management.py|  32 
 4 files changed, 142 insertions(+), 110 deletions(-)
 delete mode 100755 patchwork/bin/parsearchive.py
 create mode 100644 patchwork/management/commands/parsearchive.py

diff --git a/docs/development.md b/docs/development.md
index e51f7b1..36d2fdf 100644
--- a/docs/development.md
+++ b/docs/development.md
@@ -268,8 +268,8 @@ using the aptly-named `createsuperuser` command:
 
 Once this is done, it's beneficial to load some real emails into the system.
 This can be done manually, however it's generally much easier to download
-an archive from a Mailman instance and load these using the `parsearchive.py`
-tool. You can do this like so:
+an archive from a Mailman instance and load these using the `parsearchive`
+command. You can do this like so:
 
 (.venv)$ mm_user=myusername
 (.venv)$ mm_pass=mypassword
@@ -288,8 +288,8 @@ find more informations about this [here][ref-mman-bulk].
 Load these archives into Patchwork. Depending on the size of the downloaded
 archives this may take some time:
 
-(.venv)$ PYTHONPATH=. ./patchwork/bin/parsearchive.py \
-  --list-id=patchwork.ozlabs.org patchwork.mbox
+(.venv)$ ./manage.py parsearchive --list-id=patchwork.ozlabs.org \
+  patchwork.mbox
 
 Finally, run the server and browse to the IP address of your board using your
 browser of choice:
diff --git a/patchwork/bin/parsearchive.py b/patchwork/bin/parsearchive.py
deleted file mode 100755
index 8986b22..000
--- a/patchwork/bin/parsearchive.py
+++ /dev/null
@@ -1,106 +0,0 @@
-#!/usr/bin/env python
-#
-# Patchwork - automated patch tracking system
-# Copyright (C) 2015 Intel Corporation
-#
-# This file is part of the Patchwork package.
-#
-# Patchwork is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
-#
-# Patchwork is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Patchwork; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
-
-"""Utility to parse an mbox archive file."""
-
-from __future__ import absolute_import
-
-import argparse
-import logging
-import mailbox
-
-import django
-
-from patchwork.parser import parse_mail
-from patchwork import models
-
-LOGGER = logging.getLogger(__name__)
-
-VERBOSITY_LEVELS = {
-'debug': logging.DEBUG,
-'info': logging.INFO,
-'warning': logging.WARNING,
-'error': logging.ERROR,
-'critical': logging.CRITICAL
-}
-
-
-def parse_mbox(path, list_id):
-results = {
-models.Patch: 0,
-models.CoverLetter: 0,
-models.Comment: 0,
-}
-duplicates = 0
-dropped = 0
-
-mbox = mailbox.mbox(path)
-for msg in mbox:
-try:
-obj = parse_mail(msg, list_id)
-if obj:
-results[type(obj)] += 1
-else:
-dropped += 1
-except django.db.utils.IntegrityError:
-duplicates += 1
-print('Processed %(total)d messages -->\n'
-  '  %(covers)4d cover letters\n'
-  '  %(patches)4d patches\n'
-  '  %(comments)4d comments\n'
-  '  %(duplicates)4d duplicates\n'
-  '  %(dropped)4d dropped\n'
-  'Total: %(new)s new entries' % {
-  'total': len(mbox),
-  'covers': results[models.CoverLetter],
-  'patches': results[models.Patch],
-  'comments': results[models.Comment],
-  'duplicates': duplicates,
-  'dropped': dropped,
-  'new': len(mbox) - duplicates - dropped,
-  })
-
-
-def main():
-django.setup()
-parser = argparse.ArgumentParser(description=__doc__)
-
-def list_logging_levels():
-"""Give a summary of all available logging levels."""
-return sorted(VERBOSITY_LEVELS.keys(),
-  key=lambda x: VERBOSITY_LEVELS[x])
-
-parser.add_argument('inpath', help='input mbox filename')
-
-group = parser.add_argument_group('Mail