William Grant has proposed merging lp:~wgrant/launchpad/explode-spec-exploder into lp:launchpad.
Requested reviews: Launchpad code reviewers (launchpad-reviewers) For more details, see: https://code.launchpad.net/~wgrant/launchpad/explode-spec-exploder/+merge/98560 Remove the obsolete spec wiki notification exploder. As described in bug #54005, the wikis stopped sending mailing to it nearly 2 years ago. -- https://code.launchpad.net/~wgrant/launchpad/explode-spec-exploder/+merge/98560 Your team Launchpad code reviewers is requested to review the proposed merge of lp:~wgrant/launchpad/explode-spec-exploder into lp:launchpad.
=== removed file 'lib/lp/blueprints/doc/spec-mail-exploder.txt' --- lib/lp/blueprints/doc/spec-mail-exploder.txt 2011-12-29 05:29:36 +0000 +++ lib/lp/blueprints/doc/spec-mail-exploder.txt 1970-01-01 00:00:00 +0000 @@ -1,270 +0,0 @@ -The Spec Mail Exploder -====================== - -There's an address, [email protected], which looks at -all incoming email to see if it looks like a change notifications from a -Moin wiki. If it receive such notification, it will look up the relevant -Launchpad specification, and re-send the mail to the spec's related -people. - -First, let's take a look at how a Moin change notification looks like: - - >>> from lp.services.mail.tests.helpers import read_test_message - >>> moin_change = read_test_message('moin-change.txt') - -It contains some headers, which are of no use to us, and the body is -base64 encoded. - - >>> print str(moin_change) #doctest: -NORMALIZE_WHITESPACE - From [email protected] Mon Mar 20 10:31:59 2006 - Return-path: <[email protected]> - Content-Type: text/plain; charset="utf-8" - MIME-Version: 1.0 - Content-Transfer-Encoding: base64 - From: Launchpad Wiki <[email protected]> - To: Launchpad Wiki <[email protected]> - Date: Mon, 20 Mar 2006 10:26:28 -0000 - Message-ID: <[email protected]> - Subject: [Launchpad Wiki] Update of "MediaIntegrityCheck" by ... - Status: RO - Content-Length: 1220 - <BLANKLINE> - RGVhciBXaWtpIHVzZXIsDQo... - -Let's take a look at the decode body, since we'll have to use it in -order to extract any useful information about the wiki page. - - >>> print moin_change.get_payload(decode=True) - Dear Wiki user, - <BLANKLINE> - You have subscribed to a wiki page or wiki category on "Ubuntu Wiki" - for change notification. - <BLANKLINE> - The following page has been changed by JamesHenstridge: - https://wiki.ubuntu.com/MediaIntegrityCheck - <BLANKLINE> - The comment on the change is: - A comment about the change - <BLANKLINE> - -----------------------------------------------------------------... - ... - - -The Mail Handler ----------------- - -The mail handler that handles mail on [email protected] -is BlueprintHandler. - - >>> from lp.services.config import config - >>> from lp.services.mail.handlers import mail_handlers - >>> handler = mail_handlers.get(config.launchpad.specs_domain) - >>> handler is not None - True - -It has a helper method to help associate a URL with a specification. We -use this method instead of simply ISpecificationSet.getByURL() since the -Ubuntu wikis are special, they have more than one domain name mapping to -the same wiki. So if we start with the correct URL, of course we get the -specification with that URL. - - >>> spec = handler._getSpecByURL( - ... 'https://wiki.ubuntu.com/MediaIntegrityCheck') - >>> spec.specurl - u'https://wiki.ubuntu.com/MediaIntegrityCheck' - -But if someone would edit the same wiki page on wiki.edubuntu.org, we -would want the same spec's subscribers to be notified about it. - - >>> spec = handler._getSpecByURL( - ... 'https://wiki.edubuntu.org/MediaIntegrityCheck') - >>> spec.specurl - u'https://wiki.ubuntu.com/MediaIntegrityCheck' - -And the same with wiki.kubuntu.org URLs. - - >>> spec = handler._getSpecByURL( - ... 'https://wiki.kubuntu.org/MediaIntegrityCheck') - >>> spec.specurl - u'https://wiki.ubuntu.com/MediaIntegrityCheck' - -If no matching specification is found, None is returned. - - >>> spec = handler._getSpecByURL( - ... 'https://wiki.kubuntu.org/NonExistant') - >>> spec is None - True - -We use a logger to see what the handler does: - - >>> from lp.services.scripts import log - >>> OLD_LOG_LEVEL = log._log.getEffectiveLevel() - >>> log._log.setLevel(10) - -Now, if we pass the email to it, we can see that it finds the correct -spec, and mails it to the related people. - - >>> handler.process( - ... moin_change, '[email protected]', log=log) - True - - >>> log._log.setLevel(OLD_LOG_LEVEL) - - >>> from lp.blueprints.interfaces.specification import ISpecificationSet - >>> spec = getUtility(ISpecificationSet).getByURL( - ... 'https://wiki.ubuntu.com/MediaIntegrityCheck') - >>> spec.notificationRecipientAddresses() - ['[email protected]'] - -Let's look at the email that got sent: - - >>> import transaction - >>> from lp.services.mail import stub - >>> transaction.commit() - >>> len(stub.test_emails) - 1 - -The email got sent to all related people: - - >>> from_addr, to_addrs, raw_message = stub.test_emails.pop() - >>> to_addrs == spec.notificationRecipientAddresses() - True - -We practically bounced the message, we didn't change anything about it, -except for setting the Sender header to our bounce address: - - >>> import email - >>> sent_msg = email.message_from_string(raw_message) - >>> sent_body = sent_msg.get_payload(decode=True) - >>> sent_body == moin_change.get_payload(decode=True) - True - - >>> sent_msg['To'] == moin_change['To'] - True - - >>> sent_msg['From'] == moin_change['From'] - True - - >>> sent_msg['Subject'] == moin_change['Subject'] - True - - >>> sent_msg['Sender'] == config.canonical.bounce_address - True - -If we get a change notification about a spec which doesn't exist in -Launchpad, nothing happens. - - >>> log._log.setLevel(10) - >>> moin_change = read_test_message('moin-change-nonexistant.txt') - >>> handler.process( - ... moin_change, '[email protected]', log=log) - True - - >>> print getUtility(ISpecificationSet).getByURL( - ... 'https://wiki.ubuntu.com/NonExistant') - None - -In order to prevent loops, we set the X-Loop header: - - >>> sent_msg['X-Loop'] - '[email protected]' - -So that if we someone re-sends the notification we sent to us, we'll -simply emit a warning and drop the email. - - >>> class FakeFileAlias: - ... http_url = 'http://librarian/foo.txt' - >>> del sent_msg['Sender'] - >>> sent_msg['Sender'] = '[email protected]' - >>> handler.process( - ... sent_msg, '[email protected]', - ... filealias=FakeFileAlias(), log=log) - WARNING...:Got back a notification we sent: http://librarian/foo.txt - True - - >>> log._log.setLevel(OLD_LOG_LEVEL) - -No emails were sent: - - >>> transaction.commit() - >>> len(stub.test_emails) - 0 - -To prevent bad things happening if someone subscribes this address to -specifications for example, we ignore all email that are sent from -Launchpad: - - >>> del sent_msg['X-Loop'] - >>> del sent_msg['Sender'] - >>> sent_msg['Sender'] = config.canonical.bounce_address - >>> log._log.setLevel(10) - >>> handler.process( - ... sent_msg, '[email protected]', - ... filealias=FakeFileAlias(), log=log) - WARNING...:We received an email from Launchpad: http://librarian/foo.txt - True - - >>> log._log.setLevel(OLD_LOG_LEVEL) - -Again, no emails were sent: - - >>> transaction.commit() - >>> len(stub.test_emails) - 0 - -Let's pass a notification from wiki.kubuntu.org regarding -MediaIntegrityCheck to see that the correct specification will be found, -and that the notification will be forwarded to the specification's -subscribers: - - >>> kubuntu_change = read_test_message('moin-change-kubuntu.txt') - >>> log._log.setLevel(10) - >>> handler.process( - ... kubuntu_change, '[email protected]', log=log) - True - - >>> log._log.setLevel(OLD_LOG_LEVEL) - - >>> transaction.commit() - >>> len(stub.test_emails) - 1 - - >>> from_addr, to_addrs, raw_message = stub.test_emails.pop() - >>> to_addrs == spec.notificationRecipientAddresses() - True - -Lastly, let's simulate sending a moin notification, and use handleMail() -instead to ensure that handler above handles the email. This will make -sure that the handler is setup properly to handle unknown users, since [email protected] doesn't belong to any Person in Launchpad: - - >>> from lp.registry.interfaces.person import IPersonSet - >>> getUtility(IPersonSet).getByEmail('[email protected]') is None - True - - >>> moin_change = read_test_message('moin-change.txt') - >>> moin_change['X-Launchpad-Original-To'] = "notifications@%s" % ( - ... config.launchpad.specs_domain) - >>> moin_change['Sender'] = '[email protected]' - - >>> from lp.services.mail.incoming import handleMail - >>> from lp.services.mail.sendmail import sendmail - >>> sendmail(moin_change, bulk=False) - '...' - - >>> transaction.commit() - >>> handleMail() - - >>> transaction.commit() - >>> len(stub.test_emails) - 1 - - >>> from_addr, to_addrs, raw_message = stub.test_emails.pop() - >>> to_addrs == spec.notificationRecipientAddresses() - True - - >>> sent_msg = email.message_from_string(raw_message) - >>> sent_msg['Message-Id'] == moin_change['Message-Id'] - True - - === removed file 'lib/lp/blueprints/mail/handler.py' --- lib/lp/blueprints/mail/handler.py 2012-01-01 02:58:52 +0000 +++ lib/lp/blueprints/mail/handler.py 1970-01-01 00:00:00 +0000 @@ -1,126 +0,0 @@ -# Copyright 2010-2011 Canonical Ltd. This software is licensed under the -# GNU Affero General Public License version 3 (see the file LICENSE). - -"""Handle incoming Blueprints email.""" - -__metaclass__ = type -__all__ = [ - "BlueprintHandler", - ] - -import re -from urlparse import urlunparse - -from zope.component import getUtility -from zope.interface import implements - -from lp.blueprints.interfaces.specification import ISpecificationSet -from lp.services.config import config -from lp.services.mail.helpers import get_main_body -from lp.services.mail.interfaces import IMailHandler -from lp.services.mail.sendmail import sendmail -from lp.services.webapp import urlparse - - -MOIN_URL_RE = re.compile(r'(https?://[^ \r\n]+)') - - -def get_spec_url_from_moin_mail(moin_text): - """Extract a specification URL from Moin change notification.""" - if not isinstance(moin_text, basestring): - return None - match = MOIN_URL_RE.search(moin_text) - if match: - return match.group(1) - else: - return None - - -class BlueprintHandler: - """Handles emails sent to specs.launchpad.net.""" - - implements(IMailHandler) - - allow_unknown_users = True - - _spec_changes_address = re.compile(r'^notifications@.*') - - # The list of hosts where the Ubuntu wiki is located. We could do a - # more general solution, but this kind of setup is unusual, and it - # will be mainly the Ubuntu and Launchpad wikis that will use this - # notification forwarder. - UBUNTU_WIKI_HOSTS = [ - 'wiki.ubuntu.com', 'wiki.edubuntu.org', 'wiki.kubuntu.org'] - - def _getSpecByURL(self, url): - """Returns a spec that is associated with the URL. - - It takes into account that the same Ubuntu wiki is on three - different hosts. - """ - scheme, host, path, params, query, fragment = urlparse(url) - if host in self.UBUNTU_WIKI_HOSTS: - for ubuntu_wiki_host in self.UBUNTU_WIKI_HOSTS: - possible_url = urlunparse( - (scheme, ubuntu_wiki_host, path, params, query, - fragment)) - spec = getUtility(ISpecificationSet).getByURL(possible_url) - if spec is not None: - return spec - else: - return getUtility(ISpecificationSet).getByURL(url) - - def get_spec_url_from_email(self, signed_msg): - """Return the first url found in the email body.""" - mail_body = get_main_body(signed_msg) - return get_spec_url_from_moin_mail(mail_body) - - def process(self, signed_msg, to_addr, filealias=None, log=None): - """See IMailHandler.""" - match = self._spec_changes_address.match(to_addr) - if not match: - # We handle only spec-changes at the moment. - return False - our_address = "notifications@%s" % config.launchpad.specs_domain - # Check for emails that we sent. - xloop = signed_msg['X-Loop'] - if xloop and our_address in signed_msg.get_all('X-Loop'): - if log and filealias: - log.warning( - 'Got back a notification we sent: %s' % - filealias.http_url) - return True - # Check for emails that Launchpad sent us. - if signed_msg['Sender'] == config.canonical.bounce_address: - if log and filealias: - log.warning('We received an email from Launchpad: %s' - % filealias.http_url) - return True - # When sending the email, the sender will be set so that it's - # clear that we're the one sending the email, not the original - # sender. - del signed_msg['Sender'] - - spec_url = self.get_spec_url_from_email(signed_msg) - if spec_url is not None: - if log is not None: - log.debug('Found a spec URL: %s' % spec_url) - spec = self._getSpecByURL(spec_url) - if spec is not None: - if log is not None: - log.debug('Found a corresponding spec: %s' % spec.name) - # Add an X-Loop header, in order to prevent mail loop. - signed_msg.add_header('X-Loop', our_address) - notification_addresses = spec.notificationRecipientAddresses() - if log is not None: - log.debug( - 'Sending notification to: %s' % - ', '.join(notification_addresses)) - sendmail(signed_msg, to_addrs=notification_addresses) - - elif log is not None: - log.debug( - "Didn't find a corresponding spec for %s" % spec_url) - elif log is not None: - log.debug("Didn't find a specification URL") - return True === removed file 'lib/lp/blueprints/mail/tests/test_handler.py' --- lib/lp/blueprints/mail/tests/test_handler.py 2012-01-01 02:58:52 +0000 +++ lib/lp/blueprints/mail/tests/test_handler.py 1970-01-01 00:00:00 +0000 @@ -1,83 +0,0 @@ -# Copyright 2011 Canonical Ltd. This software is licensed under the -# GNU Affero General Public License version 3 (see the file LICENSE). - -"""Testing the Blueprint email handler.""" - -__metaclass__ = type - -from testtools.matchers import ( - Equals, - Is, - ) - -from lp.blueprints.mail.handler import ( - BlueprintHandler, - get_spec_url_from_moin_mail, - ) -from lp.testing import ( - TestCase, - TestCaseWithFactory, - ) -from lp.testing.layers import DatabaseFunctionalLayer - - -class TestGetSpecUrlFromMoinMail(TestCase): - """Tests for get_spec_url_from_moin_mail.""" - - def test_invalid_params(self): - # Only strings and unicode are OK. - self.assertThat(get_spec_url_from_moin_mail(None), Is(None)) - self.assertThat(get_spec_url_from_moin_mail(42), Is(None)) - self.assertThat(get_spec_url_from_moin_mail(object()), Is(None)) - - def test_missing_urls(self): - # Strings with missing URLs also return None - self.assertThat( - get_spec_url_from_moin_mail('nothing here'), - Is(None)) - - def test_string_contains_url(self): - body = """ - Testing a big string - - An url, http://example.com/foo in a string - """ - self.assertThat( - get_spec_url_from_moin_mail(body), - Equals('http://example.com/foo')) - - def test_two_urls(self): - # Given two urls, only the first is returned. - body = """ - Testing two urls: - http://example.com/first - http://example.com/second - """ - self.assertThat( - get_spec_url_from_moin_mail(body), - Equals('http://example.com/first')) - - def test_unicode(self): - # Unicode strings and urls are fine. - body = u""" - Testing unicode: - http://example.com/\N{SNOWMAN} - """ - self.assertThat( - get_spec_url_from_moin_mail(body), - Equals(u'http://example.com/\N{SNOWMAN}')) - - -class TestBlueprintEmailHandler(TestCaseWithFactory): - - layer = DatabaseFunctionalLayer - - def test_find_url_in_multipart_message(self): - """Multipart email is common, and we should be able to handle it.""" - message = self.factory.makeSignedMessage( - body="An url http://example.com/foo in the body.", - attachment_contents="Nothing here either") - handler = BlueprintHandler() - self.assertThat( - handler.get_spec_url_from_email(message), - Equals('http://example.com/foo')) === modified file 'lib/lp/blueprints/tests/test_doc.py' --- lib/lp/blueprints/tests/test_doc.py 2011-12-22 11:12:50 +0000 +++ lib/lp/blueprints/tests/test_doc.py 2012-03-21 02:58:23 +0000 @@ -5,30 +5,13 @@ Run the doctests and pagetests. """ -import logging import os -from lp.services.mail.tests.test_doc import ProcessMailLayer from lp.services.testing import build_test_suite -from lp.testing.systemdocs import ( - LayeredDocFileSuite, - setUp, - tearDown, - ) here = os.path.dirname(os.path.realpath(__file__)) -special = { - 'spec-mail-exploder.txt': LayeredDocFileSuite( - "../doc/spec-mail-exploder.txt", - setUp=setUp, tearDown=tearDown, - layer=ProcessMailLayer, - stdout_logging=True, - stdout_logging_level=logging.WARNING), - } - - def test_suite(): - return build_test_suite(here, special) + return build_test_suite(here) === modified file 'lib/lp/services/mail/handlers.py' --- lib/lp/services/mail/handlers.py 2012-01-01 02:58:52 +0000 +++ lib/lp/services/mail/handlers.py 2012-03-21 02:58:23 +0000 @@ -7,7 +7,6 @@ ] from lp.answers.mail.handler import AnswerTrackerHandler -from lp.blueprints.mail.handler import BlueprintHandler from lp.bugs.mail.handler import MaloneHandler from lp.code.mail.codehandler import CodeHandler from lp.services.config import config @@ -18,7 +17,6 @@ DEFAULT = ( (config.launchpad.bugs_domain, MaloneHandler), - (config.launchpad.specs_domain, BlueprintHandler), (config.answertracker.email_domain, AnswerTrackerHandler), # XXX flacoste 2007-04-23 Backward compatibility for old domain. # We probably want to remove it in the future.
_______________________________________________ Mailing list: https://launchpad.net/~launchpad-reviewers Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-reviewers More help : https://help.launchpad.net/ListHelp
