Re: [PATCH] parser: Fix parsing of pull request emails with CRLF line endings on Python 2
On 08/01/18 18:49, Andrew Donnellan wrote:
On 08/01/18 18:46, Andrew Donnellan wrote:
When using Python 2, an incoming email that uses CRLF line endings won't
have the line endings converted by the email parser. This means that the
regex to detect pull request emails will fail to match.
Clean up the line endings when we extract the email body to fix this. (On
Python 3, the email parser policy fixes this for us at the initial email
parsing stage.)
Add a test pull request mbox with CRLF line endings to ensure we don't
regress.
Hmm, sudden realisation that git handles line endings for us, so perhaps
we don't have an easy way of including a test here.
Also this should probably be applied to stable too...
Closes: #148 ("Parsing pull request emails broken on Python 2")
Signed-off-by: Andrew Donnellan
---
patchwork/parser.py | 4 +
.../mail/0017-git-pull-request-crlf-newlines.mbox | 348
+
patchwork/tests/test_parser.py | 3 +
3 files changed, 355 insertions(+)
create mode 100644
patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
diff --git a/patchwork/parser.py b/patchwork/parser.py
index 1568bc4..7c677db 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -666,9 +666,13 @@ def clean_content(content):
"""Remove cruft from the email message.
Catch signature (-- ) and list footer (_) cruft.
+
+ Change to Unix line endings (the Python 3 email module does this
for us,
+ but not Python 2).
"""
sig_re = re.compile(r'^(-- |_+)\n.*', re.S | re.M)
content = sig_re.sub('', content)
+ content = content.replace('\r\n', '\n')
return content.strip()
diff --git
a/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
b/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
new file mode 100644
index 000..bad78ae
--- /dev/null
+++ b/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
@@ -0,0 +1,348 @@
+From b...@kernel.crashing.org Fri Oct 22 11:51:02 2010
+Return-Path:
+X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
bilbo.ozlabs.org
+X-Spam-Level:
+X-Spam-Status: No, score=0.0 required=3.0 tests=none autolearn=disabled
+ version=3.3.1
+X-Original-To: j...@ozlabs.org
+Delivered-To: j...@ozlabs.org
+Received: from bilbo.ozlabs.org (localhost [127.0.0.1])
+ by ozlabs.org (Postfix) with ESMTP id ED4B3100937
+ for ; Fri, 22 Oct 2010 14:51:54 +1100 (EST)
+Received: by ozlabs.org (Postfix)
+ id BF799B70CB; Fri, 22 Oct 2010 14:51:50 +1100 (EST)
+Delivered-To: linuxppc-...@ozlabs.org
+Received: from gate.crashing.org (gate.crashing.org [63.228.1.57])
+ (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
+ (Client did not present a certificate)
+ by ozlabs.org (Postfix) with ESMTPS id 94629B7043
+ for ; Fri, 22 Oct 2010 14:51:49 +1100 (EST)
+Received: from [IPv6:::1] (localhost.localdomain [127.0.0.1])
+ by gate.crashing.org (8.14.1/8.13.8) with ESMTP id o9M3p3SP018234;
+ Thu, 21 Oct 2010 22:51:04 -0500
+Subject: [git pull] Please pull powerpc.git next branch
+From: Benjamin Herrenschmidt
+To: Linus Torvalds
+Date: Fri, 22 Oct 2010 14:51:02 +1100
+Message-ID: <1287719462.2198.37.camel@pasglop>
+Mime-Version: 1.0
+X-Mailer: Evolution 2.30.3
+Cc: linuxppc-dev list ,
+ Andrew Morton ,
+ Linux Kernel list
+X-BeenThere: linuxppc-...@lists.ozlabs.org
+X-Mailman-Version: 2.1.13
+Precedence: list
+List-Id: Linux on PowerPC Developers Mail List
+List-Unsubscribe: ,
+
+List-Archive:
+List-Post:
+List-Help:
+List-Subscribe: ,
+
+Content-Type: text/plain;
+ charset="us-ascii"
+Content-Transfer-Encoding: 7bit
+Sender: linuxppc-dev-bounces+jk=ozlabs@lists.ozlabs.org
+Errors-To: linuxppc-dev-bounces+jk=ozlabs@lists.ozlabs.org
+X-UID: 11446
+X-Length: 16781
+Status: R
+X-Status: N
+X-KMail-EncryptionState:
+X-KMail-SignatureState:
+X-KMail-MDN-Sent:
+
+Hi Linus !
+
+Here's powerpc's batch for this merge window. Mostly bits and pieces,
+such as Anton doing some performance tuning left and right, and the
+usual churn. One hilight is the support for the new Freescale e5500 core
+(64-bit BookE). Another one is that we now wire up the whole lot of
+socket calls as direct syscalls in addition to the old style indirect
+method.
+
+Cheers,
+Ben.
+
+The following changes since commit
e10117d36ef758da0690c95ecffc09d5dd7da479:
+ Linus Torvalds (1):
+ Merge branch 'upstream-linus' of
git://git.kernel.org/.../jgarzik/libata-dev
+
+are available in the git repository at:
+
+ git://git.kernel.org/pub/scm/linux/
Re: [PATCH] parser: Fix parsing of pull request emails with CRLF line endings on Python 2
On 08/01/18 18:46, Andrew Donnellan wrote:
When using Python 2, an incoming email that uses CRLF line endings won't
have the line endings converted by the email parser. This means that the
regex to detect pull request emails will fail to match.
Clean up the line endings when we extract the email body to fix this. (On
Python 3, the email parser policy fixes this for us at the initial email
parsing stage.)
Add a test pull request mbox with CRLF line endings to ensure we don't
regress.
Hmm, sudden realisation that git handles line endings for us, so perhaps
we don't have an easy way of including a test here.
Closes: #148 ("Parsing pull request emails broken on Python 2")
Signed-off-by: Andrew Donnellan
---
patchwork/parser.py| 4 +
.../mail/0017-git-pull-request-crlf-newlines.mbox | 348 +
patchwork/tests/test_parser.py | 3 +
3 files changed, 355 insertions(+)
create mode 100644
patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
diff --git a/patchwork/parser.py b/patchwork/parser.py
index 1568bc4..7c677db 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -666,9 +666,13 @@ def clean_content(content):
"""Remove cruft from the email message.
Catch signature (-- ) and list footer (_) cruft.
+
+Change to Unix line endings (the Python 3 email module does this for us,
+but not Python 2).
"""
sig_re = re.compile(r'^(-- |_+)\n.*', re.S | re.M)
content = sig_re.sub('', content)
+content = content.replace('\r\n', '\n')
return content.strip()
diff --git a/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox b/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
new file mode 100644
index 000..bad78ae
--- /dev/null
+++ b/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
@@ -0,0 +1,348 @@
+From b...@kernel.crashing.org Fri Oct 22 11:51:02 2010
+Return-Path:
+X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on bilbo.ozlabs.org
+X-Spam-Level:
+X-Spam-Status: No, score=0.0 required=3.0 tests=none autolearn=disabled
+ version=3.3.1
+X-Original-To: j...@ozlabs.org
+Delivered-To: j...@ozlabs.org
+Received: from bilbo.ozlabs.org (localhost [127.0.0.1])
+ by ozlabs.org (Postfix) with ESMTP id ED4B3100937
+ for ; Fri, 22 Oct 2010 14:51:54 +1100 (EST)
+Received: by ozlabs.org (Postfix)
+ id BF799B70CB; Fri, 22 Oct 2010 14:51:50 +1100 (EST)
+Delivered-To: linuxppc-...@ozlabs.org
+Received: from gate.crashing.org (gate.crashing.org [63.228.1.57])
+ (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
+ (Client did not present a certificate)
+ by ozlabs.org (Postfix) with ESMTPS id 94629B7043
+ for ; Fri, 22 Oct 2010 14:51:49 +1100 (EST)
+Received: from [IPv6:::1] (localhost.localdomain [127.0.0.1])
+ by gate.crashing.org (8.14.1/8.13.8) with ESMTP id o9M3p3SP018234;
+ Thu, 21 Oct 2010 22:51:04 -0500
+Subject: [git pull] Please pull powerpc.git next branch
+From: Benjamin Herrenschmidt
+To: Linus Torvalds
+Date: Fri, 22 Oct 2010 14:51:02 +1100
+Message-ID: <1287719462.2198.37.camel@pasglop>
+Mime-Version: 1.0
+X-Mailer: Evolution 2.30.3
+Cc: linuxppc-dev list ,
+ Andrew Morton ,
+ Linux Kernel list
+X-BeenThere: linuxppc-...@lists.ozlabs.org
+X-Mailman-Version: 2.1.13
+Precedence: list
+List-Id: Linux on PowerPC Developers Mail List
+List-Unsubscribe: ,
+
+List-Archive:
+List-Post:
+List-Help:
+List-Subscribe: ,
+
+Content-Type: text/plain;
+ charset="us-ascii"
+Content-Transfer-Encoding: 7bit
+Sender: linuxppc-dev-bounces+jk=ozlabs@lists.ozlabs.org
+Errors-To: linuxppc-dev-bounces+jk=ozlabs@lists.ozlabs.org
+X-UID: 11446
+X-Length: 16781
+Status: R
+X-Status: N
+X-KMail-EncryptionState:
+X-KMail-SignatureState:
+X-KMail-MDN-Sent:
+
+Hi Linus !
+
+Here's powerpc's batch for this merge window. Mostly bits and pieces,
+such as Anton doing some performance tuning left and right, and the
+usual churn. One hilight is the support for the new Freescale e5500 core
+(64-bit BookE). Another one is that we now wire up the whole lot of
+socket calls as direct syscalls in addition to the old style indirect
+method.
+
+Cheers,
+Ben.
+
+The following changes since commit e10117d36ef758da0690c95ecffc09d5dd7da479:
+ Linus Torvalds (1):
+Merge branch 'upstream-linus' of
git://git.kernel.org/.../jgarzik/libata-dev
+
+are available in the git repository at:
+
+ git://git.kernel.org/pub/scm/linux/kernel/git/benh/powerpc.git next
+
+Andreas Schwab (1):
+ power
[PATCH] parser: Fix parsing of pull request emails with CRLF line endings on Python 2
When using Python 2, an incoming email that uses CRLF line endings won't
have the line endings converted by the email parser. This means that the
regex to detect pull request emails will fail to match.
Clean up the line endings when we extract the email body to fix this. (On
Python 3, the email parser policy fixes this for us at the initial email
parsing stage.)
Add a test pull request mbox with CRLF line endings to ensure we don't
regress.
Closes: #148 ("Parsing pull request emails broken on Python 2")
Signed-off-by: Andrew Donnellan
---
patchwork/parser.py| 4 +
.../mail/0017-git-pull-request-crlf-newlines.mbox | 348 +
patchwork/tests/test_parser.py | 3 +
3 files changed, 355 insertions(+)
create mode 100644
patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
diff --git a/patchwork/parser.py b/patchwork/parser.py
index 1568bc4..7c677db 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -666,9 +666,13 @@ def clean_content(content):
"""Remove cruft from the email message.
Catch signature (-- ) and list footer (_) cruft.
+
+Change to Unix line endings (the Python 3 email module does this for us,
+but not Python 2).
"""
sig_re = re.compile(r'^(-- |_+)\n.*', re.S | re.M)
content = sig_re.sub('', content)
+content = content.replace('\r\n', '\n')
return content.strip()
diff --git a/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
b/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
new file mode 100644
index 000..bad78ae
--- /dev/null
+++ b/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
@@ -0,0 +1,348 @@
+From b...@kernel.crashing.org Fri Oct 22 11:51:02 2010
+Return-Path:
+X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on bilbo.ozlabs.org
+X-Spam-Level:
+X-Spam-Status: No, score=0.0 required=3.0 tests=none autolearn=disabled
+ version=3.3.1
+X-Original-To: j...@ozlabs.org
+Delivered-To: j...@ozlabs.org
+Received: from bilbo.ozlabs.org (localhost [127.0.0.1])
+ by ozlabs.org (Postfix) with ESMTP id ED4B3100937
+ for ; Fri, 22 Oct 2010 14:51:54 +1100 (EST)
+Received: by ozlabs.org (Postfix)
+ id BF799B70CB; Fri, 22 Oct 2010 14:51:50 +1100 (EST)
+Delivered-To: linuxppc-...@ozlabs.org
+Received: from gate.crashing.org (gate.crashing.org [63.228.1.57])
+ (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
+ (Client did not present a certificate)
+ by ozlabs.org (Postfix) with ESMTPS id 94629B7043
+ for ; Fri, 22 Oct 2010 14:51:49 +1100 (EST)
+Received: from [IPv6:::1] (localhost.localdomain [127.0.0.1])
+ by gate.crashing.org (8.14.1/8.13.8) with ESMTP id o9M3p3SP018234;
+ Thu, 21 Oct 2010 22:51:04 -0500
+Subject: [git pull] Please pull powerpc.git next branch
+From: Benjamin Herrenschmidt
+To: Linus Torvalds
+Date: Fri, 22 Oct 2010 14:51:02 +1100
+Message-ID: <1287719462.2198.37.camel@pasglop>
+Mime-Version: 1.0
+X-Mailer: Evolution 2.30.3
+Cc: linuxppc-dev list ,
+ Andrew Morton ,
+ Linux Kernel list
+X-BeenThere: linuxppc-...@lists.ozlabs.org
+X-Mailman-Version: 2.1.13
+Precedence: list
+List-Id: Linux on PowerPC Developers Mail List
+List-Unsubscribe: ,
+
+List-Archive:
+List-Post:
+List-Help:
+List-Subscribe: ,
+
+Content-Type: text/plain;
+ charset="us-ascii"
+Content-Transfer-Encoding: 7bit
+Sender: linuxppc-dev-bounces+jk=ozlabs@lists.ozlabs.org
+Errors-To: linuxppc-dev-bounces+jk=ozlabs@lists.ozlabs.org
+X-UID: 11446
+X-Length: 16781
+Status: R
+X-Status: N
+X-KMail-EncryptionState:
+X-KMail-SignatureState:
+X-KMail-MDN-Sent:
+
+Hi Linus !
+
+Here's powerpc's batch for this merge window. Mostly bits and pieces,
+such as Anton doing some performance tuning left and right, and the
+usual churn. One hilight is the support for the new Freescale e5500 core
+(64-bit BookE). Another one is that we now wire up the whole lot of
+socket calls as direct syscalls in addition to the old style indirect
+method.
+
+Cheers,
+Ben.
+
+The following changes since commit e10117d36ef758da0690c95ecffc09d5dd7da479:
+ Linus Torvalds (1):
+Merge branch 'upstream-linus' of
git://git.kernel.org/.../jgarzik/libata-dev
+
+are available in the git repository at:
+
+ git://git.kernel.org/pub/scm/linux/kernel/git/benh/powerpc.git next
+
+Andreas Schwab (1):
+ powerpc: Remove fpscr use from [kvm_]cvt_{fd,df}
+
+Anton Blanchard (5):
+ powerpc: Optimise 64bit csum_partial
+ powerpc: Optimise 64bit csum_partial_copy_generic and add
csum_a
Re: [PATCH] docs/development: Fix tox invocation for listing targets
Andrew Donnellan writes: > Signed-off-by: Andrew Donnellan > --- > docs/development/contributing.rst | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/docs/development/contributing.rst > b/docs/development/contributing.rst > index 0cab79e..8988c9f 100644 > --- a/docs/development/contributing.rst > +++ b/docs/development/contributing.rst > @@ -45,7 +45,7 @@ to do. To start, you can show the default targets like so: > > .. code-block:: shell > > - $ tox --list > + $ tox -l Reviewed-by: Daniel Axtens It turns out what we wanted was --listenvs, not list. Regards, Daniel > > You'll see that this includes a number of targets to run unit tests against > the > different versions of Django supported, along with some other targets related > -- > 2.11.0 > > ___ > Patchwork mailing list > Patchwork@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/patchwork ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: patchwork.ozlabs.org down, and e-mails not recorded
On 06/01/18 07:43, Thomas Petazzoni wrote: Hello, Just a quick note to let you know that from here, patchwork.ozlabs.org seems to be down at this time (or awfully slow). bilbo (i.e. ozlabs.org) required a reboot a couple of days ago for a kernel upgrade iirc. ajd@bilbo:~$ uptime 10:27:36 up 2 days, 1:10, 5 users, load average: 1.38, 1.96, 1.64 Also, since a few weeks, we have seen that patchwork misses some patches that are sent on the Buildroot mailing list. The mails are sent on the mailing list, they are in the mailing list archives, but patchwork does not record them. An example is: http://lists.busybox.net/pipermail/buildroot/2017-December/209885.html PATCH 3/3 was not recorded by patchwork, while other patches in the series have been. Hmm, looking at the patch list, I also see that patch 2/3 in that series (https://patchwork.ozlabs.org/patch/852063/) wasn't correctly identified in the same series as patch 1 (https://patchwork.ozlabs.org/patch/852062/) which is a bit weird. This again happened today. The first send of: http://lists.busybox.net/pipermail/buildroot/2018-January/210797.html only had a few of its patches recorded by patchwork. It was resent a few minutes later: http://lists.busybox.net/pipermail/buildroot/2018-January/210858.html and then all the patches got recorded. This is quite annoying. Is there a way to look in the patchwork logs what has happened with those patches ? I don't have access to the patchwork logs myself - jk, any thoughts? -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
