Re: hacked
Hi Aaron On 6/10/20 8:19 am, Aaron Kaufman wrote: I’ve never set up an ozlabs.org account; I have no idea how I could be your default bounceback email other than you’ve been hacked, These being unsolicited emails, if I count them up and bring a lawsuit, Id get $500 each. And these are starting to get really annoying, and I’m a lawyer, so the only cost to me is the filing fee. Could you clarify exactly what kind of "bounceback" emails you're getting? If you wish to unsubscribe from the list, you can do so from the page at https://lists.ozlabs.org/listinfo/patchwork. Andrew -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] models: Validate Project.linkname does not contain forward slash
On 6/9/20 10:55 pm, Thomas Bracht Laumann Jespersen wrote:
I started by creating a project that contained a forward slash
(importing patches from https://lists.sr.ht/~sircmpwn/sr.ht-dev/) and
it fails to render the "projects" main page.
The specific error reads:
NoReverseMatch at /
Reverse for 'patch-list' with keyword arguments
'{'project_id': 'foo/bar'}' not found. 1 pattern(s) tried:
['project/(?P[^/]+)/list/$']
which appears to explicitly disallow forward slashes.
So I think it makes sense to validate that project linkname doesn't
contain forward slahes.
Signed-off-by: Thomas Bracht Laumann Jespersen
---
I hard a hard time satisfying flake8, so I ended up disabling the pre-commit
hook. I also looked over the documentation for contributors and figure that if
a release note is required, just let me know then I'll add it.
TIL we have precommit hooks. I've been contributing to patchwork for a
few years now...
In any case, it also causes the test suite to fail its flake8 check.
Perhaps we need to exclude the migrations directory?
.../0044_add_project_linkname_validation.py | 19 +++
patchwork/models.py | 10 +-
2 files changed, 28 insertions(+), 1 deletion(-)
create mode 100644
patchwork/migrations/0044_add_project_linkname_validation.py
diff --git a/patchwork/migrations/0044_add_project_linkname_validation.py
b/patchwork/migrations/0044_add_project_linkname_validation.py
new file mode 100644
index 000..2fff7df
--- /dev/null
+++ b/patchwork/migrations/0044_add_project_linkname_validation.py
@@ -0,0 +1,19 @@
+# Generated by Django 3.0.10 on 2020-09-06 22:47
+
+from django.db import migrations, models
+import patchwork.models
+
+
+class Migration(migrations.Migration):
+
+dependencies = [
+('patchwork', '0043_merge_patch_submission'),
+]
+
+operations = [
+migrations.AlterField(
+model_name='project',
+name='linkname',
+field=models.CharField(max_length=255, unique=True,
validators=[patchwork.models.validate_project_linkname]),
+),
+]
diff --git a/patchwork/models.py b/patchwork/models.py
index 77ab924..9027219 100644
--- a/patchwork/models.py
+++ b/patchwork/models.py
@@ -31,6 +31,13 @@ def validate_regex_compiles(regex_string):
raise ValidationError('Invalid regular expression entered!')
+def validate_project_linkname(linkname):
+if re.fullmatch(r'[^/]+', linkname) is None:
This could just be "if '/' in linkname:"
+raise ValidationError(
+'Invalid project linkname: Value cannot contain forward slash (/)'
+)
+
+
class Person(models.Model):
# properties
@@ -56,7 +63,8 @@ class Person(models.Model):
class Project(models.Model):
# properties
-linkname = models.CharField(max_length=255, unique=True)
+linkname = models.CharField(max_length=255, unique=True,
+validators=[validate_project_linkname])
name = models.CharField(max_length=255, unique=True)
listid = models.CharField(max_length=255)
listemail = models.CharField(max_length=200)
--
Andrew Donnellan OzLabs, ADL Canberra
a...@linux.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH 1/5] templates: Replace ifequal and ifnotequal with if
Django 3.1 deprecates the ifequal and ifnotequal tags, for removal in 4.0.
Replace all occurrences of ifequal and ifnotequal with if.
Signed-off-by: Andrew Donnellan
---
.../patchwork/partials/pagination.html| 8
.../patchwork/partials/patch-list.html| 20 +--
patchwork/templates/patchwork/profile.html| 4 ++--
3 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/patchwork/templates/patchwork/partials/pagination.html
b/patchwork/templates/patchwork/partials/pagination.html
index 04f4d16671c2..ee4b555d62c2 100644
--- a/patchwork/templates/patchwork/partials/pagination.html
+++ b/patchwork/templates/patchwork/partials/pagination.html
@@ -1,6 +1,6 @@
{% load listurl %}
-{% ifnotequal page.paginator.num_pages 1 %}
+{% if page.paginator.num_pages != 1 %}
{% if page.has_previous %}
@@ -18,12 +18,12 @@
{% endif %}
{% for p in page.paginator.adjacent_set %}
- {% ifequal p page.number %}
+ {% if p == page.number %}
{{ p }}
{% else %}
{{ p }}
- {% endifequal %}
+ {% endif %}
{% endfor %}
{% if page.paginator.leading_set %}
@@ -42,4 +42,4 @@
{% endif %}
-{% endifnotequal %}
+{% endif %}
diff --git a/patchwork/templates/patchwork/partials/patch-list.html
b/patchwork/templates/patchwork/partials/patch-list.html
index 985e9bee05a0..02d6dff87c97 100644
--- a/patchwork/templates/patchwork/partials/patch-list.html
+++ b/patchwork/templates/patchwork/partials/patch-list.html
@@ -71,7 +71,7 @@ $(document).ready(function() {
{% endif %}
-{% ifequal order.name "name" %}
+{% if order.name == "name" %}
@@ -84,7 +84,7 @@ $(document).ready(function() {
{% else %}
Patch
{% endif %}
-{% endifequal %}
+{% endif %}
@@ -100,7 +100,7 @@ $(document).ready(function() {
-{% ifequal order.name "date" %}
+{% if order.name == "date" %}
@@ -113,11 +113,11 @@ $(document).ready(function() {
{% else %}
Date
{% endif %}
-{% endifequal %}
+{% endif %}
-{% ifequal order.name "submitter" %}
+{% if order.name == "submitter" %}
@@ -132,11 +132,11 @@ $(document).ready(function() {
{% else %}
Submitter
{% endif %}
-{% endifequal %}
+{% endif %}
-{% ifequal order.name "delegate" %}
+{% if order.name == "delegate" %}
@@ -149,11 +149,11 @@ $(document).ready(function() {
{% else %}
Delegate
{% endif %}
-{% endifequal %}
+{% endif %}
-{% ifequal order.name "state" %}
+{% if order.name == "state" %}
@@ -166,7 +166,7 @@ $(document).ready(function() {
{% else %}
State
{% endif %}
-{% endifequal %}
+{% endif %}
diff --git a/patchwork/templates/patchwork/profile.html
b/patchwork/templates/patchwork/profile.html
index 4ca78dae430e..4a4b55826d03 100644
--- a/patchwork/templates/patchwork/profile.html
+++ b/patchwork/templates/patchwork/profile.html
@@ -55,13 +55,13 @@ address.
{{ email.email }}
- {% ifnotequal user.email email.email %}
+ {% if user.email != email.email %}
{% csrf_token %}
-{% endifnotequal %}
+{% endif %}
{% if email.is_optout %}
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH 2/5] urls: Update url pattern functions
Django 3.1 deprecates django.conf.urls.url() as an alias for django.urls.re_path(). Also switch to using django.urls.include() rather than django.conf.urls.include(). Signed-off-by: Andrew Donnellan --- patchwork/urls.py | 331 +++--- 1 file changed, 167 insertions(+), 164 deletions(-) diff --git a/patchwork/urls.py b/patchwork/urls.py index 7d888d4a3dc0..280d28d8e5bd 100644 --- a/patchwork/urls.py +++ b/patchwork/urls.py @@ -4,10 +4,9 @@ # SPDX-License-Identifier: GPL-2.0-or-later from django.conf import settings -from django.conf.urls import url, include from django.contrib import admin from django.contrib.auth import views as auth_views -from django.urls import reverse_lazy +from django.urls import include, re_path, reverse_lazy from patchwork.views import about as about_views from patchwork.views import api as api_views @@ -27,15 +26,16 @@ from patchwork.views import xmlrpc as xmlrpc_views admin.autodiscover() urlpatterns = [ -url(r'^admin/', admin.site.urls), +re_path(r'^admin/', admin.site.urls), -url(r'^$', project_views.project_list, name='project-list'), -url(r'^project/(?P[^/]+)/list/$', patch_views.patch_list, -name='patch-list'), -url(r'^project/(?P[^/]+)/bundles/$', bundle_views.bundle_list, -name='bundle-list'), -url(r'^project/(?P[^/]+)/$', project_views.project_detail, -name='project-detail'), +re_path(r'^$', project_views.project_list, name='project-list'), +re_path(r'^project/(?P[^/]+)/list/$', patch_views.patch_list, +name='patch-list'), +re_path(r'^project/(?P[^/]+)/bundles/$', +bundle_views.bundle_list, +name='bundle-list'), +re_path(r'^project/(?P[^/]+)/$', project_views.project_detail, +name='project-detail'), # patch views # NOTE(dja): Per the RFC, msgids can contain slashes. There doesn't seem @@ -49,128 +49,129 @@ urlpatterns = [ # work, but it is RECOMMENDED by the RFC that the right hand side of the @ # contains a domain, so I think breaking on messages that have "domains" # ending in /raw/ or /mbox/ is good enough. -url(r'^project/(?P[^/]+)/patch/(?P.+)/raw/$', -patch_views.patch_raw, name='patch-raw'), -url(r'^project/(?P[^/]+)/patch/(?P.+)/mbox/$', -patch_views.patch_mbox, name='patch-mbox'), -url(r'^project/(?P[^/]+)/patch/(?P.+)/$', -patch_views.patch_detail, name='patch-detail'), +re_path(r'^project/(?P[^/]+)/patch/(?P.+)/raw/$', +patch_views.patch_raw, name='patch-raw'), +re_path(r'^project/(?P[^/]+)/patch/(?P.+)/mbox/$', +patch_views.patch_mbox, name='patch-mbox'), +re_path(r'^project/(?P[^/]+)/patch/(?P.+)/$', +patch_views.patch_detail, name='patch-detail'), # ... old-style /patch/N/* urls -url(r'^patch/(?P\d+)/raw/$', patch_views.patch_raw_by_id, -name='patch-raw-redirect'), -url(r'^patch/(?P\d+)/mbox/$', patch_views.patch_mbox_by_id, -name='patch-mbox-redirect'), -url(r'^patch/(?P\d+)/$', patch_views.patch_by_id, -name='patch-id-redirect'), +re_path(r'^patch/(?P\d+)/raw/$', patch_views.patch_raw_by_id, +name='patch-raw-redirect'), +re_path(r'^patch/(?P\d+)/mbox/$', patch_views.patch_mbox_by_id, +name='patch-mbox-redirect'), +re_path(r'^patch/(?P\d+)/$', patch_views.patch_by_id, +name='patch-id-redirect'), # cover views -url(r'^project/(?P[^/]+)/cover/(?P.+)/mbox/$', -cover_views.cover_mbox, name='cover-mbox'), -url(r'^project/(?P[^/]+)/cover/(?P.+)/$', -cover_views.cover_detail, name='cover-detail'), +re_path(r'^project/(?P[^/]+)/cover/(?P.+)/mbox/$', +cover_views.cover_mbox, name='cover-mbox'), +re_path(r'^project/(?P[^/]+)/cover/(?P.+)/$', +cover_views.cover_detail, name='cover-detail'), # ... old-style /cover/N/* urls -url(r'^cover/(?P\d+)/mbox/$', cover_views.cover_mbox_by_id, -name='cover-mbox-redirect'), -url(r'^cover/(?P\d+)/$', cover_views.cover_by_id, -name='cover-id-redirect'), +re_path(r'^cover/(?P\d+)/mbox/$', cover_views.cover_mbox_by_id, +name='cover-mbox-redirect'), +re_path(r'^cover/(?P\d+)/$', cover_views.cover_by_id, +name='cover-id-redirect'), # comment views -url(r'^comment/(?P\d+)/$', comment_views.comment, -name='comment-redirect'), +re_path(r'^comment/(?P\d+)/$', comment_views.comment, +name='comment-redirect'), # series views -url(r'^series/(?P\d+)/mbox/$', series_views.series_mbox, -name='series-mbox'), +re_path(r'^series/(?P\d+)/mbox/$', series_views.series_mbox, +name='series-mbox'), # logged-in user stuff -url(r'^user/$', user_views.profile, name='user-profile'), -url(r'^user/todo/$', user_views.todo_lists, -name='user-todos'), -url(r'^use
[PATCH 3/5] settings: Add context processor django.template.context_processors.request
Django 3.1 adds a new admin sidebar feature that requires the django.template.context_processors.request context processor to be enabled in the settings. Signed-off-by: Andrew Donnellan --- patchwork/settings/base.py | 1 + 1 file changed, 1 insertion(+) diff --git a/patchwork/settings/base.py b/patchwork/settings/base.py index 001878acb134..c1bb9b27fac8 100644 --- a/patchwork/settings/base.py +++ b/patchwork/settings/base.py @@ -57,6 +57,7 @@ TEMPLATES = [ 'django.template.context_processors.debug', 'django.template.context_processors.i18n', 'django.template.context_processors.media', +'django.template.context_processors.request', 'django.template.context_processors.static', 'django.template.context_processors.tz', 'django.contrib.messages.context_processors.messages', -- 2.20.1 ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH 5/5] Add support for Django 3.1
Signed-off-by: Andrew Donnellan
---
releasenotes/notes/django-3-1-support-f0450ed3e7983fe2.yaml | 5 +
requirements-dev.txt| 2 +-
requirements-prod.txt | 2 +-
tox.ini | 5 -
4 files changed, 11 insertions(+), 3 deletions(-)
create mode 100644 releasenotes/notes/django-3-1-support-f0450ed3e7983fe2.yaml
diff --git a/releasenotes/notes/django-3-1-support-f0450ed3e7983fe2.yaml
b/releasenotes/notes/django-3-1-support-f0450ed3e7983fe2.yaml
new file mode 100644
index ..dff8af13a846
--- /dev/null
+++ b/releasenotes/notes/django-3-1-support-f0450ed3e7983fe2.yaml
@@ -0,0 +1,5 @@
+---
+features:
+ - |
+`Django 3.1 <https://docs.djangoproject.com/en/dev/releases/3.1/>`_ is
+now supported.
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 022f465deee5..ce306cf3761d 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -1,4 +1,4 @@
-Django~=3.0.0
+Django~=3.1.0
djangorestframework~=3.11.0
django-filter~=2.3.0
django-debug-toolbar~=2.2.0
diff --git a/requirements-prod.txt b/requirements-prod.txt
index dc86ce0c042f..b7ca00b59310 100644
--- a/requirements-prod.txt
+++ b/requirements-prod.txt
@@ -1,4 +1,4 @@
-Django~=3.0.0
+Django~=3.1.0
djangorestframework~=3.11.0
django-filter~=2.3.0
psycopg2-binary~=2.8.0
diff --git a/tox.ini b/tox.ini
index 0e4c78fbff2f..80b43a600bad 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,6 +1,6 @@
[tox]
minversion = 3.2
-envlist = pep8,docs,py{36,37,38}-django{22,30}
+envlist = pep8,docs,py{36,37,38}-django{22,30,31}
skipsdist = true
ignore_basepython_conflict = true
@@ -14,6 +14,9 @@ deps =
django30: django>=3.0,<3.1
django30: djangorestframework>=3.10,<3.12
django30: django-filter>=2.2,<3.0
+django31: django>=3.1,<3.2
+django31: djangorestframework>=3.10,<3.12
+django31: django-filter>=2.3,<3.0
setenv =
DJANGO_SETTINGS_MODULE = patchwork.settings.dev
PYTHONDONTWRITEBYTECODE = 1
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH 4/5] requirements: Update django-filter
Update django-filter dependency to a version that's compatible with Django 3.1. Signed-off-by: Andrew Donnellan --- requirements-dev.txt | 2 +- requirements-prod.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/requirements-dev.txt b/requirements-dev.txt index e5da0b831fd6..022f465deee5 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,6 +1,6 @@ Django~=3.0.0 djangorestframework~=3.11.0 -django-filter~=2.2.0 +django-filter~=2.3.0 django-debug-toolbar~=2.2.0 # django-dbbackup~=3.2.0 -r requirements-test.txt diff --git a/requirements-prod.txt b/requirements-prod.txt index dbd0bd543cf8..dc86ce0c042f 100644 --- a/requirements-prod.txt +++ b/requirements-prod.txt @@ -1,5 +1,5 @@ Django~=3.0.0 djangorestframework~=3.11.0 -django-filter~=2.2.0 +django-filter~=2.3.0 psycopg2-binary~=2.8.0 sqlparse~=0.3.0 -- 2.20.1 ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH] parser: Don't crash when From: is list email but has weird mangle format
get_original_sender() tries to demangle DMARC-mangled From headers, in
the case where the email's From address is the list address. It knows how
to handle Google Groups and Mailman style mangling, where the original
submitter's name will be turned into e.g. "Andrew Donnellan via
linuxppc-dev".
If an email has the From header set to the list address but has a name that
doesn't include " via ", we'll throw an exception because stripped_name
hasn't been set. Sometimes this is because the list name is seemingly
empty, resulting in a mangled name like "Andrew Donnellan via"
without the space after "via" that we detect. Handle this as well as we can
instead, and add a test.
Fixes: 8279a84238c10 ("parser: Unmangle From: headers that have been mangled
for DMARC purposes")
Reported-by: Jeremy Kerr
Signed-off-by: Andrew Donnellan
---
Backport to stable?
---
patchwork/parser.py| 7 +++
patchwork/tests/test_parser.py | 23 +++
2 files changed, 30 insertions(+)
diff --git a/patchwork/parser.py b/patchwork/parser.py
index a09fd754c4be..dce03a4ff827 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -373,6 +373,13 @@ def get_original_sender(mail, name, email):
# Mailman uses the format " via "
# Google Groups uses "'' via "
stripped_name = name[:name.rfind(' via ')].strip().strip("'")
+elif name.endswith(' via'):
+# Sometimes this seems to happen (perhaps if Mailman isn't set up with
+# any list name)
+stripped_name = name[:name.rfind(' via')].strip().strip("'")
+else:
+# We've hit a format that we don't expect
+stripped_name = None
original_from = clean_header(mail.get('X-Original-From', ''))
if original_from:
diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py
index f5631bee8329..a60eb6b4bac9 100644
--- a/patchwork/tests/test_parser.py
+++ b/patchwork/tests/test_parser.py
@@ -366,6 +366,29 @@ class SenderCorrelationTest(TestCase):
self.assertEqual(person_b._state.adding, False)
self.assertEqual(person_b.id, person_a.id)
+def test_weird_dmarc_munging(self):
+project = create_project()
+real_sender = 'Existing Sender '
+munged_sender1 = "'Existing Sender' via <{}>".format(project.listemail)
+munged_sender2 = "'Existing Sender' <{}>".format(project.listemail)
+
+# Unmunged author
+mail = self._create_email(real_sender)
+person_a = get_or_create_author(mail, project)
+person_a.save()
+
+# Munged with no list name
+mail = self._create_email(munged_sender1, None, None, real_sender)
+person_b = get_or_create_author(mail, project)
+self.assertEqual(person_b._state.adding, False)
+self.assertEqual(person_b.id, person_a.id)
+
+# Munged with no 'via'
+mail = self._create_email(munged_sender2, None, None, real_sender)
+person_b = get_or_create_author(mail, project)
+self.assertEqual(person_b._state.adding, False)
+self.assertEqual(person_b.id, person_a.id)
+
class SeriesCorrelationTest(TestCase):
"""Validate correct behavior of find_series."""
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 2/2] api: allow filtering patches and covers by msgid
On 14/4/20 4:21 pm, Daniel Axtens wrote:
In the process of fixing the previous bug, I realised that:
a) /api/patches/msgid is a perfectly reasonable thing to attempt
b) We have no way of finding a patch by message id in the API
We can't actualy make /api/patches/msgid work because it may not
be unique, but we can add a filter.
I'm shoehorning this into stable/2.2, even though it's technically
an API change: it's minor, not incompatible and in hindsight a
glaring hole.
Cc: Michael Ellerman
Signed-off-by: Daniel Axtens
This is indeed something we would expect to be able to do in the API now
that we support lookup by message ID for the web.
Reviewed-by: Andrew Donnellan
---
docs/api/schemas/latest/patchwork.yaml | 16
docs/api/schemas/patchwork.j2 | 18 ++
docs/api/schemas/v1.2/patchwork.yaml | 16
patchwork/api/filters.py | 14 ++
patchwork/tests/api/test_cover.py | 12
patchwork/tests/api/test_patch.py | 12
.../rest-filter-msgid-41f693cd4e53cf93.yaml| 6 ++
7 files changed, 90 insertions(+), 4 deletions(-)
create mode 100644 releasenotes/notes/rest-filter-msgid-41f693cd4e53cf93.yaml
diff --git a/docs/api/schemas/latest/patchwork.yaml
b/docs/api/schemas/latest/patchwork.yaml
index 13cdc9cd78fd..cc0d97e696b6 100644
--- a/docs/api/schemas/latest/patchwork.yaml
+++ b/docs/api/schemas/latest/patchwork.yaml
@@ -246,6 +246,14 @@ paths:
schema:
title: ''
type: string
+- in: query
+ name: msgid
+ description: >
+The cover message-id as a case-sensitive string, without leading or
+trailing angle brackets, to filter by.
+ schema:
+title: ''
+type: string
responses:
'200':
description: ''
@@ -474,6 +482,14 @@ paths:
schema:
title: ''
type: string
+- in: query
+ name: msgid
+ description: >
+The patch message-id as a case-sensitive string, without leading or
+trailing angle brackets, to filter by.
+ schema:
+title: ''
+type: string
responses:
'200':
description: ''
diff --git a/docs/api/schemas/patchwork.j2 b/docs/api/schemas/patchwork.j2
index bd714d5e7a2a..f5618d41faa0 100644
--- a/docs/api/schemas/patchwork.j2
+++ b/docs/api/schemas/patchwork.j2
@@ -251,6 +251,16 @@ paths:
schema:
title: ''
type: string
+{% if version >= (1, 2) %}
+- in: query
+ name: msgid
+ description: >
+The cover message-id as a case-sensitive string, without leading or
+trailing angle brackets, to filter by.
+ schema:
+title: ''
+type: string
+{% endif %}
responses:
'200':
description: ''
@@ -488,6 +498,14 @@ paths:
schema:
title: ''
type: string
+- in: query
+ name: msgid
+ description: >
+The patch message-id as a case-sensitive string, without leading or
+trailing angle brackets, to filter by.
+ schema:
+title: ''
+type: string
{% endif %}
responses:
'200':
diff --git a/docs/api/schemas/v1.2/patchwork.yaml
b/docs/api/schemas/v1.2/patchwork.yaml
index db2ed122eec2..7bdbe66997c0 100644
--- a/docs/api/schemas/v1.2/patchwork.yaml
+++ b/docs/api/schemas/v1.2/patchwork.yaml
@@ -246,6 +246,14 @@ paths:
schema:
title: ''
type: string
+- in: query
+ name: msgid
+ description: >
+The cover message-id as a case-sensitive string, without leading or
+trailing angle brackets, to filter by.
+ schema:
+title: ''
+type: string
responses:
'200':
description: ''
@@ -474,6 +482,14 @@ paths:
schema:
title: ''
type: string
+- in: query
+ name: msgid
+ description: >
+The patch message-id as a case-sensitive string, without leading or
+trailing angle brackets, to filter by.
+ schema:
+title: ''
+type: string
responses:
'200':
description: ''
diff --git a/patchwork/api/filters.py b/patchwork/api/filters.py
index deb5ace11880..93e6281bf5e6 100644
--- a/patchwork/api/filters.py
+++ b/patchwork/api/filters.py
@@ -184,6 +184,10 @@ class SeriesFilterSet(TimestampMixin, BaseFilterSet):
fields = ('submitter', 'project')
+def msgid_filter(queryset, name, value):
+return queryset.filter(**{name: '<' + value + '>'})
+
+
Re: [PATCH 3/5] Revert "Be sensible computing project patch counts"
On 4/3/20 10:54 pm, Stephen Finucane wrote:
This reverts commit cfcf2f2a80ac0709f1a5fd9aa212c8403daa5a18.
This will no longer be necessary once we remove the Patch-Submission
split. Revert it now to avoid needing to rejig this later.
Conflicts:
patchwork/views/project.py
NOTE(stephenfin): Conflicts are due to commit 880ec8c5 ("Fetch
maintainer information in one query") which changed nearby lines.
Signed-off-by: Stephen Finucane
Cc: Daniel Axtens
Looks fine
Reviewed-by: Andrew Donnellan
---
patchwork/views/project.py | 29 -
1 file changed, 4 insertions(+), 25 deletions(-)
diff --git a/patchwork/views/project.py b/patchwork/views/project.py
index 8fa41794..4c25f715 100644
--- a/patchwork/views/project.py
+++ b/patchwork/views/project.py
@@ -10,10 +10,9 @@ from django.shortcuts import get_object_or_404
from django.shortcuts import render
from django.urls import reverse
+from patchwork.models import Patch
from patchwork.models import Project
-from django.db import connection
-
def project_list(request):
projects = Project.objects.all()
@@ -31,34 +30,14 @@ def project_list(request):
def project_detail(request, project_id):
project = get_object_or_404(Project, linkname=project_id)
-
-# So, we revert to raw sql because if we do what you'd think would
-# be the correct thing in Django-ese, it ends up doing a *pointless*
-# join with patchwork_submissions that ends up ruining the query.
-# So, we do not do this, as this is wrong:
-#
-# patches = Patch.objects.filter(
-# patch_project_id=project.id).only('archived')
-# patches = patches.annotate(c=Count('archived'))
-#
-# and instead do this, because it's simple and fast
-
-n_patches = {}
-with connection.cursor() as cursor:
-cursor.execute('SELECT archived,COUNT(submission_ptr_id) as c '
- 'FROM patchwork_patch '
- 'WHERE patch_project_id=%s GROUP BY archived',
- [project.id])
-
-for r in cursor:
-n_patches[r[0]] = r[1]
+patches = Patch.objects.filter(project=project)
context = {
'project': project,
'maintainers': User.objects.filter(
profile__maintainer_projects=project).select_related('profile'),
-'n_patches': n_patches[False] if False in n_patches else 0,
-'n_archived_patches': n_patches[True] if True in n_patches else 0,
+'n_patches': patches.filter(archived=False).count(),
+'n_archived_patches': patches.filter(archived=True).count(),
'enable_xmlrpc': settings.ENABLE_XMLRPC,
}
return render(request, 'patchwork/project.html', context)
--
Andrew Donnellan OzLabs, ADL Canberra
a...@linux.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 2/5] Remove unnecessary references to Submission model
On 4/3/20 10:54 pm, Stephen Finucane wrote:
We want to drop this in future changes. Start by removing any
unnecessary references.
this = Submission model I assume
Signed-off-by: Stephen Finucane
This patch needs a trivial rebasing.
Other than that
Reviewed-by: Andrew Donnellan
---
patchwork/admin.py | 6 ++
patchwork/views/cover.py | 10 ++
patchwork/views/patch.py | 10 ++
3 files changed, 14 insertions(+), 12 deletions(-)
diff --git a/patchwork/admin.py b/patchwork/admin.py
index f9a94c6f..d4ab109c 100644
--- a/patchwork/admin.py
+++ b/patchwork/admin.py
@@ -19,7 +19,6 @@ from patchwork.models import Project
from patchwork.models import Series
from patchwork.models import SeriesReference
from patchwork.models import State
-from patchwork.models import Submission
from patchwork.models import Tag
from patchwork.models import UserProfile
@@ -75,15 +74,14 @@ class StateAdmin(admin.ModelAdmin):
admin.site.register(State, StateAdmin)
-class SubmissionAdmin(admin.ModelAdmin):
+class CoverLetterAdmin(admin.ModelAdmin):
list_display = ('name', 'submitter', 'project', 'date')
list_filter = ('project', )
search_fields = ('name', 'submitter__name', 'submitter__email')
date_hierarchy = 'date'
-admin.site.register(Submission, SubmissionAdmin)
-admin.site.register(CoverLetter, SubmissionAdmin)
+admin.site.register(CoverLetter, CoverLetterAdmin)
class PatchAdmin(admin.ModelAdmin):
diff --git a/patchwork/views/cover.py b/patchwork/views/cover.py
index 54962abb..e90b7373 100644
--- a/patchwork/views/cover.py
+++ b/patchwork/views/cover.py
@@ -11,8 +11,8 @@ from django.shortcuts import render
from django.urls import reverse
from patchwork.models import CoverLetter
+from patchwork.models import Patch
from patchwork.models import Project
-from patchwork.models import Submission
from patchwork.views.utils import cover_to_mbox
@@ -25,9 +25,11 @@ def cover_detail(request, project_id, msgid):
cover = get_object_or_404(CoverLetter, project_id=project.id,
msgid=db_msgid)
except Http404 as exc:
-submissions = Submission.objects.filter(project_id=project.id,
-msgid=db_msgid)
-if submissions:
+patches = Patch.objects.filter(
+project_id=project.id,
+msgid=db_msgid,
+)
+if patches:
return HttpResponseRedirect(
reverse('patch-detail',
kwargs={'project_id': project.linkname,
diff --git a/patchwork/views/patch.py b/patchwork/views/patch.py
index f34053ce..bc508bee 100644
--- a/patchwork/views/patch.py
+++ b/patchwork/views/patch.py
@@ -15,9 +15,9 @@ from django.urls import reverse
from patchwork.forms import CreateBundleForm
from patchwork.forms import PatchForm
from patchwork.models import Bundle
+from patchwork.models import CoverLetter
from patchwork.models import Patch
from patchwork.models import Project
-from patchwork.models import Submission
from patchwork.views import generic_list
from patchwork.views.utils import patch_to_mbox
from patchwork.views.utils import series_patch_to_mbox
@@ -42,9 +42,11 @@ def patch_detail(request, project_id, msgid):
try:
patch = Patch.objects.get(project_id=project.id, msgid=db_msgid)
except Patch.DoesNotExist as exc:
-submissions = Submission.objects.filter(project_id=project.id,
-msgid=db_msgid)
-if submissions:
+covers = CoverLetter.objects.filter(
+project_id=project.id,
+msgid=db_msgid,
+)
+if covers:
return HttpResponseRedirect(
reverse('cover-detail',
kwargs={'project_id': project.linkname,
--
Andrew Donnellan OzLabs, ADL Canberra
a...@linux.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 1/5] trivial: Rename 'CoverLetter' references to 'Cover'
On 4/3/20 10:54 pm, Stephen Finucane wrote: We're going to be doing some model surgery shortly. Do the necessary renaming of variables ahead of this. Signed-off-by: Stephen Finucane Was confused by this until I realised that this patch doesn't rename CoverLetter, just the other classes that have "CoverLetter" in the name... Anyway, this does appear to remove any references to "CoverLetter" other than CoverLetter and CoverLetterAdmin. Reviewed-by: Andrew Donnellan -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: Feature request: Patch tracking in custom lists for logged users
On 19/2/20 11:40 am, Matthias Kaehlcke wrote: Hi patchworkers! I'm a regular user of patchwork and have a feature in mind that would simplify my workflow and probably be also useful for others: On some lists the amount of patches can be overwhelming, often I see a see a patch/series and have just enough time to make a mental (or actual) note that I should review, test, integrate ... this patch or series. It would be great if this sort of tracking could be done in patchwork itself, i.e. allow logged users to create/manage custom lists of patches. Probably you already have enough on your plates, but it would be great if you could consider to implement this feature eventually :) Patchwork does support "bundles" - if you select patches from the list, then use the bundling form at the bottom of the list to create a bundle, you can keep them grouped together. Does that do at least some of what you'd be looking for? -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] models: Add State.slug field
On 11/12/19 2:32 am, Stephen Finucane wrote: I think the migration can still fail without printing a helpful exception message if two states have names that are different but still slugify to the same thing? Good point. I can fix this if we think it's worth it. Personally, I think the uniqueness check itself is overkill and could probably be removed. Agreed, though if we're gonna do it I want it done thoroughly ;) -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] models: Add State.slug field
On 9/12/19 3:00 am, Stephen Finucane wrote:
--- /dev/null
+++ b/patchwork/migrations/0038_state_slug.py
@@ -0,0 +1,68 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import unicode_literals
+
+from django.db import migrations, models, transaction
+from django.utils.text import slugify
+
+
+def validate_uniqueness(apps, schema_editor):
+"""Ensure all State.name entries are unique.
+
+We need to do this before enforcing a uniqueness constraint.
+"""
+
+State = apps.get_model('patchwork', 'State')
+
+unique_count = len(State.objects.order_by().values_list(
+'name', flat=True).distinct())
+total_count = State.objects.count()
+
+if unique_count != total_count:
+raise Exception(
+'You have non-unique States entries that need to be combined '
+'before you can run this migration. This migration must be done '
+'by hand. If you need assistance, please contact '
+'patchw...@ozlabs.org')
I think the migration can still fail without printing a helpful
exception message if two states have names that are different but still
slugify to the same thing?
--
Andrew Donnellan OzLabs, ADL Canberra
a...@linux.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] Disable i18n machinery, use correct locale
On 8/11/19 1:01 am, Daniel Axtens wrote: Stephen Finucane writes: Two issues here. Firstly, the use of the 'USE_I18N'. The Django docs describe this as such: A boolean that specifies whether Django’s translation system should be enabled. This provides an easy way to turn it off, for performance. If this is set to False, Django will make some optimizations so as not to load the translation machinery. We don't do translations and won't until such a time as someone comes asking for them. Optimize things accordingly by setting 'USE_I18N' to False and removing the now-unnecessary 'LANGUAGE_CODE' setting. Secondly, the use of en-AU is a bit of a lie since our UI is actually written in US English (or should be). The primary reason for a lang tag to be present is to assist screenreaders and other accessibility tools, so make their lives easier by reflecting the truth. I am not sold on the primacy of US English, and from a historical point of view the UI was absolutely written in Australian English, but before I go down that rabbit-hole, what are the actual implications of setting en-AU vs en-US for anything that might parse the web page? If there's actually a case where the difference would matter, I'd be much happer to consider changing it. I don't suppose there's a plain "en" language code? There is a plain en language code, and I strongly ACK its use over the incorrect so-called English, en_US. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH v2] Improve pull request URL matching regex
On 17/11/19 3:16 am, Konstantin Ryabitsev wrote: When git-request-pull output is pasted into a mail client instead of mailed directly, the ref part of the pull URL may end up wrapped to the next line. Example: https://lore.kernel.org/r/294422a4-37b2-def5-5d32-8988f27c3...@gmail.com/ This change properly parses URLs both with and without newlines. Signed-off-by: Konstantin Ryabitsev Thanks! Reviewed-by: Andrew Donnellan -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: DB-murdering API query (index suggestions needed)
On 15/11/19 11:44 am, Konstantin Ryabitsev wrote: Hi, all: Today, the DB behind patchwork.kernel.org was in a semi-permanent state of suffering due to someone trying to suck down all patches in the linux-arm-kernel project. This is what the API request looked like: GET /api/1.1/patches/?project=62=2019-11-01T00:00:00_page=100=6150 The query behind this takes about 1 minute to run on a 20-core HT Xeon system and requires creating a huge temporary file (there are 18375 patches in that project). So, two questions, really: 1. Any indexes we can put in place to make this query perform better? 2. Is there a way to disable anonymous API access? Not currently, but it would be fairly easy to do... https://github.com/getpatchwork/patchwork/issues/325 -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH v3 2/3] Include the responsible actor in applicable events
On 15/11/19 9:37 am, Johan Herland wrote:>>> The remaining events (cover-created, series-created) are both triggered by incoming emails, hence have no real actor as such, so we simply leave the actor as None/NULL. How is cover-created different from patch-created? In practice, it turns out there is no difference, really: The patch-created event is triggered by a signal from the Patch model which potentially carries the ._edited_by attribute. However, AFAICS, when patches are created, there is no preceding call to Patch.is_editable(), hence Patch._edited_by is not set, and we end up passing actor=None to Event.objects.create(). The cover-created event is triggered by a signal from CoverLetter which has no ._edited_by, hence we pass no actor to Events.objects.create() and it defaults to None. I still figured I'd wire up the logic for patch-created, just in case we at some point were to start setting ._edited_by on Patch objects when they are first created. OK. Personally I'd rather we not wire it up, as it's confusing to see something like that wired up and wonder why the field isn't being set. It's also not clear how you should set the actor on email-triggered events as not every sender has a Patchwork account. Andrew -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] templates: Override page title based on set filters
On 15/1/19 12:25 am, Ali Alnubani wrote:
Set the document's title based on the applied filters,
giving priority to the series' title, and then to the
submitter's name if the series filter wasn't applied.
Suggested-by: Thomas Monjalon
Signed-off-by: Ali Alnubani
I see what you're trying to do here.
I don't think this should be done in JavaScript, until we fulfil our
grand dreams of rewriting the entire web frontend as something fancier
than what it is, this should be rendered in the template server-side.
---
patchwork/templates/patchwork/partials/filters.html | 5 +
1 file changed, 5 insertions(+)
diff --git a/patchwork/templates/patchwork/partials/filters.html
b/patchwork/templates/patchwork/partials/filters.html
index 41ed2c2..9566cc0 100644
--- a/patchwork/templates/patchwork/partials/filters.html
+++ b/patchwork/templates/patchwork/partials/filters.html
@@ -115,6 +115,11 @@ $(document).ready(function() {
}
});
});
+{% if "series" in filters.applied_filters %}
+document.title = "{{ filters.applied_filters.series.condition }}";
+{% elif "submitter" in filters.applied_filters %}
+document.title = "{{ filters.applied_filters.submitter.condition }}";
+{% endif %}
--
Andrew Donnellan OzLabs, ADL Canberra
a...@linux.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH v3 3/3] /api/events: Add 'actor' field to generated JSON
On 17/10/19 9:44 am, Johan Herland wrote: Cc: Mauro Carvalho Chehab Signed-off-by: Johan Herland Acked-by: Daniel Axtens Looks alright Reviewed-by: Andrew Donnellan -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH v3 2/3] Include the responsible actor in applicable events
On 17/10/19 9:44 am, Johan Herland wrote: We want to use the events as an audit log. An important part of this is recording _who_ made the changes that the events represent. To accomplish this, we need to know the current user (aka. request.user) at the point where we create the Event instance. Event creation is currently triggered by signals, but neither the signal handlers, nor the model classes themselves have easy access to request.user. For Patch-based events (patch-created, patch-state-changed, patch-delegated, patch-completed), we can do the following hack: The relevant events are created in signal handlers that are all hooked up to either the pre_save or post_save signals sent by Patch.save(). But before calling Patch.save(), Patchwork must naturally query Patch.is_editable() to ascertain whether the patch can in fact be changed by the current user. Thus, we only need a way to communicate the current user from Patch.is_editable() to the signal handlers that create the resulting Events. The Patch object itself is available in both places, so we simply add an ._edited_by attribute to the instance (which fortunately is not detected as a persistent db field by Django). The series-completed event is also triggered by Patch.save(), so uses the same trick as above. For the check-created event the current user always happens to be the same as the .user field recorded in the Check object itself. The remaining events (cover-created, series-created) are both triggered by incoming emails, hence have no real actor as such, so we simply leave the actor as None/NULL. How is cover-created different from patch-created? The rest of this seems reasonable. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH v3 1/3] models.Event: Add the user responsible for the event
On 17/10/19 9:44 am, Johan Herland wrote:
This allows using the events as a kind of audit log, to see how a
patch came to its current state/delegate.
Cc: Mauro Carvalho Chehab
Signed-off-by: Johan Herland
Reviewed-by: Stephen Finucane
Reviewed-by: Andrew Donnellan
---
patchwork/migrations/0037_event_actor.py | 21 +
patchwork/models.py | 4
2 files changed, 25 insertions(+)
create mode 100644 patchwork/migrations/0037_event_actor.py
diff --git a/patchwork/migrations/0037_event_actor.py
b/patchwork/migrations/0037_event_actor.py
new file mode 100644
index 000..6607228
--- /dev/null
+++ b/patchwork/migrations/0037_event_actor.py
@@ -0,0 +1,21 @@
+# Generated by Django 2.2.6 on 2019-10-08 04:21
+
+from django.conf import settings
+from django.db import migrations, models
+import django.db.models.deletion
+
+
+class Migration(migrations.Migration):
+
+dependencies = [
+migrations.swappable_dependency(settings.AUTH_USER_MODEL),
+('patchwork', '0036_project_commit_url_format'),
+]
+
+operations = [
+migrations.AddField(
+model_name='event',
+name='actor',
+field=models.ForeignKey(blank=True, help_text='The user that
caused/created this event.', null=True,
on_delete=django.db.models.deletion.SET_NULL, related_name='+',
to=settings.AUTH_USER_MODEL),
+),
+]
diff --git a/patchwork/models.py b/patchwork/models.py
index c198bc2..b43c15a 100644
--- a/patchwork/models.py
+++ b/patchwork/models.py
@@ -944,6 +944,10 @@ class Event(models.Model):
date = models.DateTimeField(
default=datetime.datetime.utcnow,
help_text='The time this event was created.')
+actor = models.ForeignKey(
+User, related_name='+', null=True, blank=True,
+on_delete=models.SET_NULL,
+help_text='The user that caused/created this event.')
# event object
--
Andrew Donnellan OzLabs, ADL Canberra
a...@linux.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] Python 3.8 support
On 5/11/19 2:18 pm, Stephen Finucane wrote: On Thu, 2019-10-24 at 14:10 +1100, Andrew Donnellan wrote: Enable Python 3.8 in our tests and list it as a supported version. Signed-off-by: Andrew Donnellan LGTM, but let's hold off on this until 2.2 is out. Given there are no code changes necessary (AFAICT) for Python 3.8 I'd rather merge this now, so we avoid adding regressions in stable backports during the period where distros start transitioning to 3.8. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH 1/2] templates: Get rid of type attribute in script tags
In HTML5, the type attribute of a script tag is optional if it's
JavaScript.
Remove all occurrences. The only real gain is slightly smaller page output,
but it also shuts up validators that like to be noisy about this.
Signed-off-by: Andrew Donnellan
---
patchwork/templates/patchwork/bundle.html | 4 ++--
patchwork/templates/patchwork/login.html | 2 +-
patchwork/templates/patchwork/submission.html | 2 +-
templates/base.html | 14 +++---
4 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/patchwork/templates/patchwork/bundle.html
b/patchwork/templates/patchwork/bundle.html
index b5c9f90a4afd..411c18b5b345 100644
--- a/patchwork/templates/patchwork/bundle.html
+++ b/patchwork/templates/patchwork/bundle.html
@@ -4,8 +4,8 @@
{% load static %}
{% block headers %}
-
-
+
+
{% endblock %}
{% block title %}{{project.name}}{% endblock %}
diff --git a/patchwork/templates/patchwork/login.html
b/patchwork/templates/patchwork/login.html
index e8b1a4e8f242..46f77edf6984 100644
--- a/patchwork/templates/patchwork/login.html
+++ b/patchwork/templates/patchwork/login.html
@@ -4,7 +4,7 @@
{% block heading %}Sign in to Patchwork{% endblock %}
{% block headers %}
-
+ <script>
$(function() {
$('#id_username').focus()
});
diff --git a/patchwork/templates/patchwork/submission.html
b/patchwork/templates/patchwork/submission.html
index 6fa816397bd0..1fa89f945da1 100644
--- a/patchwork/templates/patchwork/submission.html
+++ b/patchwork/templates/patchwork/submission.html
@@ -8,7 +8,7 @@
{% block title %}{{submission.name}}{% endblock %}
{% block body %}
-<script type="text/javascript">
+<script>
function toggle_div(link_id, headers_id)
{
var link = document.getElementById(link_id)
diff --git a/templates/base.html b/templates/base.html
index 802ea98b0745..01b0d6b5e598 100644
--- a/templates/base.html
+++ b/templates/base.html
@@ -7,9 +7,9 @@
<link rel="stylesheet" type="text/css" href="{% static
"css/bootstrap.min.css" %}"/>
<link rel="stylesheet" type="text/css" href="{% static
"css/selectize.bootstrap3.css" %}"/>
<link rel="stylesheet" type="text/css" href="{% static "css/style.css" %}"/>
- <script type="text/javascript" src="{% static "js/jquery-1.10.1.min.js"
%}">
-
-
+
+
+
-
-
-
-
+ <script src="{% static "js/bootstrap.min.js" %}">
+
+
+
[PATCH 2/2] templates: Specify language
Specifying language in the tag is recommended in HTML5.
Signed-off-by: Andrew Donnellan
---
templates/base.html | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/templates/base.html b/templates/base.html
index 01b0d6b5e598..40b6cda65350 100644
--- a/templates/base.html
+++ b/templates/base.html
@@ -1,6 +1,6 @@
{% load static %}
-http://www.w3.org/1999/xhtml;>
+http://www.w3.org/1999/xhtml; lang="en-AU">
{% block title %}Patchwork{% endblock %} - Patchwork
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH v2] templates: Move download buttons outside h1 tag
It's not valid to put a inside an . Move the download buttons in
the submission template outside the tag.
Signed-off-by: Andrew Donnellan
---
v1->v2
- make it look the same as before (Daniel)
---
patchwork/templates/patchwork/submission.html | 16
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/patchwork/templates/patchwork/submission.html
b/patchwork/templates/patchwork/submission.html
index b5b55dbd9241..b3849789b1fd 100644
--- a/patchwork/templates/patchwork/submission.html
+++ b/patchwork/templates/patchwork/submission.html
@@ -27,9 +27,10 @@ function toggle_div(link_id, headers_id)
}
-{{ submission.name }}
-{% include "patchwork/partials/download-buttons.html" %}
-
+
+ {% include "patchwork/partials/download-buttons.html" %}
+ {{ submission.name }}
+
@@ -273,11 +274,10 @@ function toggle_div(link_id, headers_id)
{% endfor %}
{% if submission.diff %}
-
- Patch
- {% include "patchwork/partials/download-buttons.html" %}
-
-
+
+ {% include "patchwork/partials/download-buttons.html" %}
+ Patch
+
{{ submission|patchsyntax }}
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH v2 1/2] [PW3] tests: Fix escaping in bundle tests on Django 3.0
Django 3.0 switches to using Python 3's built-in HTML escaper, which
prefers to escape entities using hex rather than decimal.
Some of our tests check rendered HTML output against pre-escaped strings,
and fail because '' is now ''.
Fix this by using the escape function so we get consistent escaping no
matter which Django version..
Signed-off-by: Andrew Donnellan
---
patchwork/tests/test_bundles.py | 14 --
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/patchwork/tests/test_bundles.py b/patchwork/tests/test_bundles.py
index 63f943c033d6..c5e7ee62f435 100644
--- a/patchwork/tests/test_bundles.py
+++ b/patchwork/tests/test_bundles.py
@@ -10,6 +10,7 @@ import unittest
from django.conf import settings
from django.test import TestCase
from django.urls import reverse
+from django.utils.html import escape
from django.utils.http import urlencode
from patchwork.models import Bundle
@@ -548,8 +549,8 @@ class BundleAddFromListTest(BundleTestBase):
'project_id': self.project.linkname}),
params)
-self.assertContains(response, 'Patch %s already in bundle'
-% patch.name, count=1, status_code=200)
+expected = escape(f"Patch '{patch.name}' already in bundle")
+self.assertContains(response, expected, count=1, status_code=200)
self.assertEqual(count, self.bundle.patches.count())
@@ -570,11 +571,12 @@ class BundleAddFromListTest(BundleTestBase):
'project_id': self.project.linkname}),
params)
-self.assertContains(response, 'Patch %s already in bundle'
-% patch.name, count=1, status_code=200)
-self.assertContains(response, 'Patch %s added to bundle'
-% self.patches[1].name, count=1,
+expected_already = escape(f"Patch '{patch.name}' already in bundle")
+expected_added = escape(
+f"Patch '{self.patches[1].name}' added to bundle")
+self.assertContains(response, expected_already, count=1,
status_code=200)
+self.assertContains(response, expected_added, count=1, status_code=200)
self.assertEqual(count + 1, self.bundle.patches.count())
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH v2 2/2] [PW3] Add Django 3.0 support
Now that we've dropped Python 2, we can get ready for Django 3.0, which is
still in beta.
Add Django 3.0b1 as a tox environment. Add a release note for Django 3.0
support, as it will be released by the time Patchwork 3.0 is out.
Closes: #311 ("Django 3.0 support")
Signed-off-by: Andrew Donnellan
---
v1->v2:
- rebase on top of python 3.8 patch [github.com/ajdlinux/patchwork, django3
branch]
- add release note as suggested by dja
Not adding upper bounds on the version numbers until 3.0 actually drops.
I'm considering this as closing the issue for Django 3.0 support, as now
that Django 3.0 is in beta it's unlikely to get further breaking changes
and we'll catch anything as we test it.
---
releasenotes/notes/django-3-0-support-763b5c7b5aab0010.yaml | 5 +
tox.ini | 5 -
2 files changed, 9 insertions(+), 1 deletion(-)
create mode 100644 releasenotes/notes/django-3-0-support-763b5c7b5aab0010.yaml
diff --git a/releasenotes/notes/django-3-0-support-763b5c7b5aab0010.yaml
b/releasenotes/notes/django-3-0-support-763b5c7b5aab0010.yaml
new file mode 100644
index ..a21c70540404
--- /dev/null
+++ b/releasenotes/notes/django-3-0-support-763b5c7b5aab0010.yaml
@@ -0,0 +1,5 @@
+---
+features:
+ - |
+`Django 3.0 <https://docs.djangoproject.com/en/dev/releases/3.0/>`_ is now
+supported. This requires Python 3.6 or higher.
diff --git a/tox.ini b/tox.ini
index 6be8141e7c66..693d02de3c40 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,6 +1,6 @@
[tox]
minversion = 2.0
-envlist = pep8,docs,py{36,37,38}-django{20,21,22}
+envlist = pep8,docs,py{36,37,38}-django{20,21,22,30b1}
skipsdist = True
[testenv]
@@ -13,6 +13,9 @@ deps =
django22: django>=2.2,<2.3
django22: djangorestframework>=3.10,<3.11
django22: django-filter>=2.1,<3.0
+django30b1: django==3.0b1
+django30b1: djangorestframework>=3.10
+django30b1: django-filter>=2.2
setenv =
DJANGO_SETTINGS_MODULE = patchwork.settings.dev
PYTHONDONTWRITEBYTECODE = 1
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH v2] Python 3.8 support
Enable Python 3.8 in our tests and list it as a supported version.
Signed-off-by: Andrew Donnellan
---
v1->v2:
- Add release note that I forgot to git add
---
README.rst | 2 +-
releasenotes/notes/python-3-8-support-59150fb6391f9b73.yaml | 5 +
tools/docker/Dockerfile | 3 ++-
tox.ini | 2 +-
4 files changed, 9 insertions(+), 3 deletions(-)
create mode 100644 releasenotes/notes/python-3-8-support-59150fb6391f9b73.yaml
diff --git a/README.rst b/README.rst
index 01da41969168..074ded9a7db2 100644
--- a/README.rst
+++ b/README.rst
@@ -41,7 +41,7 @@ of community projects.
Requirements
-- Python (2.7, 3.5 - 3.7)
+- Python (2.7, 3.5 - 3.8)
- Django (1.11 - 2.2)
diff --git a/releasenotes/notes/python-3-8-support-59150fb6391f9b73.yaml
b/releasenotes/notes/python-3-8-support-59150fb6391f9b73.yaml
new file mode 100644
index ..935cfb91a260
--- /dev/null
+++ b/releasenotes/notes/python-3-8-support-59150fb6391f9b73.yaml
@@ -0,0 +1,5 @@
+---
+features:
+ - |
+`Python 3.8 <https://www.python.org/downloads/release/python-380/>`_ is now
+supported.
diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile
index 61678134d418..5ef11203e850 100644
--- a/tools/docker/Dockerfile
+++ b/tools/docker/Dockerfile
@@ -49,7 +49,8 @@ RUN curl -L
https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-instal
RUN pyenv latest install 2.7 && \
pyenv latest install 3.5 && \
pyenv latest install 3.6 && \
-pyenv latest install 3.7
+pyenv latest install 3.7 && \
+pyenv latest install 3.8
RUN pyenv global $(pyenv versions --bare | tac)
RUN pip install tox tox-pyenv
diff --git a/tox.ini b/tox.ini
index 617e73cdbee6..4267f47241b6 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,6 +1,6 @@
[tox]
minversion = 2.0
-envlist = pep8,docs,py27-django111,py{35,36,37}-django{111,20,21,22}
+envlist = pep8,docs,py27-django111,py{35,36,37,38}-django{111,20,21,22}
skipsdist = True
[testenv]
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH] Python 3.8 support
Enable Python 3.8 in our tests and list it as a supported version.
Signed-off-by: Andrew Donnellan
---
README.rst | 2 +-
tools/docker/Dockerfile | 3 ++-
tox.ini | 2 +-
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/README.rst b/README.rst
index 01da41969168..074ded9a7db2 100644
--- a/README.rst
+++ b/README.rst
@@ -41,7 +41,7 @@ of community projects.
Requirements
-- Python (2.7, 3.5 - 3.7)
+- Python (2.7, 3.5 - 3.8)
- Django (1.11 - 2.2)
diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile
index 61678134d418..5ef11203e850 100644
--- a/tools/docker/Dockerfile
+++ b/tools/docker/Dockerfile
@@ -49,7 +49,8 @@ RUN curl -L
https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-instal
RUN pyenv latest install 2.7 && \
pyenv latest install 3.5 && \
pyenv latest install 3.6 && \
-pyenv latest install 3.7
+pyenv latest install 3.7 && \
+pyenv latest install 3.8
RUN pyenv global $(pyenv versions --bare | tac)
RUN pip install tox tox-pyenv
diff --git a/tox.ini b/tox.ini
index 617e73cdbee6..4267f47241b6 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,6 +1,6 @@
[tox]
minversion = 2.0
-envlist = pep8,docs,py27-django111,py{35,36,37}-django{111,20,21,22}
+envlist = pep8,docs,py27-django111,py{35,36,37,38}-django{111,20,21,22}
skipsdist = True
[testenv]
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] parser: extend SERIES_DELAY_INTERVAL
On 21/10/19 4:44 pm, Daniel Axtens wrote: There was a series on linuxppc today that was spread over ~13 mins, so the last two patches were put into a new series. Extend the time window to 20 mins, and attempt to document it. Signed-off-by: Daniel Axtens Reviewed-by: Andrew Donnellan --- patchwork/parser.py | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git patchwork/parser.py patchwork/parser.py index be1e51652dd3..c794f093c412 100644 --- patchwork/parser.py +++ patchwork/parser.py @@ -35,7 +35,12 @@ _hunk_re = re.compile(r'^\@\@ -\d+(?:,(\d+))? \+\d+(?:,(\d+))? \@\@') _filename_re = re.compile(r'^(---|\+\+\+) (\S+)') list_id_headers = ['List-ID', 'X-Mailing-List', 'X-list'] -SERIES_DELAY_INTERVAL = 10 +# How many minutes must pass since the first email of a series before we +# say that subsequent mails are definitely not part of that same series? +# +# Only used when there are not proper references to determine the series +# (such as when the mail is not threaded) +SERIES_DELAY_INTERVAL = 20 # @see https://git-scm.com/docs/git-diff#_generating_patches_with_p EXTENDED_HEADER_LINES = ( -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH 1/2] [PW3] tests: Fix escaping in bundle tests on Django 3.0
Django 3.0 switches to using Python 3's built-in HTML escaper, which
prefers to escape entities using hex rather than decimal.
Some of our tests check rendered HTML output against pre-escaped strings,
and fail because '' is now ''.
Fix this by using the escape function so we get consistent escaping no
matter which Django version..
Signed-off-by: Andrew Donnellan
---
patchwork/tests/test_bundles.py | 14 --
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/patchwork/tests/test_bundles.py b/patchwork/tests/test_bundles.py
index 63f943c033d6..c5e7ee62f435 100644
--- a/patchwork/tests/test_bundles.py
+++ b/patchwork/tests/test_bundles.py
@@ -10,6 +10,7 @@ import unittest
from django.conf import settings
from django.test import TestCase
from django.urls import reverse
+from django.utils.html import escape
from django.utils.http import urlencode
from patchwork.models import Bundle
@@ -548,8 +549,8 @@ class BundleAddFromListTest(BundleTestBase):
'project_id': self.project.linkname}),
params)
-self.assertContains(response, 'Patch %s already in bundle'
-% patch.name, count=1, status_code=200)
+expected = escape(f"Patch '{patch.name}' already in bundle")
+self.assertContains(response, expected, count=1, status_code=200)
self.assertEqual(count, self.bundle.patches.count())
@@ -570,11 +571,12 @@ class BundleAddFromListTest(BundleTestBase):
'project_id': self.project.linkname}),
params)
-self.assertContains(response, 'Patch %s already in bundle'
-% patch.name, count=1, status_code=200)
-self.assertContains(response, 'Patch %s added to bundle'
-% self.patches[1].name, count=1,
+expected_already = escape(f"Patch '{patch.name}' already in bundle")
+expected_added = escape(
+f"Patch '{self.patches[1].name}' added to bundle")
+self.assertContains(response, expected_already, count=1,
status_code=200)
+self.assertContains(response, expected_added, count=1, status_code=200)
self.assertEqual(count + 1, self.bundle.patches.count())
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH 2/2] [PW3] tox: Add Django 3.0b1
Now that we've dropped Python 2, we can get ready for Django 3.0.
Add Django 3.0b1 as a tox environment.
Closes: #311 ("Django 3.0 support")
Signed-off-by: Andrew Donnellan
---
Not adding upper bounds on the version numbers until 3.0 actually drops.
I'm considering this as closing the issue for Django 3.0 support, as now
that Django 3.0 is in beta it's unlikely to get further breaking changes
and we'll catch anything as we test it.
---
tox.ini | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/tox.ini b/tox.ini
index ae15ce815f26..8ee11de56d14 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,6 +1,6 @@
[tox]
minversion = 2.0
-envlist = pep8,docs,py{36,37}-django{20,21,22}
+envlist = pep8,docs,py{36,37}-django{20,21,22,30b1}
skipsdist = True
[testenv]
@@ -13,6 +13,9 @@ deps =
django22: django>=2.2,<2.3
django22: djangorestframework>=3.10,<3.11
django22: django-filter>=2.1,<3.0
+django30b1: django==3.0b1
+django30b1: djangorestframework>=3.10
+django30b1: django-filter>=2.2
setenv =
DJANGO_SETTINGS_MODULE = patchwork.settings.dev
PYTHONDONTWRITEBYTECODE = 1
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH 1/2] templates: Move download buttons outside h1 tag
It's not valid to put a inside an . Move the download buttons in
the submission template outside the tag.
Signed-off-by: Andrew Donnellan
---
patchwork/templates/patchwork/submission.html | 9 +++--
1 file changed, 3 insertions(+), 6 deletions(-)
diff --git a/patchwork/templates/patchwork/submission.html
b/patchwork/templates/patchwork/submission.html
index b5b55dbd9241..407f9ece85fb 100644
--- a/patchwork/templates/patchwork/submission.html
+++ b/patchwork/templates/patchwork/submission.html
@@ -27,9 +27,8 @@ function toggle_div(link_id, headers_id)
}
-{{ submission.name }}
{% include "patchwork/partials/download-buttons.html" %}
-
+{{ submission.name }}
@@ -273,10 +272,8 @@ function toggle_div(link_id, headers_id)
{% endfor %}
{% if submission.diff %}
-
- Patch
- {% include "patchwork/partials/download-buttons.html" %}
-
+{% include "patchwork/partials/download-buttons.html" %}
+Patch
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH 2/2] templates: Fix mismatched close tags
There's a rather than in the bundle list. Fix it. Signed-off-by: Andrew Donnellan --- patchwork/templates/patchwork/bundles.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patchwork/templates/patchwork/bundles.html b/patchwork/templates/patchwork/bundles.html index 749aaedcf010..1bb3b0da71ed 100644 --- a/patchwork/templates/patchwork/bundles.html +++ b/patchwork/templates/patchwork/bundles.html @@ -14,7 +14,7 @@ Bundle Project Public - Patches + Patches Download Delete -- 2.20.1 ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH v2] parser: Unmangle From: headers that have been mangled for DMARC purposes
On 18/10/19 3:06 pm, Daniel Axtens wrote: I'll apply it shortly to master. You could convince me to get it backported to stable if you really wanted. Do we need something to go through the db and fix up old ones? Or is it best to let sleeping patches lie? Personally I was thinking we should leave previous patches as-is, it seems like a fair bit of extra code for fairly small gain. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] tests: Rename inaccurately named test_patchwork_from_header
On 18/10/19 3:02 pm, Daniel Axtens wrote: Andrew Donnellan writes: The test_patchwork_from_header test claims to test for the presence of the X-Patchwork-From header, when we actually call it X-Patchwork-Sender. X-Patchwork-Submitter, surely? Wow, the irony of me getting this wrong in the commit message... I get it right in the actual commit! Can you fix in merge or do you want me to respin? (I didn't even know that one existed!) Regards, Daniel Fix it. Signed-off-by: Andrew Donnellan --- patchwork/tests/test_mboxviews.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/patchwork/tests/test_mboxviews.py b/patchwork/tests/test_mboxviews.py index 3854a856c4db..ab0c1669944f 100644 --- a/patchwork/tests/test_mboxviews.py +++ b/patchwork/tests/test_mboxviews.py @@ -153,8 +153,8 @@ class MboxHeaderTest(TestCase): patch.url_msgid])) self.assertContains(response, 'X-Patchwork-Delegate: %s' % user.email) -def test_patchwork_from_header(self): -"""Validate inclusion of generated 'X-Patchwork-From' header.""" +def test_patchwork_submitter_header(self): +"""Validate inclusion of generated 'X-Patchwork-Submitter' header.""" email = 'j...@doe.com' from_header = 'From: Jon Doe <%s>\n' % email -- 2.20.1 ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH] tests: Rename inaccurately named test_patchwork_from_header
The test_patchwork_from_header test claims to test for the presence of the X-Patchwork-From header, when we actually call it X-Patchwork-Sender. Fix it. Signed-off-by: Andrew Donnellan --- patchwork/tests/test_mboxviews.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/patchwork/tests/test_mboxviews.py b/patchwork/tests/test_mboxviews.py index 3854a856c4db..ab0c1669944f 100644 --- a/patchwork/tests/test_mboxviews.py +++ b/patchwork/tests/test_mboxviews.py @@ -153,8 +153,8 @@ class MboxHeaderTest(TestCase): patch.url_msgid])) self.assertContains(response, 'X-Patchwork-Delegate: %s' % user.email) -def test_patchwork_from_header(self): -"""Validate inclusion of generated 'X-Patchwork-From' header.""" +def test_patchwork_submitter_header(self): +"""Validate inclusion of generated 'X-Patchwork-Submitter' header.""" email = 'j...@doe.com' from_header = 'From: Jon Doe <%s>\n' % email -- 2.20.1 ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH v2] parser: Unmangle From: headers that have been mangled for DMARC purposes
To avoid triggering spam filters due to failed signature validation, many
mailing lists mangle the From header to change the From address to be the
address of the list, typically where the sender's domain has a strict DMARC
policy enabled.
In this case, we should try to unmangle the From header.
Add support for using the X-Original-From or Reply-To headers, as used by
Google Groups and Mailman respectively, to unmangle the From header when
necessary and associate the patch with the correct submitter based on the
unmangled email address.
When downloading mboxes, rewrite the From header using the unmangled
address, and preserve the original header as X-Patchwork-Original-From in
case someone needs it for some reason. The original From header will still
be stored in the database and exposed via the API, as we want to keep
messages as close to the original received format as possible.
Closes: #64 ("Incorrect submitter when using googlegroups")
Reported-by: Alexandre Belloni
Reported-by: Stephen Rothwell
Signed-off-by: Andrew Donnellan
---
v1->v2:
- use X-Original-From rather than X-Original-Sender
- unmangle From header when downloading mbox
rewrite from header
Signed-off-by: Andrew Donnellan
use x original from
Signed-off-by: Andrew Donnellan
---
patchwork/parser.py | 75 +++
patchwork/tests/test_mboxviews.py | 21 +
patchwork/tests/test_parser.py| 68 ++--
patchwork/views/utils.py | 12 +
4 files changed, 163 insertions(+), 13 deletions(-)
diff --git a/patchwork/parser.py b/patchwork/parser.py
index 7dc66bc05a5b..be1e51652dd3 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -321,12 +321,7 @@ def find_series(project, mail, author):
return _find_series_by_markers(project, mail, author)
-def get_or_create_author(mail):
-from_header = clean_header(mail.get('From'))
-
-if not from_header:
-raise ValueError("Invalid 'From' header")
-
+def split_from_header(from_header):
name, email = (None, None)
# tuple of (regex, fn)
@@ -355,6 +350,65 @@ def get_or_create_author(mail):
(name, email) = fn(match.groups())
break
+return (name, email)
+
+
+# Unmangle From addresses that have been mangled for DMARC purposes.
+#
+# To avoid triggering spam filters due to failed signature validation, many
+# mailing lists mangle the From header to change the From address to be the
+# address of the list, typically where the sender's domain has a strict
+# DMARC policy enabled.
+#
+# Unfortunately, there's no standardised way of preserving the original
+# From address.
+#
+# Google Groups adds an X-Original-From header. If present, we use that.
+#
+# Mailman preserves the original address by adding a Reply-To, except in the
+# case where the list is set to either reply to list, or reply to a specific
+# address, in which case the original From is added to Cc instead. These corner
+# cases are dumb, but we try and handle things as sensibly as possible by
+# looking for a name in Reply-To/Cc that matches From. It's inexact but should
+# be good enough for our purposes.
+def get_original_sender(mail, name, email):
+if name and ' via ' in name:
+# Mailman uses the format " via "
+# Google Groups uses "'' via "
+stripped_name = name[:name.rfind(' via ')].strip().strip("'")
+
+original_from = clean_header(mail.get('X-Original-From', ''))
+if original_from:
+new_email = split_from_header(original_from)[1].strip()[:255]
+return (stripped_name, new_email)
+
+addrs = []
+reply_to_headers = mail.get_all('Reply-To') or []
+cc_headers = mail.get_all('Cc') or []
+for header in reply_to_headers + cc_headers:
+header = clean_header(header)
+addrs = header.split(",")
+for addr in addrs:
+new_name, new_email = split_from_header(addr)
+if new_name:
+new_name = new_name.strip()[:255]
+if new_email:
+new_email = new_email.strip()[:255]
+if new_name == stripped_name:
+return (stripped_name, new_email)
+
+# If we can't figure out the original sender, just keep it as is
+return (name, email)
+
+
+def get_or_create_author(mail, project=None):
+from_header = clean_header(mail.get('From'))
+
+if not from_header:
+raise ValueError("Invalid 'From' header")
+
+name, email = split_from_header(from_header)
+
if not email:
raise ValueError("Invalid 'From' header")
@@ -362,6 +416,9 @@ def get_or_create_author(mail):
if name is not None:
name = name.strip()[:255]
+if project and email.lower() == project.listemail.lower():
+name, email = get_original_sender(mail, name, email)
+
# this correctly handles the case where we lose the race to crea
Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes
On 12/10/19 2:51 am, Jeff King wrote: On Sat, Oct 12, 2019 at 02:42:49AM +1100, Daniel Axtens wrote: where a possible solution was to get senders to use in-body From headers even when sending their own patches. [...] I'm not sure this solution is correct. If I take a patch from Andrew, backport it, and send to the list, Andrew will be listed in the in-body From. However, he shouldn't be the sender from the Patchwork point of view: he shouldn't get the patch status notification emails - I should. We don't want to spam an original author if their patch is backported to several different releases, or picked up and resent in someone else's series, etc etc. So unless I've misunderstood something, we can't rely on the in-body from matching Patchwork's understanding of the sender. Yeah, it may be that patchwork and git have two different priorities here. From my perspective, the problem is getting the patch into a git repo with the right author name. But patchwork may want to make the distinction between author and sender. Yes, I was referring to the git am case, not the Patchwork case. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes
On 11/10/19 3:36 pm, Andrew Donnellan wrote: It would be nice if Mailman could adopt X-Original-Sender too. As it is, (which I have gone ahead and reported as https://gitlab.com/mailman/mailman/issues/641) -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes
On 11/10/19 3:29 pm, Junio C Hamano wrote: Jeff King writes: This might provide an alternate solution (or vice versa). I kind of like this one better in that it doesn't require the sender to do anything differently (but it may be less robust, as it assumes the receiver reliably de-mangling). I share the assessment. I also feel that relying on Reply-To: would make the result a lot less reliable (I do not have much problem with the use of X-Original-Sender, though). It would be nice if Mailman could adopt X-Original-Sender too. As it is, it adds the original sender to Reply-To, but in some cases (where the list is set as reply-to-list, or has a custom reply-to setting) it adds to Cc instead. (In the patch that started this thread, I match the name from the munged From field against the name in Reply-To/Cc for the case where there's multiple Reply-Tos/Ccs.) For the Patchwork use case, I'm quite okay with accepting the risk of using Reply-To, as the alternative is worse, the corner cases are rare, and ultimately a maintainer can still fix the odd stuff-up before applying the patch. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes
On 11/10/19 9:54 am, Jeff King wrote: Neat. There was discussion on a similar issue recently in: https://public-inbox.org/git/305577c2-709a-b632-4056-658277117...@redhat.com/ where a possible solution was to get senders to use in-body From headers even when sending their own patches. I think that's a good idea. This might provide an alternate solution (or vice versa). I kind of like this one better in that it doesn't require the sender to do anything differently (but it may be less robust, as it assumes the receiver reliably de-mangling). Yep, it's less robust - but OTOH there's always a long tail of users stuck on old versions of git for whatever reason and having some logic to detect DMARC munging may thus still be useful. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes
On 11/10/19 6:41 am, Jonathan Nieder wrote: Interesting! I'm cc-ing the Git mailing list in case "git am" might wnat to learn the same support. Argh, that reminds me... this patch only rewrites the name and email that is recorded as the Patchwork submitter, it doesn't actually rewrite the From: header when you fetch the mbox off Patchwork. Part of me would really like to keep Patchwork mboxes as close as possible to the mbox we ingested, but on the other hand it means the mangled address is still going to land in the git repo at the end... so I should probably just change it? -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes
To avoid triggering spam filters due to failed signature validation, many
mailing lists mangle the From header to change the From address to be the
address of the list, typically where the sender's domain has a strict DMARC
policy enabled.
In this case, we should try to unmangle the From header.
Add support for using the X-Original-Sender or Reply-To headers, as used by
Google Groups and Mailman respectively, to unmangle the From header when
necessary.
Closes: #64 ("Incorrect submitter when using googlegroups")
Reported-by: Alexandre Belloni
Reported-by: Stephen Rothwell
Signed-off-by: Andrew Donnellan
---
patchwork/parser.py| 75 ++
patchwork/tests/test_parser.py | 68 --
2 files changed, 130 insertions(+), 13 deletions(-)
diff --git a/patchwork/parser.py b/patchwork/parser.py
index 7dc66bc05a5b..79beac5617b1 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -321,12 +321,7 @@ def find_series(project, mail, author):
return _find_series_by_markers(project, mail, author)
-def get_or_create_author(mail):
-from_header = clean_header(mail.get('From'))
-
-if not from_header:
-raise ValueError("Invalid 'From' header")
-
+def split_from_header(from_header):
name, email = (None, None)
# tuple of (regex, fn)
@@ -355,6 +350,65 @@ def get_or_create_author(mail):
(name, email) = fn(match.groups())
break
+return (name, email)
+
+
+# Unmangle From addresses that have been mangled for DMARC purposes.
+#
+# To avoid triggering spam filters due to failed signature validation, many
+# mailing lists mangle the From header to change the From address to be the
+# address of the list, typically where the sender's domain has a strict
+# DMARC policy enabled.
+#
+# Unfortunately, there's no standardised way of preserving the original
+# From address.
+#
+# Google Groups adds an X-Original-Sender header. If present, we use that.
+#
+# Mailman preserves the original address by adding a Reply-To, except in the
+# case where the list is set to either reply to list, or reply to a specific
+# address, in which case the original From is added to Cc instead. These corner
+# cases are dumb, but we try and handle things as sensibly as possible by
+# looking for a name in Reply-To/Cc that matches From. It's inexact but should
+# be good enough for our purposes.
+def get_original_sender(mail, name, email):
+if name and ' via ' in name:
+# Mailman uses the format " via "
+# Google Groups uses "'' via "
+stripped_name = name[:name.rfind(' via ')].strip().strip("'")
+
+original_sender = clean_header(mail.get('X-Original-Sender', ''))
+if original_sender:
+new_email = split_from_header(original_sender)[1].strip()[:255]
+return (stripped_name, new_email)
+
+addrs = []
+reply_to_headers = mail.get_all('Reply-To') or []
+cc_headers = mail.get_all('Cc') or []
+for header in reply_to_headers + cc_headers:
+header = clean_header(header)
+addrs = header.split(",")
+for addr in addrs:
+new_name, new_email = split_from_header(addr)
+if new_name:
+new_name = new_name.strip()[:255]
+if new_email:
+new_email = new_email.strip()[:255]
+if new_name == stripped_name:
+return (stripped_name, new_email)
+
+# If we can't figure out the original sender, just keep it as is
+return (name, email)
+
+
+def get_or_create_author(mail, project=None):
+from_header = clean_header(mail.get('From'))
+
+if not from_header:
+raise ValueError("Invalid 'From' header")
+
+name, email = split_from_header(from_header)
+
if not email:
raise ValueError("Invalid 'From' header")
@@ -362,6 +416,9 @@ def get_or_create_author(mail):
if name is not None:
name = name.strip()[:255]
+if project and email.lower() == project.listemail.lower():
+name, email = get_original_sender(mail, name, email)
+
# this correctly handles the case where we lose the race to create
# the person and another process beats us to it. (If the record
# does not exist, g_o_c invokes _create_object_from_params which
@@ -1004,7 +1061,7 @@ def parse_mail(mail, list_id=None):
if not is_comment and (diff or pull_url): # patches or pull requests
# we delay the saving until we know we have a patch.
-author = get_or_create_author(mail)
+author = get_or_create_author(mail, project)
delegate = find_delegate_by_header(mail)
if not delegate and diff:
@@ -1099,7 +1156,7 @@ def parse_mail(mail, list_id=None):
is_cover_letter = True
if is_cover_letter:
-author = get_or_create_author(mail)
+author = get_or_create_author(m
Re: [RFC PATCH] docker: Add support for using eatmydata in the database
On 17/9/19 11:36 pm, Stephen Finucane wrote: On Fri, 2019-05-03 at 17:33 +1000, Russell Currey wrote: When running tox on a VM with presumably pretty busy spinning disks, using eatmydata with the database took running one configuration's test suite from (no exaggeration) 20 minutes down to 60 seconds. It makes a huge difference to test speed, so we should make it easily available for developers. The primary motivation here was to automatically test each patch in a timeframe that isn't insane. Open to ideas on how to organise this, whether we do it for MySQL too (which we probably should), whether the base directory should have these files in it, what to call the Dockerfile, etc. I think it's a good thing to have in the repo, though. Signed-off-by: Russell Currey What are the implications of doing this _from a development perspective_ and can we do this by default in the two existing docker- compose files? Given that we're only talking about a development environmnent where information presumably isn't that important allied to the fact that we have the ability to back up and restore from known good points (the dbbackup, dbrestore management commands, respectively), do we need to be seriously concerned about dataloss? We don't need to worry about dataloss in a development environment, though perhaps some people are borrowing our dockerfiles for a production deployment? -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] templates: Use 'static' rather than 'statictags' library
On 16/9/19 1:50 pm, Geert Stappers wrote: IMHO should commit message and actual change match. Currently I don't see "statictags" matching "staticfiles". Please make them matching. Too late, somehow myself, Daniel and Stephen all managed to miss this before merge... I must have had the phrase "template tags" running around in my head when I wrote the commit message. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] templates: Use 'static' rather than 'statictags' library
On 13/9/19 10:25 pm, Stephen Finucane wrote:
On Fri, 2019-09-13 at 12:19 +0100, Andrew Donnellan wrote:
statictags is being renamed to static, use of {% load statictags %} is
deprecated and will break in Django 3.
Signed-off-by: Andrew Donnellan
I was trying to figure out if this was okay for Django 1.11 as well as
2.x. I eventually found the release note for it [1] and indeed, this
looks like a Django 1.10+ thing so we're golden.
Reviewed-by: Stephen Finucane
...and it's already merged :)
I did run the test suite on quite a few combinations to double check!
--
Andrew Donnellan OzLabs, ADL Canberra
a...@linux.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH] templates: Use 'static' rather than 'statictags' library
statictags is being renamed to static, use of {% load statictags %} is
deprecated and will break in Django 3.
Signed-off-by: Andrew Donnellan
---
templates/base.html | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/templates/base.html b/templates/base.html
index 501208f01e59..802ea98b0745 100644
--- a/templates/base.html
+++ b/templates/base.html
@@ -1,4 +1,4 @@
-{% load staticfiles %}
+{% load static %}
http://www.w3.org/1999/xhtml;>
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Fwd: New list for people to share maintainer workflows
For everyone's information. Forwarded Message Subject: New list for people to share maintainer workflows Date: Fri, 13 Sep 2019 03:38:49 -0400 From: Theodore Y. Ts'o To: ksummit-disc...@lists.linuxfoundation.org CC: linux-ker...@vger.kernel.org, workfl...@vger.kernel.org At the Maintainer's Summit yesterday, we created a new mailing list: workfl...@vger.kernel.org, where various Maintainers can share their workflows for handling patch review, collection, testing, and submission. We will also be discussing what requirements should be for infrastructure that will be best suited for common use. That is, if we can find rough agreement on what it is needed to make kernel development more scalable, efficient, and fun, what would we hand as a set of requirements to a development team that could be funded by our various sponsors to turn into reality. Please note that this is a discussion about *requirements* not implementation. So for example, if one of the requirements were: "it should be like patchwork, in that it is fully compatible with e-mail review, except it should work off-line and use something like git as its transport layer" it doesn't mean that patchwork would be used as its implementation layer. Or, if there is a requirement such as, "it should have a web interface, and it should be easy to pull down patch series via a git pull, and we should be able to easily view diffs between the v49 and v50 version of the patch series" ... it also doesn't follow that Gerrit should be the basis of the implentation. (In fact, both of these requirements were expressed as requirements at the Maintainer's Summit, and there was general agreement that both of these would be highly desirable as requirements.) Of course, if the Gerrit and patchwork teams would like to participate in the discussion, and work with a smaller working group we will be forming to refine the requirements and to work with the LF and other companies to find said funding to implement what these requirements should look like, that would be excellent. I think it's fair to say that in their current forms, *neither* Gerrit nor patchwork completely meets all of the needs of the kernel development community as a whole, and as Dmitry stated in his "Reflections on Kernel Development Process" talk, spreading out our efforts towards improving engineering productivity may not be the best path to succeess. Let's continue the discussion on workfl...@vger.kernel.org. Cheers, - Ted ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] docker: pyenv: build at make -j$(nproc)
On 12/9/19 4:41 pm, Daniel Axtens wrote: This speeds up builds. I haven't measured by how much, but I have observed 8 threads being complied rather than 1 on my laptop. Signed-off-by: Daniel Axtens Reviewed-by: Andrew Donnellan --- tools/docker/Dockerfile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index 35324b13c66a..dfc9c2f3dab1 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -43,7 +43,8 @@ RUN curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-instal git clone https://github.com/momo-lab/xxenv-latest $PYENV_ROOT/plugins/xxenv-latest && \ pyenv update -RUN pyenv latest install 2.7 && \ +RUN export MAKE_OPTS=-j$(nproc) && \ +pyenv latest install 2.7 && \ pyenv latest install 3.5 && \ pyenv latest install 3.6 && \ pyenv latest install 3.7 -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH v2 5/7] docs: Add API v1.2
Add API v1.2, including the new fields for list archive URLs. Signed-off-by: Andrew Donnellan --- v1->v2: - switch to using format string (Daniel) --- docs/api/rest/index.rst| 11 +- docs/api/rest/schemas/v1.1.rst |4 +- docs/api/rest/schemas/v1.2.rst |5 + docs/api/schemas/generate_schema.py|4 +- docs/api/schemas/latest/patchwork.yaml | 57 +- docs/api/schemas/patchwork.j2 | 69 + docs/api/schemas/v1.2/patchwork.yaml | 2369 7 files changed, 2512 insertions(+), 7 deletions(-) create mode 100644 docs/api/rest/schemas/v1.2.rst create mode 100644 docs/api/schemas/v1.2/patchwork.yaml diff --git a/docs/api/rest/index.rst b/docs/api/rest/index.rst index 1a094d732594..d1169e56e07a 100644 --- a/docs/api/rest/index.rst +++ b/docs/api/rest/index.rst @@ -33,6 +33,11 @@ If all you want is reference guides, skip straight to :ref:`rest-api-schemas`. The API version was bumped to v1.1 in Patchwork v2.1. The older v1.0 API is still supported. For more information, refer to :ref:`rest-api-versions`. +.. versionchanged:: 2.2 + + The API version was bumped to v1.2 in Patchwork v2.2. The older APIs are + still supported. For more information, refer to :ref:`rest-api-versions`. + Getting Started --- @@ -98,7 +103,7 @@ Versioning -- By default, all requests will receive the latest version of the API: currently -``1.1``: +``1.2``: .. code-block:: http @@ -109,7 +114,7 @@ changes breaking your application: .. code-block:: http -GET /api/1.1 HTTP/1.1 +GET /api/1.2 HTTP/1.1 Older API versions will be deprecated and removed over time. For more information, refer to :ref:`rest-api-versions`. @@ -263,6 +268,7 @@ Supported Versions 1.0, 2.0, ✓ 1.1, 2.1, ✓ + 1.2, 2.2, ✓ Further information about this and more can typically be found in :doc:`the release notes `. @@ -278,6 +284,7 @@ Auto-generated schema documentation is provided below. /api/rest/schemas/v1.0 /api/rest/schemas/v1.1 + /api/rest/schemas/v1.2 .. Links diff --git a/docs/api/rest/schemas/v1.1.rst b/docs/api/rest/schemas/v1.1.rst index e18f81322688..1189f31571a5 100644 --- a/docs/api/rest/schemas/v1.1.rst +++ b/docs/api/rest/schemas/v1.1.rst @@ -1,5 +1,5 @@ -API v1.1 (latest) -= +API v1.1 + .. openapi:: ../../schemas/v1.1/patchwork.yaml :examples: diff --git a/docs/api/rest/schemas/v1.2.rst b/docs/api/rest/schemas/v1.2.rst new file mode 100644 index ..8a96519b7356 --- /dev/null +++ b/docs/api/rest/schemas/v1.2.rst @@ -0,0 +1,5 @@ +API v1.2 (latest) += + +.. openapi:: ../../schemas/v1.2/patchwork.yaml + :examples: diff --git a/docs/api/schemas/generate_schema.py b/docs/api/schemas/generate_schema.py index 39f5cf06a9e9..d4645d1f53fe 100644 --- a/docs/api/schemas/generate_schema.py +++ b/docs/api/schemas/generate_schema.py @@ -5,8 +5,8 @@ import os import jinja2 ROOT_DIR = os.path.dirname(os.path.realpath(__file__)) -VERSIONS = [(1, 0), (1, 1), None] -LATEST_VERSION = (1, 1) +VERSIONS = [(1, 0), (1, 1), (1, 2), None] +LATEST_VERSION = (1, 2) def generate_schema(): diff --git a/docs/api/schemas/latest/patchwork.yaml b/docs/api/schemas/latest/patchwork.yaml index 724b05ebf1b3..394655de46e7 100644 --- a/docs/api/schemas/latest/patchwork.yaml +++ b/docs/api/schemas/latest/patchwork.yaml @@ -12,7 +12,7 @@ info: license: name: GPL v2 License url: https://www.gnu.org/licenses/gpl-2.0.html - version: '1.1' + version: '1.2' paths: /api/: get: @@ -1368,6 +1368,11 @@ components: readOnly: true minLength: 1 maxLength: 255 +list_archive_url: + title: List archive URL + type: string + readOnly: true + nullable: true date: title: Date type: string @@ -1418,6 +1423,11 @@ components: readOnly: true minLength: 1 maxLength: 255 +list_archive_url: + title: List archive URL + type: string + readOnly: true + nullable: true date: title: Date type: string @@ -1631,6 +1641,11 @@ components: readOnly: true minLength: 1 maxLength: 255 +list_archive_url: + title: List archive URL + type: string + readOnly: true + nullable: true date: title: Date type: string @@ -1863,6 +1878,21 @@ components: type: string readOnly: true maxLength: 64 +list_archive_url: + title: List archive URL + type: string + format: uri + maxLength: 2000 + nullable: true +list_archive_url_format: + title: List archive URL format + type: string + format: uri + maxLength: 2000 + nullable: t
[PATCH v2 4/7] api: Add list archive fields
Add the new list archive fields to the API. As this is a
backwards-compatible change, this requires only a minor version increment
to v1.2.
Signed-off-by: Andrew Donnellan
---
v1->v2:
- switch to using format string (Daniel)
---
patchwork/api/comment.py | 5 +++--
patchwork/api/cover.py| 6 --
patchwork/api/embedded.py | 14 +++---
patchwork/api/patch.py| 15 ---
patchwork/api/project.py | 7 +--
patchwork/urls.py | 4 ++--
6 files changed, 33 insertions(+), 18 deletions(-)
diff --git a/patchwork/api/comment.py b/patchwork/api/comment.py
index 57b37111c30b..290b9cd3f3ce 100644
--- a/patchwork/api/comment.py
+++ b/patchwork/api/comment.py
@@ -47,11 +47,12 @@ class CommentListSerializer(BaseHyperlinkedModelSerializer):
class Meta:
model = Comment
-fields = ('id', 'web_url', 'msgid', 'date', 'subject', 'submitter',
- 'content', 'headers')
+fields = ('id', 'web_url', 'msgid', 'list_archive_url', 'date',
+ 'subject', 'submitter', 'content', 'headers')
read_only_fields = fields
versioned_fields = {
'1.1': ('web_url', ),
+'1.2': ('list_archive_url',),
}
diff --git a/patchwork/api/cover.py b/patchwork/api/cover.py
index 7a663226ade8..caf9a386efa5 100644
--- a/patchwork/api/cover.py
+++ b/patchwork/api/cover.py
@@ -50,11 +50,13 @@ class
CoverLetterListSerializer(BaseHyperlinkedModelSerializer):
class Meta:
model = CoverLetter
-fields = ('id', 'url', 'web_url', 'project', 'msgid', 'date', 'name',
- 'submitter', 'mbox', 'series', 'comments')
+fields = ('id', 'url', 'web_url', 'project', 'msgid',
+ 'list_archive_url', 'date', 'name', 'submitter', 'mbox',
+ 'series', 'comments')
read_only_fields = fields
versioned_fields = {
'1.1': ('web_url', 'mbox', 'comments'),
+'1.2': ('list_archive_url',),
}
extra_kwargs = {
'url': {'view_name': 'api-cover-detail'},
diff --git a/patchwork/api/embedded.py b/patchwork/api/embedded.py
index 60fb9a4e9701..968cb7f91f10 100644
--- a/patchwork/api/embedded.py
+++ b/patchwork/api/embedded.py
@@ -108,10 +108,12 @@ class CoverLetterSerializer(SerializedRelatedField):
class Meta:
model = models.CoverLetter
-fields = ('id', 'url', 'web_url', 'msgid', 'date', 'name', 'mbox')
+fields = ('id', 'url', 'web_url', 'msgid', 'list_archive_url',
+ 'date', 'name', 'mbox')
read_only_fields = fields
versioned_fields = {
'1.1': ('web_url', 'mbox', ),
+'1.2': ('list_archive_url',),
}
extra_kwargs = {
'url': {'view_name': 'api-cover-detail'},
@@ -124,10 +126,12 @@ class PatchSerializer(SerializedRelatedField):
class Meta:
model = models.Patch
-fields = ('id', 'url', 'web_url', 'msgid', 'date', 'name', 'mbox')
+fields = ('id', 'url', 'web_url', 'msgid', 'list_archive_url',
+ 'date', 'name', 'mbox')
read_only_fields = fields
versioned_fields = {
'1.1': ('web_url', ),
+'1.2': ('list_archive_url',),
}
extra_kwargs = {
'url': {'view_name': 'api-patch-detail'},
@@ -158,11 +162,15 @@ class ProjectSerializer(SerializedRelatedField):
class Meta:
model = models.Project
fields = ('id', 'url', 'name', 'link_name', 'list_id',
- 'list_email', 'web_url', 'scm_url', 'webscm_url')
+ 'list_email', 'web_url', 'scm_url', 'webscm_url',
+ 'list_archive_url', 'list_archive_url_format')
read_only_fields = fields
extra_kwargs = {
'url': {'view_name': 'api-project-detail'},
}
+versioned_fields = {
+'1.2': ('list_archive_url', 'list_archive_url_format'),
+}
class SeriesSerializer(SerializedRelatedField):
diff --git a/patchwork/api/patch.py b/patchwork/api/patch.py
index 05f6cea84c45..c9360308a56a 100644
--- a/patchwork/api/patch.py
+++ b/patchwork/api/patch.py
@@ -121,15 +121,16 @@ class PatchListSerializer(BaseHyperlinkedModelSerializer):
class Meta:
model = Patch
-fields = ('id', 'url', 'web_url', 'project', 'msgid', 'date', 'name',
- 'commit_ref', 'pull_url', 'state', 'archived', 'hash',
- 'submitter', 'delegate', 'mbox', 'series', 'comments',
- 'check', 'checks', 'tags')
-read_only_fields = ('web_url', 'project', 'msgid', 'date', 'name',
-'hash', 'submitter', 'mbox', 'series', 'comments',
-'check', 'checks', 'tags')
+fie
[PATCH v2 7/7] fixtures: Update Patchwork list ID
The patchwork list uses patchwork.lists.ozlabs.org as its list ID nowadays.
Fix it in the example fixture.
Signed-off-by: Andrew Donnellan
---
patchwork/fixtures/default_projects.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/patchwork/fixtures/default_projects.xml
b/patchwork/fixtures/default_projects.xml
index 0895502dac71..a0877d986333 100644
--- a/patchwork/fixtures/default_projects.xml
+++ b/patchwork/fixtures/default_projects.xml
@@ -3,7 +3,7 @@
patchwork
Patchwork
-patchwork.ozlabs.org
+patchwork.lists.ozlabs.org
patchwork@lists.ozlabs.org
https://lists.ozlabs.org/pipermail/patchwork/
http://mid.mail-archive.com/{}
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH v2 6/7] releasenotes: Add release note for new list archive fields
Signed-off-by: Andrew Donnellan --- v1->v2: - switch to using format string (Daniel) --- .../list-archive-urls-604e69cd92c6b943.yaml| 18 ++ 1 file changed, 18 insertions(+) create mode 100644 releasenotes/notes/list-archive-urls-604e69cd92c6b943.yaml diff --git a/releasenotes/notes/list-archive-urls-604e69cd92c6b943.yaml b/releasenotes/notes/list-archive-urls-604e69cd92c6b943.yaml new file mode 100644 index ..b3cd119b59af --- /dev/null +++ b/releasenotes/notes/list-archive-urls-604e69cd92c6b943.yaml @@ -0,0 +1,18 @@ +--- +features: + - | +Add a field to Project to store a link to the project's mailing list +archive, and display that on the project info page. + - | +Add a field to Project to store a URL format for a Message-ID redirector +for the project's mailing list archive, and display a link to the email +thread for each patch. +api: + - | +The API version has been updated to v1.2. + - | +Projects now expose the ``list_archive_url`` and +``list_archive_url_format`` attributes. + - | +Patches, comments and cover letters now expose a ``list_archive_url`` +attribute. -- 2.20.1 ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH v2 1/7] models, templates: Add project list archive URL field
Add a field to link to a project's mailing list archive, and display it on
the project info page.
Add the new field to the patchwork project in the supplied example fixture.
Signed-off-by: Andrew Donnellan
---
v1->v2:
- add field to fixtures (Daniel)
---
patchwork/fixtures/default_projects.xml | 1 +
.../0034_project_list_archive_url.py | 20 +++
patchwork/models.py | 1 +
patchwork/templates/patchwork/project.html| 6 ++
4 files changed, 28 insertions(+)
create mode 100644 patchwork/migrations/0034_project_list_archive_url.py
diff --git a/patchwork/fixtures/default_projects.xml
b/patchwork/fixtures/default_projects.xml
index 30d346127234..39d269836394 100644
--- a/patchwork/fixtures/default_projects.xml
+++ b/patchwork/fixtures/default_projects.xml
@@ -5,5 +5,6 @@
Patchwork
patchwork.ozlabs.org
patchwork@lists.ozlabs.org
+https://lists.ozlabs.org/pipermail/patchwork/
diff --git a/patchwork/migrations/0034_project_list_archive_url.py
b/patchwork/migrations/0034_project_list_archive_url.py
new file mode 100644
index ..70d1b2bf8542
--- /dev/null
+++ b/patchwork/migrations/0034_project_list_archive_url.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.21 on 2019-07-01 12:30
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+dependencies = [
+('patchwork', '0033_remove_patch_series_model'),
+]
+
+operations = [
+migrations.AddField(
+model_name='project',
+name='list_archive_url',
+field=models.CharField(blank=True, max_length=2000),
+),
+]
diff --git a/patchwork/models.py b/patchwork/models.py
index a7eee4dbad9a..e43b062b6f89 100644
--- a/patchwork/models.py
+++ b/patchwork/models.py
@@ -77,6 +77,7 @@ class Project(models.Model):
web_url = models.CharField(max_length=2000, blank=True)
scm_url = models.CharField(max_length=2000, blank=True)
webscm_url = models.CharField(max_length=2000, blank=True)
+list_archive_url = models.CharField(max_length=2000, blank=True)
# configuration options
diff --git a/patchwork/templates/patchwork/project.html
b/patchwork/templates/patchwork/project.html
index 99e36ff79d6a..bd9d20e263d8 100644
--- a/patchwork/templates/patchwork/project.html
+++ b/patchwork/templates/patchwork/project.html
@@ -15,6 +15,12 @@
List address
{{project.listemail}}
+{% if project.list_archive_url %}
+
+ List archive
+ {{ project.list_archive_url
}}
+
+{% endif %}
Maintainer{{maintainers|length|pluralize}}
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH v2 3/7] templates: Add mailing list archive link to patch detail page
Add a link to the mailing list archive link to the patch detail page.
Suggested-by: Takashi Iwai
Signed-off-by: Andrew Donnellan
---
patchwork/templates/patchwork/submission.html | 4
1 file changed, 4 insertions(+)
diff --git a/patchwork/templates/patchwork/submission.html
b/patchwork/templates/patchwork/submission.html
index b1ae5429191d..9cebbbeb89aa 100644
--- a/patchwork/templates/patchwork/submission.html
+++ b/patchwork/templates/patchwork/submission.html
@@ -34,7 +34,11 @@ function toggle_div(link_id, headers_id)
Message ID
+ {% if submission.list_archive_url %}
+ {{ submission.msgid|msgid }} (mailing list archive)
+ {% else %}
{{ submission.msgid|msgid }}
+ {% endif %}
{% if submission.state %}
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH v2 2/7] models: Add list archive lookup
Add a list_archive_url_format field to Project, which will contain the
address of a Message-ID redirector, e.g. "https://lore.kernel.org/r/{};.
Add a list_archive_url property to Submission and Comment, to generate an
archive lookup URL based on the Message-ID.
We will use this to display links to mailing list archives.
Also add the new field to the default patchwork project fixture.
Suggested-by: Takashi Iwai
Signed-off-by: Andrew Donnellan
---
v1->v2:
- add to default fixture (Daniel)
- switch to using format string (Daniel)
---
patchwork/fixtures/default_projects.xml | 1 +
.../0035_project_list_archive_url_format.py | 20 +
patchwork/models.py | 22 +++
3 files changed, 43 insertions(+)
create mode 100644 patchwork/migrations/0035_project_list_archive_url_format.py
diff --git a/patchwork/fixtures/default_projects.xml
b/patchwork/fixtures/default_projects.xml
index 39d269836394..0895502dac71 100644
--- a/patchwork/fixtures/default_projects.xml
+++ b/patchwork/fixtures/default_projects.xml
@@ -6,5 +6,6 @@
patchwork.ozlabs.org
patchwork@lists.ozlabs.org
https://lists.ozlabs.org/pipermail/patchwork/
+http://mid.mail-archive.com/{}
diff --git a/patchwork/migrations/0035_project_list_archive_url_format.py
b/patchwork/migrations/0035_project_list_archive_url_format.py
new file mode 100644
index ..376d6edf2ba8
--- /dev/null
+++ b/patchwork/migrations/0035_project_list_archive_url_format.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.21 on 2019-07-01 12:57
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+dependencies = [
+('patchwork', '0034_project_list_archive_url'),
+]
+
+operations = [
+migrations.AddField(
+model_name='project',
+name='list_archive_url_format',
+field=models.CharField(blank=True, help_text=b"URL format for the
list archive's Message-ID redirector. {} will be replaced by the Message-ID.",
max_length=2000),
+),
+]
diff --git a/patchwork/models.py b/patchwork/models.py
index e43b062b6f89..4d23396033cf 100644
--- a/patchwork/models.py
+++ b/patchwork/models.py
@@ -78,6 +78,10 @@ class Project(models.Model):
scm_url = models.CharField(max_length=2000, blank=True)
webscm_url = models.CharField(max_length=2000, blank=True)
list_archive_url = models.CharField(max_length=2000, blank=True)
+list_archive_url_format = models.CharField(
+max_length=2000, blank=True,
+help_text="URL format for the list archive's Message-ID redirector. "
+"{} will be replaced by the Message-ID.")
# configuration options
@@ -358,6 +362,15 @@ class Submission(FilenameMixin, EmailMixin, models.Model):
name = models.CharField(max_length=255)
+@property
+def list_archive_url(self):
+if not self.project.list_archive_url_format:
+return None
+if not self.msgid:
+return None
+return self.project.list_archive_url_format.format(
+self.msgid.strip('<>'))
+
# patchwork metadata
def is_editable(self, user):
@@ -591,6 +604,15 @@ class Comment(EmailMixin, models.Model):
related_query_name='comment',
on_delete=models.CASCADE)
+@property
+def list_archive_url(self):
+if not self.submission.project.list_archive_url_format:
+return None
+if not self.msgid:
+return None
+return self.project.list_archive_url_format.format(
+self.msgid.strip('<>'))
+
def get_absolute_url(self):
return reverse('comment-redirect', kwargs={'comment_id': self.id})
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] models: Add commit_url_format to Project
On 22/8/19 11:55 am, Daniel Axtens wrote: It looks like you're going to do a v2 anyway to mesh with Andrew's changes - please could you pop in update to the fixtures that demonstrates/exercises this? I've had a look at the mark_safe bit. I don't love it - it allows someone with priv-esc to admin to XSS everyone who visits a patch page. Having said that I'm not entirely sure what the best way to handle it is. Andrew you did a few follow-up patches for our XSS adventures - do you have any thoughts? I think you probably want to wrap the patch.project.commit_url_format.format(commit=commit) in an escape. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 2/6] models: Add list archive lookup
On 22/8/19 11:38 am, Daniel Axtens wrote: Would it be better to do this with str.format(), as with mpe's commit_url_format patch? We should do it for consistency, I guess - the main reason I didn't do it is that I think the case of a Message-ID redirect URL where the ID isn't at the end is going to be rather rare. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 3/6] templates: Add mailing list archive link to patch detail page
On 21/8/19 8:24 pm, Daniel Axtens wrote:
Hi Andrew,
Add a link to the mailing list archive link to the patch detail page.
I don't know if the patchwork archive is on such an archive service,,
but if so please could you add it to the fixture?
We don't have a message-id redirector on lists.ozlabs.org, but I can use
the mail-archive.com service which does index the patchwork list.
Andrew
Regards,
Daniel
Suggested-by: Takashi Iwai
Signed-off-by: Andrew Donnellan
---
patchwork/templates/patchwork/submission.html | 4
1 file changed, 4 insertions(+)
diff --git a/patchwork/templates/patchwork/submission.html
b/patchwork/templates/patchwork/submission.html
index b1ae5429191d..9cebbbeb89aa 100644
--- a/patchwork/templates/patchwork/submission.html
+++ b/patchwork/templates/patchwork/submission.html
@@ -34,7 +34,11 @@ function toggle_div(link_id, headers_id)
Message ID
+ {% if submission.list_archive_url %}
+ {{ submission.msgid|msgid }} (mailing
list archive)
+ {% else %}
{{ submission.msgid|msgid }}
+ {% endif %}
{% if submission.state %}
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
--
Andrew Donnellan OzLabs, ADL Canberra
a...@linux.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 1/6] models, templates: Add project list archive URL field
On 21/8/19 8:21 pm, Daniel Axtens wrote:
Hi Andrew,
Add a field to link to a project's mailing list archive, and display it on
the project info page.
This is a good idea.
Would you mind adding the relevant field to
patchwork/fixtures/default_projects.xml please?
Done in v2
Regards,
Daniel
Signed-off-by: Andrew Donnellan
---
.../0034_project_list_archive_url.py | 20 +++
patchwork/models.py | 1 +
patchwork/templates/patchwork/project.html| 6 ++
3 files changed, 27 insertions(+)
create mode 100644 patchwork/migrations/0034_project_list_archive_url.py
diff --git a/patchwork/migrations/0034_project_list_archive_url.py
b/patchwork/migrations/0034_project_list_archive_url.py
new file mode 100644
index ..70d1b2bf8542
--- /dev/null
+++ b/patchwork/migrations/0034_project_list_archive_url.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.21 on 2019-07-01 12:30
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+dependencies = [
+('patchwork', '0033_remove_patch_series_model'),
+]
+
+operations = [
+migrations.AddField(
+model_name='project',
+name='list_archive_url',
+field=models.CharField(blank=True, max_length=2000),
+),
+]
diff --git a/patchwork/models.py b/patchwork/models.py
index a7eee4dbad9a..e43b062b6f89 100644
--- a/patchwork/models.py
+++ b/patchwork/models.py
@@ -77,6 +77,7 @@ class Project(models.Model):
web_url = models.CharField(max_length=2000, blank=True)
scm_url = models.CharField(max_length=2000, blank=True)
webscm_url = models.CharField(max_length=2000, blank=True)
+list_archive_url = models.CharField(max_length=2000, blank=True)
# configuration options
diff --git a/patchwork/templates/patchwork/project.html b/patchwork/templates/patchwork/project.html
index 99e36ff79d6a..bd9d20e263d8 100644
--- a/patchwork/templates/patchwork/project.html
+++ b/patchwork/templates/patchwork/project.html
@@ -15,6 +15,12 @@
List address
{{project.listemail}}
+{% if project.list_archive_url %}
+
+ List archive
+ {{ project.list_archive_url
}}
+
+{% endif %}
Maintainer{{maintainers|length|pluralize}}
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
--
Andrew Donnellan OzLabs, ADL Canberra
a...@linux.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Patch version changes in commit logs?
On 1/7/19 11:35 am, Andrew Donnellan wrote: On 1/7/19 2:01 am, Mauro Carvalho Chehab wrote: Em Sat, 29 Jun 2019 12:20:55 +0100 Mark Brown escreveu: On Sat, Jun 29, 2019 at 09:18:28AM +0200, Takashi Iwai wrote: BTW, can the URL be reached from patchwork? That'd be really handy. Even better, could patchwork add it to the mboxes you download from it like acks and so on? Currently you can get the message ID so it's easy to construct the link for things that are on LKML. I'm all for it, but the problem with patchwork is that the tool may be used on non-kernel development and/or the ML may not be in lore. Also, there are more than one patchwork instance. We use our own for Linux media (https://patchwork.linuxtv.org)[1]. On media, patchwork is used for both Kernel development and VDR development (an userspace tool) - with is a completely unrelated project (with different people behind it - and even a different mailing list). So, whatever change at patchwork should be done in a way that the ML URL could be customized, and the new field would be added only if the URL is not blank. [+ patchwork list] I'll have a go at adding a project/list-specific archive prefix and exposing that in the web interface. I've now submitted patches to add a mailing list archive link for each patch visible via the web interface and also via the REST API. Regarding adding it to downloaded mboxes, if we do that I'd like it to be a separate option. A single patch can also land in patchwork multiple times via various lists, so the URL will depend on which project you're looking at. Haven't done this bit yet. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH 5/6] docs: Add API v1.2
Add API v1.2, including the new fields for list archive URLs. Signed-off-by: Andrew Donnellan --- docs/api/rest/index.rst| 11 +- docs/api/rest/schemas/v1.1.rst |4 +- docs/api/rest/schemas/v1.2.rst |5 + docs/api/schemas/generate_schema.py|4 +- docs/api/schemas/latest/patchwork.yaml | 59 +- docs/api/schemas/patchwork.j2 | 71 + docs/api/schemas/v1.2/patchwork.yaml | 2371 7 files changed, 2518 insertions(+), 7 deletions(-) create mode 100644 docs/api/rest/schemas/v1.2.rst create mode 100644 docs/api/schemas/v1.2/patchwork.yaml diff --git a/docs/api/rest/index.rst b/docs/api/rest/index.rst index 1a094d732594..d1169e56e07a 100644 --- a/docs/api/rest/index.rst +++ b/docs/api/rest/index.rst @@ -33,6 +33,11 @@ If all you want is reference guides, skip straight to :ref:`rest-api-schemas`. The API version was bumped to v1.1 in Patchwork v2.1. The older v1.0 API is still supported. For more information, refer to :ref:`rest-api-versions`. +.. versionchanged:: 2.2 + + The API version was bumped to v1.2 in Patchwork v2.2. The older APIs are + still supported. For more information, refer to :ref:`rest-api-versions`. + Getting Started --- @@ -98,7 +103,7 @@ Versioning -- By default, all requests will receive the latest version of the API: currently -``1.1``: +``1.2``: .. code-block:: http @@ -109,7 +114,7 @@ changes breaking your application: .. code-block:: http -GET /api/1.1 HTTP/1.1 +GET /api/1.2 HTTP/1.1 Older API versions will be deprecated and removed over time. For more information, refer to :ref:`rest-api-versions`. @@ -263,6 +268,7 @@ Supported Versions 1.0, 2.0, ✓ 1.1, 2.1, ✓ + 1.2, 2.2, ✓ Further information about this and more can typically be found in :doc:`the release notes `. @@ -278,6 +284,7 @@ Auto-generated schema documentation is provided below. /api/rest/schemas/v1.0 /api/rest/schemas/v1.1 + /api/rest/schemas/v1.2 .. Links diff --git a/docs/api/rest/schemas/v1.1.rst b/docs/api/rest/schemas/v1.1.rst index e18f81322688..1189f31571a5 100644 --- a/docs/api/rest/schemas/v1.1.rst +++ b/docs/api/rest/schemas/v1.1.rst @@ -1,5 +1,5 @@ -API v1.1 (latest) -= +API v1.1 + .. openapi:: ../../schemas/v1.1/patchwork.yaml :examples: diff --git a/docs/api/rest/schemas/v1.2.rst b/docs/api/rest/schemas/v1.2.rst new file mode 100644 index ..8a96519b7356 --- /dev/null +++ b/docs/api/rest/schemas/v1.2.rst @@ -0,0 +1,5 @@ +API v1.2 (latest) += + +.. openapi:: ../../schemas/v1.2/patchwork.yaml + :examples: diff --git a/docs/api/schemas/generate_schema.py b/docs/api/schemas/generate_schema.py index 39f5cf06a9e9..d4645d1f53fe 100644 --- a/docs/api/schemas/generate_schema.py +++ b/docs/api/schemas/generate_schema.py @@ -5,8 +5,8 @@ import os import jinja2 ROOT_DIR = os.path.dirname(os.path.realpath(__file__)) -VERSIONS = [(1, 0), (1, 1), None] -LATEST_VERSION = (1, 1) +VERSIONS = [(1, 0), (1, 1), (1, 2), None] +LATEST_VERSION = (1, 2) def generate_schema(): diff --git a/docs/api/schemas/latest/patchwork.yaml b/docs/api/schemas/latest/patchwork.yaml index 724b05ebf1b3..92f352f3e926 100644 --- a/docs/api/schemas/latest/patchwork.yaml +++ b/docs/api/schemas/latest/patchwork.yaml @@ -12,7 +12,7 @@ info: license: name: GPL v2 License url: https://www.gnu.org/licenses/gpl-2.0.html - version: '1.1' + version: '1.2' paths: /api/: get: @@ -1368,6 +1368,11 @@ components: readOnly: true minLength: 1 maxLength: 255 +list_archive_url: + title: List archive URL + type: string + readOnly: true + nullable: true date: title: Date type: string @@ -1418,6 +1423,11 @@ components: readOnly: true minLength: 1 maxLength: 255 +list_archive_url: + title: List archive URL + type: string + readOnly: true + nullable: true date: title: Date type: string @@ -1631,6 +1641,11 @@ components: readOnly: true minLength: 1 maxLength: 255 +list_archive_url: + title: List archive URL + type: string + readOnly: true + nullable: true date: title: Date type: string @@ -1863,6 +1878,22 @@ components: type: string readOnly: true maxLength: 64 +list_archive_url: + title: List archive URL + type: string + format: uri + maxLength: 2000 + nullable: true +list_archive_lookup_prefix: + title: List archive lookup prefix + type: string + format: uri + maxLength: 2000 + nullable: true + description: > +URL pre
Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Patch version changes in commit logs?
On 1/7/19 2:01 am, Mauro Carvalho Chehab wrote: Em Sat, 29 Jun 2019 12:20:55 +0100 Mark Brown escreveu: On Sat, Jun 29, 2019 at 09:18:28AM +0200, Takashi Iwai wrote: BTW, can the URL be reached from patchwork? That'd be really handy. Even better, could patchwork add it to the mboxes you download from it like acks and so on? Currently you can get the message ID so it's easy to construct the link for things that are on LKML. I'm all for it, but the problem with patchwork is that the tool may be used on non-kernel development and/or the ML may not be in lore. Also, there are more than one patchwork instance. We use our own for Linux media (https://patchwork.linuxtv.org)[1]. On media, patchwork is used for both Kernel development and VDR development (an userspace tool) - with is a completely unrelated project (with different people behind it - and even a different mailing list). So, whatever change at patchwork should be done in a way that the ML URL could be customized, and the new field would be added only if the URL is not blank. [+ patchwork list] I'll have a go at adding a project/list-specific archive prefix and exposing that in the web interface. Regarding adding it to downloaded mboxes, if we do that I'd like it to be a separate option. A single patch can also land in patchwork multiple times via various lists, so the URL will depend on which project you're looking at. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: Patchwork Registration
Hi Stuart! On 5/6/19 9:47 pm, Stuart Foster wrote: Hi, I have have attempted to register with Patchwork but I have not received a confirmation e-mail. The login page recognises my user name and e-mail but blocks my access. How can this be rectified ? Which Patchwork instance is this? patchwork.ozlabs.org or somewhere else? Andrew -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [RFC PATCH] docker: Add support for using eatmydata in the database
On 3/5/19 5:33 pm, Russell Currey wrote: When running tox on a VM with presumably pretty busy spinning disks, using eatmydata with the database took running one configuration's test suite from (no exaggeration) 20 minutes down to 60 seconds. It makes a huge difference to test speed, so we should make it easily available for developers. The primary motivation here was to automatically test each patch in a timeframe that isn't insane. Open to ideas on how to organise this, whether we do it for MySQL too (which we probably should), whether the base directory should have these files in it, what to call the Dockerfile, etc. I think it's a good thing to have in the repo, though. Signed-off-by: Russell Currey Reviewed-by: Andrew Donnellan --- docker-compose-eatmydata.yml | 32 +++ tools/docker/Dockerfile.eatmydata | 9 + 2 files changed, 41 insertions(+) create mode 100644 docker-compose-eatmydata.yml create mode 100644 tools/docker/Dockerfile.eatmydata diff --git a/docker-compose-eatmydata.yml b/docker-compose-eatmydata.yml new file mode 100644 index 000..27d1604 --- /dev/null +++ b/docker-compose-eatmydata.yml @@ -0,0 +1,32 @@ +version: "3" +services: + db: +build: + context: . + dockerfile: ./tools/docker/Dockerfile.eatmydata +volumes: + - ./tools/docker/db/postdata:/var/lib/postgresql/data +environment: + - POSTGRES_PASSWORD=password + + web: +build: + context: . + dockerfile: ./tools/docker/Dockerfile + args: +- UID +depends_on: + - db +command: python3 manage.py runserver 0.0.0.0:8000 +volumes: + - .:/home/patchwork/patchwork/ +ports: + - "8000:8000" +environment: + - UID + - PW_TEST_DB_HOST=db + - PW_TEST_DB_PORT=5432 + - PW_TEST_DB_TYPE=postgres + - PW_TEST_DB_USER=postgres + - PW_TEST_DB_PASS=password + - PGPASSWORD=password diff --git a/tools/docker/Dockerfile.eatmydata b/tools/docker/Dockerfile.eatmydata new file mode 100644 index 000..693cbb3 --- /dev/null +++ b/tools/docker/Dockerfile.eatmydata @@ -0,0 +1,9 @@ +FROM postgres:9.6 + +RUN apt-get update \ + && apt-get install -y eatmydata \ + && apt-get autoremove -y \ + && rm -rf /var/lib/apt/lists/* + +ENTRYPOINT [ "/usr/bin/eatmydata", "/usr/local/bin/docker-entrypoint.sh" ] +CMD ["postgres"] -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH 3/6] templates: Add mailing list archive link to patch detail page
Add a link to the mailing list archive link to the patch detail page.
Suggested-by: Takashi Iwai
Signed-off-by: Andrew Donnellan
---
patchwork/templates/patchwork/submission.html | 4
1 file changed, 4 insertions(+)
diff --git a/patchwork/templates/patchwork/submission.html
b/patchwork/templates/patchwork/submission.html
index b1ae5429191d..9cebbbeb89aa 100644
--- a/patchwork/templates/patchwork/submission.html
+++ b/patchwork/templates/patchwork/submission.html
@@ -34,7 +34,11 @@ function toggle_div(link_id, headers_id)
Message ID
+ {% if submission.list_archive_url %}
+ {{ submission.msgid|msgid }} (mailing list archive)
+ {% else %}
{{ submission.msgid|msgid }}
+ {% endif %}
{% if submission.state %}
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH 2/6] models: Add list archive lookup
Add a list_archive_lookup_prefix field to Project, which will contain the
address of a Message-ID redirector, e.g. "https://lore.kernel.org/r/;.
Add a list_archive_url property to Submission and Comment, to generate an
archive lookup URL based on the Message-ID.
We will use this to display links to mailing list archives.
Suggested-by: Takashi Iwai
Signed-off-by: Andrew Donnellan
---
...0035_project_list_archive_lookup_prefix.py | 20
patchwork/models.py | 23 +++
2 files changed, 43 insertions(+)
create mode 100644
patchwork/migrations/0035_project_list_archive_lookup_prefix.py
diff --git a/patchwork/migrations/0035_project_list_archive_lookup_prefix.py
b/patchwork/migrations/0035_project_list_archive_lookup_prefix.py
new file mode 100644
index ..7d5f94a462a5
--- /dev/null
+++ b/patchwork/migrations/0035_project_list_archive_lookup_prefix.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.21 on 2019-07-01 12:57
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+dependencies = [
+('patchwork', '0034_project_list_archive_url'),
+]
+
+operations = [
+migrations.AddField(
+model_name='project',
+name='list_archive_lookup_prefix',
+field=models.CharField(blank=True, help_text=b"URL prefix for the
list archive's Message-ID redirector. To generate the list archive link for a
patch, the Message-ID is appended to the end of this prefix.", max_length=2000),
+),
+]
diff --git a/patchwork/models.py b/patchwork/models.py
index e43b062b6f89..04d87a459e3a 100644
--- a/patchwork/models.py
+++ b/patchwork/models.py
@@ -78,6 +78,11 @@ class Project(models.Model):
scm_url = models.CharField(max_length=2000, blank=True)
webscm_url = models.CharField(max_length=2000, blank=True)
list_archive_url = models.CharField(max_length=2000, blank=True)
+list_archive_lookup_prefix = models.CharField(
+max_length=2000, blank=True,
+help_text="URL prefix for the list archive's Message-ID redirector. "
+"To generate the list archive link for a patch, the Message-ID is "
+"appended to the end of this prefix.")
# configuration options
@@ -358,6 +363,15 @@ class Submission(FilenameMixin, EmailMixin, models.Model):
name = models.CharField(max_length=255)
+@property
+def list_archive_url(self):
+if not self.project.list_archive_lookup_prefix:
+return None
+if not self.msgid:
+return None
+return self.project.list_archive_lookup_prefix + \
+self.msgid.strip('<>')
+
# patchwork metadata
def is_editable(self, user):
@@ -591,6 +605,15 @@ class Comment(EmailMixin, models.Model):
related_query_name='comment',
on_delete=models.CASCADE)
+@property
+def list_archive_url(self):
+if not self.submission.project.list_archive_lookup_prefix:
+return None
+if not self.msgid:
+return None
+return self.project.list_archive_lookup_prefix + \
+self.msgid.strip('<>')
+
def get_absolute_url(self):
return reverse('comment-redirect', kwargs={'comment_id': self.id})
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH 6/6] releasenotes: Add release note for new list archive fields
Signed-off-by: Andrew Donnellan --- .../list-archive-urls-604e69cd92c6b943.yaml| 18 ++ 1 file changed, 18 insertions(+) create mode 100644 releasenotes/notes/list-archive-urls-604e69cd92c6b943.yaml diff --git a/releasenotes/notes/list-archive-urls-604e69cd92c6b943.yaml b/releasenotes/notes/list-archive-urls-604e69cd92c6b943.yaml new file mode 100644 index ..261f3726cdba --- /dev/null +++ b/releasenotes/notes/list-archive-urls-604e69cd92c6b943.yaml @@ -0,0 +1,18 @@ +--- +features: + - | +Add a field to Project to store a link to the project's mailing list +archive, and display that on the project info page. + - | +Add a field to Project to store a prefix for a Message-ID redirector +for the project's mailing list archive, and display a link to the email +thread for each patch. +api: + - | +The API version has been updated to v1.2. + - | +Projects now expose the ``list_archive_url`` and +``list_archive_lookup_prefix`` attributes. + - | +Patches, comments and cover letters now expose a ``list_archive_url`` +attribute. -- 2.20.1 ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH 4/6] api: Add list archive fields
Add the new list archive fields to the API. As this is a
backwards-compatible change, this requires only a minor version increment
to v1.2.
Signed-off-by: Andrew Donnellan
---
patchwork/api/comment.py | 5 +++--
patchwork/api/cover.py| 6 --
patchwork/api/embedded.py | 14 +++---
patchwork/api/patch.py| 15 ---
patchwork/api/project.py | 7 +--
patchwork/urls.py | 4 ++--
6 files changed, 33 insertions(+), 18 deletions(-)
diff --git a/patchwork/api/comment.py b/patchwork/api/comment.py
index 57b37111c30b..290b9cd3f3ce 100644
--- a/patchwork/api/comment.py
+++ b/patchwork/api/comment.py
@@ -47,11 +47,12 @@ class CommentListSerializer(BaseHyperlinkedModelSerializer):
class Meta:
model = Comment
-fields = ('id', 'web_url', 'msgid', 'date', 'subject', 'submitter',
- 'content', 'headers')
+fields = ('id', 'web_url', 'msgid', 'list_archive_url', 'date',
+ 'subject', 'submitter', 'content', 'headers')
read_only_fields = fields
versioned_fields = {
'1.1': ('web_url', ),
+'1.2': ('list_archive_url',),
}
diff --git a/patchwork/api/cover.py b/patchwork/api/cover.py
index 7a663226ade8..caf9a386efa5 100644
--- a/patchwork/api/cover.py
+++ b/patchwork/api/cover.py
@@ -50,11 +50,13 @@ class
CoverLetterListSerializer(BaseHyperlinkedModelSerializer):
class Meta:
model = CoverLetter
-fields = ('id', 'url', 'web_url', 'project', 'msgid', 'date', 'name',
- 'submitter', 'mbox', 'series', 'comments')
+fields = ('id', 'url', 'web_url', 'project', 'msgid',
+ 'list_archive_url', 'date', 'name', 'submitter', 'mbox',
+ 'series', 'comments')
read_only_fields = fields
versioned_fields = {
'1.1': ('web_url', 'mbox', 'comments'),
+'1.2': ('list_archive_url',),
}
extra_kwargs = {
'url': {'view_name': 'api-cover-detail'},
diff --git a/patchwork/api/embedded.py b/patchwork/api/embedded.py
index 60fb9a4e9701..3a5227b33df3 100644
--- a/patchwork/api/embedded.py
+++ b/patchwork/api/embedded.py
@@ -108,10 +108,12 @@ class CoverLetterSerializer(SerializedRelatedField):
class Meta:
model = models.CoverLetter
-fields = ('id', 'url', 'web_url', 'msgid', 'date', 'name', 'mbox')
+fields = ('id', 'url', 'web_url', 'msgid', 'list_archive_url',
+ 'date', 'name', 'mbox')
read_only_fields = fields
versioned_fields = {
'1.1': ('web_url', 'mbox', ),
+'1.2': ('list_archive_url',),
}
extra_kwargs = {
'url': {'view_name': 'api-cover-detail'},
@@ -124,10 +126,12 @@ class PatchSerializer(SerializedRelatedField):
class Meta:
model = models.Patch
-fields = ('id', 'url', 'web_url', 'msgid', 'date', 'name', 'mbox')
+fields = ('id', 'url', 'web_url', 'msgid', 'list_archive_url',
+ 'date', 'name', 'mbox')
read_only_fields = fields
versioned_fields = {
'1.1': ('web_url', ),
+'1.2': ('list_archive_url',),
}
extra_kwargs = {
'url': {'view_name': 'api-patch-detail'},
@@ -158,11 +162,15 @@ class ProjectSerializer(SerializedRelatedField):
class Meta:
model = models.Project
fields = ('id', 'url', 'name', 'link_name', 'list_id',
- 'list_email', 'web_url', 'scm_url', 'webscm_url')
+ 'list_email', 'web_url', 'scm_url', 'webscm_url',
+ 'list_archive_url', 'list_archive_lookup_prefix')
read_only_fields = fields
extra_kwargs = {
'url': {'view_name': 'api-project-detail'},
}
+versioned_fields = {
+'1.2': ('list_archive_url', 'list_archive_lookup_prefix'),
+}
class SeriesSerializer(SerializedRelatedField):
diff --git a/patchwork/api/patch.py b/patchwork/api/patch.py
index 05f6cea84c45..c9360308a56a 100644
--- a/patchwork/api/patch.py
+++ b/patchwork/api/patch.py
@@ -121,15 +121,16 @@ class PatchListSerializer(BaseHyperlinkedModelSerializer):
class Meta:
model = Patch
-fields = ('id', 'url', 'web_url', 'project', 'msgid', 'date', 'name',
- 'commit_ref', 'pull_url', 'state', 'archived', 'hash',
- 'submitter', 'delegate', 'mbox', 'series', 'comments',
- 'check', 'checks', 'tags')
-read_only_fields = ('web_url', 'project', 'msgid', 'date', 'name',
-'hash', 'submitter', 'mbox', 'series', 'comments',
-'check', 'checks', 'tags')
+fields = ('id', 'url', 'web_url', 'project', 'msgid
[PATCH 1/6] models, templates: Add project list archive URL field
Add a field to link to a project's mailing list archive, and display it on
the project info page.
Signed-off-by: Andrew Donnellan
---
.../0034_project_list_archive_url.py | 20 +++
patchwork/models.py | 1 +
patchwork/templates/patchwork/project.html| 6 ++
3 files changed, 27 insertions(+)
create mode 100644 patchwork/migrations/0034_project_list_archive_url.py
diff --git a/patchwork/migrations/0034_project_list_archive_url.py
b/patchwork/migrations/0034_project_list_archive_url.py
new file mode 100644
index ..70d1b2bf8542
--- /dev/null
+++ b/patchwork/migrations/0034_project_list_archive_url.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.21 on 2019-07-01 12:30
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+dependencies = [
+('patchwork', '0033_remove_patch_series_model'),
+]
+
+operations = [
+migrations.AddField(
+model_name='project',
+name='list_archive_url',
+field=models.CharField(blank=True, max_length=2000),
+),
+]
diff --git a/patchwork/models.py b/patchwork/models.py
index a7eee4dbad9a..e43b062b6f89 100644
--- a/patchwork/models.py
+++ b/patchwork/models.py
@@ -77,6 +77,7 @@ class Project(models.Model):
web_url = models.CharField(max_length=2000, blank=True)
scm_url = models.CharField(max_length=2000, blank=True)
webscm_url = models.CharField(max_length=2000, blank=True)
+list_archive_url = models.CharField(max_length=2000, blank=True)
# configuration options
diff --git a/patchwork/templates/patchwork/project.html
b/patchwork/templates/patchwork/project.html
index 99e36ff79d6a..bd9d20e263d8 100644
--- a/patchwork/templates/patchwork/project.html
+++ b/patchwork/templates/patchwork/project.html
@@ -15,6 +15,12 @@
List address
{{project.listemail}}
+{% if project.list_archive_url %}
+
+ List archive
+ {{ project.list_archive_url
}}
+
+{% endif %}
Maintainer{{maintainers|length|pluralize}}
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: OpenAPI schema
On 7/8/19 7:41 pm, Stephen Finucane wrote: On Wed, 2019-08-07 at 17:10 +1000, Andrew Donnellan wrote: Currently working on some patches that add some extra fields to the API. Updating the OpenAPI schema, especially with the jinja2 templating that we use for versioning, is difficult, especially for those of us with no background in the OpenAPI format. patchwork.j2 is >2300 lines long, which is more than 2x the LOC in the api directory as measured by sloccount. I can help with this. Just let me know what you'd like to add. If you wanted to do it yourself, I'd personally modify the generated v1.2 API using one of the many online schema editors (like https://editor.swagger.io/) and then feed the changes back into the template when you're done. I got there in the end! Does anyone even use the OpenAPI schema for anything? We use it for two things: API documentation and validation (i.e. tests). The API documentation was my initial goal but the validation part of things helped me uncover quite a few bugs in the API and (I imagine) will help prevent more in the future. It was also a chance to play around with OpenAPI, which is nice. I'd like to keep it around, even if it means I have to make any changes necessary. Yes, I realised it was being used for validation after the number of fixes I had to make to get the tests passing :) I note that DRF apparently has some native support for OpenAPI schema generation (https://www.django-rest-framework.org/api-guide/schemas/) - can we use that? Perhaps with a custom schema generator to handle API versioning. I checked this out (and submitted a few patches to correct some of the many bugs in earlier versions of it). Unfortunately, the schema it generates is far too lossy to be useful, and it does particularly poorly with things that are autogenerated or more freeform fields/filters. I also looked through a couple of Django extensions but they suffered the same issue. As nice as it would have been to auto- generate this stuff, it just wasn't an issue, sadly :( That's unfortunate. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
OpenAPI schema
Currently working on some patches that add some extra fields to the API. Updating the OpenAPI schema, especially with the jinja2 templating that we use for versioning, is difficult, especially for those of us with no background in the OpenAPI format. patchwork.j2 is >2300 lines long, which is more than 2x the LOC in the api directory as measured by sloccount. Does anyone even use the OpenAPI schema for anything? I note that DRF apparently has some native support for OpenAPI schema generation (https://www.django-rest-framework.org/api-guide/schemas/) - can we use that? Perhaps with a custom schema generator to handle API versioning. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] models: Add commit_url_format to Project
On 7/8/19 9:22 am, Andrew Donnellan wrote: On 6/8/19 10:20 pm, Michael Ellerman wrote: Add a new field to Project, commit_url_format, which specifies a format string that can be used to generate a link to a particular commit for a project. This is used in the display of a patch, to render the patch's commit as a clickable link back to the commit on the SCM website. Signed-off-by: Michael Ellerman Argh, I've actually got a series of my own pending to do exactly this, just had to tidy up the documentation before sending it :) I'll take a look at this and compare later today. I correct myself, I have patches to add mailing list archive links, which is slightly different! My series includes a minor bump to the API versioning, which per docs/development/releasing.rst is our policy when adding new fields. I'll tidy that up and send it and perhaps you can rebase your API changes on top of that? --- Passes tox tests. Not entirely sure about the schema changes, I just cribbed from the existing fields. I think the use of mark_safe() is correct, but would appreciate some review on that. --- docs/api/schemas/latest/patchwork.yaml | 11 ++ docs/api/schemas/patchwork.j2 | 11 ++ docs/api/schemas/v1.0/patchwork.yaml | 11 ++ docs/api/schemas/v1.1/patchwork.yaml | 11 ++ patchwork/api/embedded.py | 3 ++- patchwork/api/project.py | 4 ++-- .../0034_project_commit_url_format.py | 20 +++ patchwork/models.py | 9 + patchwork/templates/patchwork/submission.html | 2 +- patchwork/templatetags/patch.py | 12 +++ 10 files changed, 90 insertions(+), 4 deletions(-) create mode 100644 patchwork/migrations/0034_project_commit_url_format.py diff --git a/docs/api/schemas/latest/patchwork.yaml b/docs/api/schemas/latest/patchwork.yaml index 724b05e..c9e6c4f 100644 --- a/docs/api/schemas/latest/patchwork.yaml +++ b/docs/api/schemas/latest/patchwork.yaml @@ -1846,6 +1846,9 @@ openapi: '3.0.0' type: string format: uri maxLength: 2000 + commit_url_format: + title: Web SCM URL format for a particular commit + type: string maintainers: type: array items: @@ -2162,6 +2165,10 @@ openapi: '3.0.0' format: uri readOnly: true maxLength: 2000 + commit_url_format: + title: Web SCM URL format for a particular commit + type: string + readOnly: true SeriesEmbedded: type: object properties: @@ -2301,6 +2308,10 @@ openapi: '3.0.0' type: string format: uri readOnly: true + commit_url_format: + title: Web SCM URL format for a particular commit + type: string + readOnly: true ErrorUserUpdate: type: object properties: diff --git a/docs/api/schemas/patchwork.j2 b/docs/api/schemas/patchwork.j2 index 5e2f5e4..c0676f2 100644 --- a/docs/api/schemas/patchwork.j2 +++ b/docs/api/schemas/patchwork.j2 @@ -1861,6 +1861,9 @@ openapi: '3.0.0' type: string format: uri maxLength: 2000 + commit_url_format: + title: Web SCM URL format for a particular commit + type: string maintainers: type: array items: @@ -2185,6 +2188,10 @@ openapi: '3.0.0' format: uri readOnly: true maxLength: 2000 + commit_url_format: + title: Web SCM URL format for a particular commit + type: string + readOnly: true SeriesEmbedded: type: object properties: @@ -2326,6 +2333,10 @@ openapi: '3.0.0' type: string format: uri readOnly: true + commit_url_format: + title: Web SCM URL format for a particular commit + type: string + readOnly: true ErrorUserUpdate: type: object properties: diff --git a/docs/api/schemas/v1.0/patchwork.yaml b/docs/api/schemas/v1.0/patchwork.yaml index 02f3a15..370dffe 100644 --- a/docs/api/schemas/v1.0/patchwork.yaml +++ b/docs/api/schemas/v1.0/patchwork.yaml @@ -1811,6 +1811,9 @@ openapi: '3.0.0' type: string format: uri maxLength: 2000 + commit_url_format: + title: Web SCM URL format for a particular commit + type: string maintainers: type: array items: @@ -2101,6 +2104,10 @@ openapi: '3.0.0' format: uri readOnly: true maxLength: 2000 + commit_url_format: + title: Web SCM URL format for a particular commit + type: string + readOnly: true SeriesEmbedded: type: object properties: @@ -2235,6 +2242,10
Re: [PATCH] models: Add commit_url_format to Project
7 +45,7 @@ function toggle_div(link_id, headers_id)
{% if submission.commit_ref %}
Commit
- {{ submission.commit_ref }}
+ {{ submission|patch_commit_display }}
{% endif %}
{% if submission.delegate %}
diff --git a/patchwork/templatetags/patch.py b/patchwork/templatetags/patch.py
index 757f873..628725d 100644
--- a/patchwork/templatetags/patch.py
+++ b/patchwork/templatetags/patch.py
@@ -66,3 +66,15 @@ register = template.Library()
@stringfilter
def msgid(value):
return escape(value.strip('<>'))
+
+
+@register.filter(name='patch_commit_display')
+def patch_commit_display(patch):
+commit = escape(patch.commit_ref)
+if not patch.project.commit_url_format:
+return commit
+
+# NB. This is safe because we escaped commit above and the format string
+# can only be set by an administrator.
+return mark_safe('%s' % (
+patch.project.commit_url_format.format(commit=commit), commit))
--
Andrew Donnellan OzLabs, ADL Canberra
a...@linux.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
[PATCH] filters: Escape State names when generating selector HTML
States with names containing special characters are not correctly escaped when generating the select list. Use escape() to fix this. Signed-off-by: Andrew Donnellan --- patchwork/filters.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patchwork/filters.py b/patchwork/filters.py index e2d2f5958dd4..fb644f982136 100644 --- a/patchwork/filters.py +++ b/patchwork/filters.py @@ -262,7 +262,7 @@ class StateFilter(Filter): selected = ' selected="true"' out += '%s' % ( -state.id, selected, state.name) +state.id, selected, escape(state.name)) out += '' return mark_safe(out) -- 2.20.1 ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH] about: Display admin contact details
Display the list of admins on the about page. Add an ADMINS_HIDE option if
you don't want the details displayed publicly.
Closes: #282 ("Display contact details for patchwork instance admins")
Signed-off-by: Andrew Donnellan
---
docs/deployment/configuration.rst| 12
patchwork/settings/base.py | 8
patchwork/settings/production.example.py | 3 ++-
patchwork/templates/patchwork/about.html | 15 +++
patchwork/views/about.py | 1 +
5 files changed, 38 insertions(+), 1 deletion(-)
diff --git a/docs/deployment/configuration.rst
b/docs/deployment/configuration.rst
index ba9a2ebc8c97..833fd26268f5 100644
--- a/docs/deployment/configuration.rst
+++ b/docs/deployment/configuration.rst
@@ -114,3 +114,15 @@ access. This is useful if SSL protocol is terminated
upstream of the server
__ https://docs.djangoproject.com/en/1.8/ref/settings/
__ http://www.django-rest-framework.org/api-guide/settings/
+
+``ADMINS``
+~~
+
+Name and email address for Patchwork instance administrators.
+
+``ADMINS_HIDE``
+~~~
+
+If True, the details in ``ADMINS`` will be hidden from the About page.
+
+.. versionadded:: 2.2
diff --git a/patchwork/settings/base.py b/patchwork/settings/base.py
index 15644b405604..c76b5af22bfc 100644
--- a/patchwork/settings/base.py
+++ b/patchwork/settings/base.py
@@ -227,3 +227,11 @@ COMPAT_REDIR = True
# the scheme based on current access. This is useful if SSL protocol
# is terminated upstream of the server (e.g. at the load balancer)
FORCE_HTTPS_LINKS = False
+
+# Administrator contact details. Will be listed on the about page for users to
+# contact, unless ADMINS_HIDE is True.
+ADMINS = (
+# Add administrator contact details in the form:
+# ('Jeremy Kerr', 'j...@ozlabs.org'),
+)
+ADMINS_HIDE = False
diff --git a/patchwork/settings/production.example.py
b/patchwork/settings/production.example.py
index f58896fc0ac7..c6aa2f2850c0 100644
--- a/patchwork/settings/production.example.py
+++ b/patchwork/settings/production.example.py
@@ -42,7 +42,8 @@ SERVER_EMAIL = DEFAULT_FROM_EMAIL
NOTIFICATION_FROM_EMAIL = DEFAULT_FROM_EMAIL
ADMINS = (
-('Jeremy Kerr', 'j...@ozlabs.org'),
+# Add administrator contact details in the form:
+# ('Jeremy Kerr', 'j...@ozlabs.org'),
)
# Database
diff --git a/patchwork/templates/patchwork/about.html
b/patchwork/templates/patchwork/about.html
index f602c1351b56..b41d5fdc15ef 100644
--- a/patchwork/templates/patchwork/about.html
+++ b/patchwork/templates/patchwork/about.html
@@ -26,6 +26,21 @@
+ {% if admins %}
+
+
+ Administrators
+
+
+ {% for admin in admins %}
+
+ {{ admin.0 }} mailto:{{ admin.1 }}">{{ admin.1 }}
+
+ {% endfor %}
+
+
+ {% endif %}
+
API Status
diff --git a/patchwork/views/about.py b/patchwork/views/about.py
index 0061a3195298..91c3b74ebf8f 100644
--- a/patchwork/views/about.py
+++ b/patchwork/views/about.py
@@ -16,6 +16,7 @@ def about(request):
'rest': settings.ENABLE_REST_API,
'xmlrpc': settings.ENABLE_XMLRPC,
},
+'admins': () if settings.ADMINS_HIDE else settings.ADMINS,
}
return render(request, 'patchwork/about.html', context)
--
2.20.1
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH v2 0/5] Update REST API: Add 'project patches as mbox' field
On 2/7/19 10:26 am, Daniel Axtens wrote: So there are two possible complimenatry approaches I can think of: - gather the data from a download of the mailing list that patchwork injests. For LKML you can get this from https://www.kernel.org/lore.html, for example. You could then pass this through a local patchwork instance. (Let me know if you want my scripts for importing a public-archive git repo into patchwork.) - add a management command to export a project as an mbox and then coordinate with patchwork admins at the instance you're interested in to run the export at a time that suits them and provide you with a heavily compressed copy of the output. If we were to add an API for this kind of bulk mbox export, I think it would need to export a fixed number of emails at a time (100 or 250 or something like that). For patchwork setups where the raw mailing list data isn't easily retrievable, a management command to export the whole project as mbox could be extended fairly easily to "export the past N days of the project as mbox". Then just put that in your crontab and have it export a new archive every so often, which you compress and then serve up statically. Regards, Daniel Regards, Mete Updated REST API version to 1.2 New REST API url: api/1.2/ New internal url: project//list/mbox/ New project rest api field: patches_mbox Like patchwork.example.com/project//list/ the final project.mbox only includes patches. Careful naming has been made in case the api should support export of a whole project in the future as well. Changes since v1: - Add missing url for api version 1.2 Mete Polat (5): Add option to get all project patches in one mbox Add urls to get all project patches in one mbox Add api endpoint for project patches as mbox Update api documentation for v1.2 Add release notes: project patches as mbox docs/api/rest/index.rst | 50 +- docs/api/rest/schemas/v1.1.rst|4 +- docs/api/rest/schemas/v1.2.rst|5 + docs/api/schemas/generate_schema.py |4 +- docs/api/schemas/latest/patchwork.yaml|7 +- docs/api/schemas/patchwork.j2 |7 + docs/api/schemas/v1.2/patchwork.yaml | 2319 + patchwork/api/project.py | 11 +- patchwork/models.py | 25 +- patchwork/urls.py |6 +- patchwork/views/patch.py | 12 + patchwork/views/utils.py | 13 + ...project-patches-mbox-623f8c9d4cf6a952.yaml |6 + 13 files changed, 2428 insertions(+), 41 deletions(-) create mode 100644 docs/api/rest/schemas/v1.2.rst create mode 100644 docs/api/schemas/v1.2/patchwork.yaml create mode 100644 releasenotes/notes/project-patches-mbox-623f8c9d4cf6a952.yaml -- 2.22.0 ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 2/2] docs: Mention Postgres for Docker development install
On 1/5/19 4:27 pm, Russell Currey wrote: Might as well since it's there, and it gives some clue to anyone trying to use Docker on non-x86. I figured it was best to leave this out of the README since it's incredibly niche. Signed-off-by: Russell Currey Postgres is also what's used in production on ozlabs.org, idk about other instances, so we should definitely mention this actively. Reviewed-by: Andrew Donnellan --- docs/development/installation.rst | 4 1 file changed, 4 insertions(+) diff --git a/docs/development/installation.rst b/docs/development/installation.rst index 52dc2c3..0ab755f 100644 --- a/docs/development/installation.rst +++ b/docs/development/installation.rst @@ -37,6 +37,10 @@ configure Patchwork using Docker: $ docker-compose build + To use Postgres instead of MySQL, give the ``-f docker-compose-pg.yml`` + argument to ``docker-compose``. This is required on non-x86 architectures + as the MySQL Docker images do not have multiarch support. + #. Run ``docker-compose up``: .. code-block:: shell -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 1/2] docker: Use Ubuntu ports repositories on non-x86 architectures
On 1/5/19 4:27 pm, Russell Currey wrote: This should allow Patchwork to run "out of the box" in Docker on any architecture with a) an Ubuntu port and b) support in the Postgres multiarch Docker image, which includes at least arm64 and ppc64le. It's a little gross hacking the Dockerfile like this, but I'm not sure there's a more elegant way to do it. Unfortunately it doesn't seem like there's any way to do conditional COPY, and anything in RUN is plain /bin/sh, so that's why it looks like it does. Tested on ppc64le and on x86_64. Signed-off-by: Russell Currey Gross, but alas. One day ppc64le will graduate to being mirrored like a real architecture... Reviewed-by: Andrew Donnellan --- tools/docker/Dockerfile| 14 -- tools/docker/trusty-ports.list | 3 +++ tools/docker/xenial-ports.list | 3 +++ 3 files changed, 18 insertions(+), 2 deletions(-) create mode 100644 tools/docker/trusty-ports.list create mode 100644 tools/docker/xenial-ports.list diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index eef40e4..76bb6b2 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -15,11 +15,21 @@ ENV DJANGO_SETTINGS_MODULE patchwork.settings.dev ENV DEBIAN_FRONTEND noninteractive ENV PYTHONUNBUFFERED 1 + # System # trusty and findutils is for python3.4; xenial is for python3.5 # TODO(stephenfin): Are curl, unzip required? -COPY tools/docker/trusty.list /etc/apt/sources.list.d/trusty.list -COPY tools/docker/xenial.list /etc/apt/sources.list.d/xenial.list +COPY tools/docker/*.list /etc/apt/sources.list.d/ + +RUN cd /etc/apt/sources.list.d; \ +echo $(uname -m) > /tmp/arch; \ +if [ $(cat /tmp/arch) != 'x86_64' ] && grep -q -v "i.86" /tmp/arch; then \ +mv trusty-ports.list trusty.list; \ +mv xenial-ports.list xenial.list; \ +else \ +rm *-ports.list; \ +fi + RUN apt-get update -qq && \ apt-get install -y --no-install-recommends --allow-downgrades \ python-dev python-pip python-setuptools python-wheel \ diff --git a/tools/docker/trusty-ports.list b/tools/docker/trusty-ports.list new file mode 100644 index 000..ebcf4fa --- /dev/null +++ b/tools/docker/trusty-ports.list @@ -0,0 +1,3 @@ +deb http://ports.ubuntu.com/ubuntu-ports/ trusty main +deb http://ports.ubuntu.com/ubuntu-ports/ trusty-updates main +deb http://ports.ubuntu.com/ubuntu-ports/ trusty-security main diff --git a/tools/docker/xenial-ports.list b/tools/docker/xenial-ports.list new file mode 100644 index 000..d84641f --- /dev/null +++ b/tools/docker/xenial-ports.list @@ -0,0 +1,3 @@ +deb http://ports.ubuntu.com/ubuntu-ports/ xenial main +deb http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main +deb http://ports.ubuntu.com/ubuntu-ports/ xenial-security main -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] docker: Install libpq-dev to fix psycopg2-binary build
On 1/5/19 2:35 pm, Russell Currey wrote: psycopg2-binary fails if pg_config isn't installed, which is provided by libpq-dev. This seems strange to me since psycopg2-binary suggests that you use psycopg2-binary instead (of itself) if you don't want to build psycopg2 so you wouldn't need pg_config, which is very confusing. It's possible that psycopg2-binary only needs to compile itself on non-x86 platforms, since I hit this on ppc64le. Anyway, it works when this is added. I don't think there is a binary psycopg2-binary build for anything other than i686/x86_64? Reviewed-by: Andrew Donnellan Signed-off-by: Russell Currey --- tools/docker/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index b9ecdb5..eef40e4 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -26,7 +26,7 @@ RUN apt-get update -qq && \ python3.5-dev python3-pip python3-setuptools python3-wheel \ python3.4-dev findutils=4.4.2-7 python3.6-dev \ libmysqlclient-dev mysql-client curl unzip build-essential \ -git postgresql-client tzdata +git postgresql-client tzdata libpq-dev # User RUN useradd --uid=$UID --create-home patchwork -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] README: add .env file to installation instructions
Reviewed-by: Andrew Donnellan On 1/5/19 1:16 pm, Russell Currey wrote: Creating the .env file is mentioned in the installation documentation but not in the README, so following only the steps mentioned there will fail. Add this and add a `cd patchwork` in there for good measure so you could straight up copy paste the steps. Signed-off-by: Russell Currey --- README.rst | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/README.rst b/README.rst index 38c1847..b45c3e6 100644 --- a/README.rst +++ b/README.rst @@ -61,11 +61,16 @@ environment. To install Patchwork: $ git clone https://github.com/getpatchwork/patchwork.git -3. Build the images. This will download over 200MB from the internet:: +3. Create a ``.env`` file in the root directory of the project and store your + ``UID`` attribute there:: + + $ cd patchwork && echo "UID=$UID" > .env + +4. Build the images. This will download over 200MB from the internet:: $ docker-compose build -4. Run `docker-compose up`:: +5. Run `docker-compose up`:: $ docker-compose up -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] parser: recognise git commit consisting only of empty new file
On 28/2/19 3:29 pm, Daniel Axtens wrote:
Commits with only an empty new file are liable to be missed.
The parser state machine doesn't recognise the headers "new
file mode" and "index": teach it about them.
Add a test to demonstrate.
It's a little bit academic as you don't usually send patches like
that but sometimes you do, especially if you're a snowpatch dev :)
Closes: #256
Reported-by: Andrew Donnellan
Signed-off-by: Daniel Axtens
LGTM, thanks for fixing this
Reviewed-by: Andrew Donnellan
---
Apologies for the long radio silence. I'm now in a job where Patchwork
is slightly adjacent to my role, so I have a bit more flexibility to work
on it. The usual lack of promises apply.
---
patchwork/parser.py | 10 +++---
.../tests/mail/0021-git-empty-new-file.mbox | 32 +++
patchwork/tests/test_parser.py| 5 +++
3 files changed, 43 insertions(+), 4 deletions(-)
create mode 100644 patchwork/tests/mail/0021-git-empty-new-file.mbox
diff --git a/patchwork/parser.py b/patchwork/parser.py
index 946b66851d7c..712780a498c4 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -742,7 +742,7 @@ def parse_patch(content):
# 3: patch header line 2 (+++)
# 4: patch hunk header line (@@ line)
# 5: patch hunk content
-# 6: patch meta header (rename from/rename to)
+# 6: patch meta header (rename from/rename to/new file/index)
#
# valid transitions:
# 0 -> 1 (diff, Index:)
@@ -752,7 +752,7 @@ def parse_patch(content):
# 3 -> 4 (@@ line)
# 4 -> 5 (patch content)
# 5 -> 1 (run out of lines from @@-specifed count)
-# 1 -> 6 (rename from / rename to)
+# 1 -> 6 (rename from / rename to / new file / index)
# 6 -> 2 (---)
# 6 -> 1 (other text)
#
@@ -782,7 +782,8 @@ def parse_patch(content):
if line.startswith('--- '):
state = 2
-if line.startswith(('rename from ', 'rename to ')):
+if line.startswith(('rename from ', 'rename to ',
+'new file mode ', 'index ')):
state = 6
elif state == 2:
if line.startswith('+++ '):
@@ -843,7 +844,8 @@ def parse_patch(content):
else:
state = 5
elif state == 6:
-if line.startswith(('rename to ', 'rename from ')):
+if line.startswith(('rename to ', 'rename from ',
+'new file mode ', 'index ')):
patchbuf += buf + line
buf = ''
elif line.startswith('--- '):
diff --git a/patchwork/tests/mail/0021-git-empty-new-file.mbox
b/patchwork/tests/mail/0021-git-empty-new-file.mbox
new file mode 100644
index ..c3be48e6eb39
--- /dev/null
+++ b/patchwork/tests/mail/0021-git-empty-new-file.mbox
@@ -0,0 +1,32 @@
+From andrew.donnel...@au1.ibm.com Thu Feb 28 00:37:42 2019
+Delivered-To: d...@axtens.net
+Received: by 2002:a4a:2812:0:0:0:0:0 with SMTP id h18csp2242ooa;
+Wed, 27 Feb 2019 16:37:59 -0800 (PST)
+From: Andrew Donnellan
+Subject: [snowpatch] [PATCH 1/3] Test commit; please ignore
+To: Daniel Axtens
+Date: Thu, 28 Feb 2019 11:37:42 +1100
+User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
+ Thunderbird/60.5.1
+MIME-Version: 1.0
+Content-Language: en-AU
+
+
+Doing some snowpatching.
+---
+ banana | 0
+ 1 file changed, 0 insertions(+), 0 deletions(-)
+ create mode 100644 banana
+
+diff --git a/banana b/banana
+new file mode 100644
+index ..e69de29bb2d1
+--
+2.11.0
+
+___
+snowpatch mailing list
+snowpa...@lists.ozlabs.org
+https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ozlabs.org_listinfo_snowpatch=DwIDAg=jf_iaSHvJObTbx-siA1ZOg=-pHOU8dm1U-U1crivyxKr_-xvZrIBB8YUqvA3el0Ee0=VWJFv-r8DBhr5gE3qkDHngtXenfi5deVGQpQcAyKocY=pn_W97wan_5Uwi-2wli6N87gaeoNFy7pchZS_cPHtOY=
+
+
diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py
index 664edd5bab44..ddbcf5b15a19 100644
--- a/patchwork/tests/test_parser.py
+++ b/patchwork/tests/test_parser.py
@@ -583,6 +583,11 @@ class PatchParseTest(PatchTest):
self.assertEqual(diff.count("\nrename to "), 2)
self.assertEqual(diff.count('\n-a\n+b'), 1)
+def test_git_new_empty_file(self):
+diff, message = self._find_content('0021-git-empty-new-file.mbox')
+self.assertTrue(diff is not None)
+self.assertTrue(message is not None)
+
def test_cvs_format(self):
diff, message = self._find_content('0007-cvs-format-diff.mbox')
self.assertTrue(diff.startswith('Index'))
--
Andrew Donnellan OzLabs, ADL Canberra
andrew.donnel...@au1.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: HTTP not redirecting to HTTPS on patchwork.ozlabs.org
On 11/09/18 19:05, Philipp Richter wrote: Hello, I noticed that accessing patchwork.ozlabs.org did not automatically redirect to HTTPS. The validation link sent in the e-mail also doesn't have https:// prepended to it. This is a bit risky while registering or logging in. This was first apparent from https://buildroot.org/contribute.html which linked to the http unsecured patchwork page, they already committed a forced https:// on the link. I would strongly suggest forcing a 301 redirect to the https version of the site in the webserver configuration. jk and sfr are looking into it. Thanks for pointing this out :) -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH] docs: Fix documentation of REST_RESULTS_PER_PAGE setting
In 8fe11180a1a5 ("REST: Add new setting for maximum API page size") I
accidentally deleted the versionadded information for
REST_RESULTS_PER_PAGE. Restore it.
Fixes: 8fe11180a1a5 ("REST: Add new setting for maximum API page size")
Signed-off-by: Andrew Donnellan
---
docs/deployment/configuration.rst | 2 ++
1 file changed, 2 insertions(+)
diff --git a/docs/deployment/configuration.rst
b/docs/deployment/configuration.rst
index e599522a412b..0601276a3cbc 100644
--- a/docs/deployment/configuration.rst
+++ b/docs/deployment/configuration.rst
@@ -88,6 +88,8 @@ Enable the :doc:`REST API <../api/rest>`.
The number of items to include in REST API responses by default. This can be
overridden by the ``per_page`` parameter for some endpoints.
+.. versionadded:: 2.0
+
``MAX_REST_RESULTS_PER_PAGE``
~
--
2.11.0
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: ozlabs.org down?
On 23/08/18 02:12, Jeff Kirsher wrote: It appears the entire site is down, including patchworks. Is this expected? And for how long? + sfr It appears up for me right now. We've had a few dropouts over the last few weeks as a result of the upstream ISP having connectivity issues - it's possible that we've just hit that again. -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 00/11] Add labels support
On 09/08/18 18:54, Daniel Axtens wrote: This series starts work on the latter of these by addressing yet another issues, #22 [3]. Full details of the feature are provided inline but tl;dr labels are arbitrary bits of metadata that can be used to represent some of the more orthogonal states like "RFC" or "Under Review" along with other maintainer-provided labels. Once we have support for this, we can build upon it to migrate some of the 'states' to labels and the 'state' field itself to a boolean field. This is all in the future though. So I haven't read through the patches in great detail, but I want to just query the idea that RFC is orthogonal. I understand a bunch of maintainers have a general policy of not merging RFC patches, so if something is posted as RFC they just mark it as RFC on Patchwork and then don't ever look at it again. + mpe: I think we touched on this issue of the orthogonality of the RFC classification when we were chatting about snowpatch things the other day? -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [RFC PATCH] REST: Add new setting for maximum API page size
On 07/08/18 17:20, Daniel Axtens wrote: I wonder if we could let authenticated users do slower queries. Maybe something for later. Let's split the difference at 250 then, I'd be happy to merge that. If you want to take this patch and s/500/250/g during merge then I'd be happy with that. -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [RFC PATCH] REST: Add new setting for maximum API page size
On 26/07/18 23:24, Daniel Axtens wrote:
Andrew Donnellan writes:
On 24/07/18 15:10, Andrew Donnellan wrote:
In 41790caf59ad ("REST: Limit max page size") we limited the maximum page
size to the default page size in the settings.
This turns out to be rather restrictive, as we usually want to keep the
default page size low, but an administrator may want to allow API clients
to fetch more than that per request.
Add a new setting, MAX_REST_RESULTS_PER_PAGE, to set the maximum page size.
Closes: #202 ("Separate max API page size and default API page size into different
settings")
Suggested-by: Stewart Smith
Suggested-by: Joel Stanley
Signed-off-by: Andrew Donnellan
FWIW we now have this applied on patchwork.ozlabs.org and it appears to
be working. Would like some more input as to what an appropriate default
limit is.
My completely fact-free feeling/opinion is that if it takes more than
~1sec to run on Ozlabs, it's probably too high. Apart from that, I'm not
fussed.
OzLabs.org is not a fast machine. 1 second round trip time == 100 per
page. 500 per page == ~2.3 seconds round trip.
A single 500 item load is still under half the time of doing 5 * 100
item loads...
--
Andrew Donnellan OzLabs, ADL Canberra
andrew.donnel...@au1.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [RFC PATCH] REST: Add new setting for maximum API page size
On 24/07/18 15:10, Andrew Donnellan wrote:
In 41790caf59ad ("REST: Limit max page size") we limited the maximum page
size to the default page size in the settings.
This turns out to be rather restrictive, as we usually want to keep the
default page size low, but an administrator may want to allow API clients
to fetch more than that per request.
Add a new setting, MAX_REST_RESULTS_PER_PAGE, to set the maximum page size.
Closes: #202 ("Separate max API page size and default API page size into different
settings")
Suggested-by: Stewart Smith
Suggested-by: Joel Stanley
Signed-off-by: Andrew Donnellan
FWIW we now have this applied on patchwork.ozlabs.org and it appears to
be working. Would like some more input as to what an appropriate default
limit is.
--
Andrew Donnellan OzLabs, ADL Canberra
andrew.donnel...@au1.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [RFC PATCH] REST: Add new setting for maximum API page size
On 24/07/18 15:10, Andrew Donnellan wrote: diff --git a/docs/deployment/configuration.rst b/docs/deployment/configuration.rst index 347485636d47..e599522a412b 100644 --- a/docs/deployment/configuration.rst +++ b/docs/deployment/configuration.rst @@ -88,7 +88,13 @@ Enable the :doc:`REST API <../api/rest>`. The number of items to include in REST API responses by default. This can be overridden by the ``per_page`` parameter for some endpoints. -.. versionadded:: 2.0 This obviously shouldn't have been there, will fix when I send non RFC +``MAX_REST_RESULTS_PER_PAGE`` +~ + +The maximum number of items that can be requested in a REST API request using +the ``per_page`` parameter. + +.. versionadded:: 2.2 -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[RFC PATCH] REST: Add new setting for maximum API page size
In 41790caf59ad ("REST: Limit max page size") we limited the maximum page
size to the default page size in the settings.
This turns out to be rather restrictive, as we usually want to keep the
default page size low, but an administrator may want to allow API clients
to fetch more than that per request.
Add a new setting, MAX_REST_RESULTS_PER_PAGE, to set the maximum page size.
Closes: #202 ("Separate max API page size and default API page size into
different settings")
Suggested-by: Stewart Smith
Suggested-by: Joel Stanley
Signed-off-by: Andrew Donnellan
---
This is completely untested but should work, sending this as an RFC because
I have no idea what the default should be, thoughts?
---
docs/deployment/configuration.rst | 8 +++-
patchwork/api/base.py | 3 ++-
patchwork/settings/base.py| 1 +
3 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/docs/deployment/configuration.rst
b/docs/deployment/configuration.rst
index 347485636d47..e599522a412b 100644
--- a/docs/deployment/configuration.rst
+++ b/docs/deployment/configuration.rst
@@ -88,7 +88,13 @@ Enable the :doc:`REST API <../api/rest>`.
The number of items to include in REST API responses by default. This can be
overridden by the ``per_page`` parameter for some endpoints.
-.. versionadded:: 2.0
+``MAX_REST_RESULTS_PER_PAGE``
+~
+
+The maximum number of items that can be requested in a REST API request using
+the ``per_page`` parameter.
+
+.. versionadded:: 2.2
``COMPAT_REDIR``
diff --git a/patchwork/api/base.py b/patchwork/api/base.py
index 8c38d5a1d5f4..bf452f78b390 100644
--- a/patchwork/api/base.py
+++ b/patchwork/api/base.py
@@ -36,7 +36,8 @@ class LinkHeaderPagination(PageNumberPagination):
https://tools.ietf.org/html/rfc5988#section-5
https://developer.github.com/guides/traversing-with-pagination
"""
-page_size = max_page_size = settings.REST_RESULTS_PER_PAGE
+page_size = settings.REST_RESULTS_PER_PAGE
+max_page_size = settings.MAX_REST_RESULTS_PER_PAGE
page_size_query_param = 'per_page'
def get_paginated_response(self, data):
diff --git a/patchwork/settings/base.py b/patchwork/settings/base.py
index 4b0d5513895a..7dc20041ec42 100644
--- a/patchwork/settings/base.py
+++ b/patchwork/settings/base.py
@@ -220,6 +220,7 @@ ENABLE_XMLRPC = False
ENABLE_REST_API = True
REST_RESULTS_PER_PAGE = 30
+MAX_REST_RESULTS_PER_PAGE = 500
# Set to True to enable redirections or URLs from previous versions
# of patchwork
--
2.11.0
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: title tag in a series list
On 23/07/18 22:28, Ali Alnubani wrote: BTW, Is it ok that the diff of my previous email was saved by patchwork (http://patchwork.ozlabs.org/patch/947699/) without it having git-send-email as the x-mailer? I see that there are no restrictions in the code, should there be? Yep, that's deliberate. There's enough people who for some reason send their patches without using git-send-email. Also not all projects that are monitored in patchwork are necessarily git projects, so patchwork will look for anything that matches a diff. -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] [RFC] tools: drop vagrant
On 24/02/18 12:22, Daniel Axtens wrote: It served us well, but it's now outdated (Trusty, Python 3.4, etc) There is no indication that anyone uses it or keeps it up to date. Signed-off-by: Daniel Axtens <d...@axtens.net> Vagrant is dead, long live Docker! Reviewed-by: Andrew Donnellan <andrew.donnel...@au1.ibm.com> -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 7/9] parser: avoid an unnecessary UPDATE of Person
On 23/02/18 00:46, Daniel Axtens wrote: Wouldn't that mean a new null name would overwrite a previously saved name? I... somehow didn't think about that case. Yes. Of course. Andrew Regards, Daniel --- patchwork/parser.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patchwork/parser.py b/patchwork/parser.py index 1f3b3dd8901f..3d40b74375e0 100644 --- a/patchwork/parser.py +++ b/patchwork/parser.py @@ -349,7 +349,7 @@ def get_or_create_author(mail): # we lost the race to create the person person = Person.objects.get(email__iexact=email) -if name: # use the latest provided name +if name and name != person.name: # use the latest provided name person.name = name person.save() -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 9/9] parser: don't fail on multiple SeriesReferences
On 22/02/18 01:17, Daniel Axtens wrote:
Parallel parsing would occasonally fail with:
patchwork.models.MultipleObjectsReturned: get() returned more than one
SeriesReference -- it returned 2!
I think these are happening if you have different processes parsing
e.g. 1/3 and 2/3 simultaneously: both will have a reference to 1/3,
in the case of 1 it will be the msgid, in the case of 2 it will be
in References. So when we come to parse 3/3, .get() finds 2 and
throws the exception.
This does not fix the creation of multiple series references; it
just causes them to be ignored. We still have serious race conditions
with series creation, but I don't yet have clear answers for them.
With this patch, they will at least not stop patches from being
processed - they'll just lead to wonky series, which we already have.
Signed-off-by: Daniel Axtens <d...@axtens.net>
Label that open bug with a FIXME for searchability? Logging it is a
great idea.
Reviewed-by: Andrew Donnellan <andrew.donnel...@au1.ibm.com>
---
patchwork/parser.py | 23 +--
1 file changed, 21 insertions(+), 2 deletions(-)
diff --git a/patchwork/parser.py b/patchwork/parser.py
index 0e53e6b9a3af..4c9e636336d9 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -240,6 +240,13 @@ def _find_series_by_references(project, mail):
msgid=ref[:255], series__project=project).series
except SeriesReference.DoesNotExist:
continue
+except SeriesReference.MultipleObjectsReturned:
+# Open bug: this can happen when we're processing messages
+# in parallel. Pick the first and log.
+logger.error("Multiple SeriesReferences for %s in project %s!" %
+ (ref[:255], project.name))
+return SeriesReference.objects.filter(
+msgid=ref[:255], series__project=project).first().series
def _find_series_by_markers(project, mail, author):
@@ -1037,6 +1044,9 @@ def parse_mail(mail, list_id=None):
series__project=project)
except SeriesReference.DoesNotExist:
SeriesReference.objects.create(series=series, msgid=ref)
+except SeriesReference.MultipleObjectsReturned:
+logger.error("Multiple SeriesReferences for %s"
+ " in project %s!" % (ref, project.name))
# add to a series if we have found one, and we have a numbered
# patch. Don't add unnumbered patches (for example diffs sent
@@ -1075,6 +1085,11 @@ def parse_mail(mail, list_id=None):
msgid=msgid, series__project=project).series
except SeriesReference.DoesNotExist:
series = None
+except SeriesReference.MultipleObjectsReturned:
+logger.error("Multiple SeriesReferences for %s"
+ " in project %s!" % (msgid, project.name))
+series = SeriesReference.objects.filter(
+msgid=msgid, series__project=project).first().series
if not series:
series = Series(project=project,
@@ -1087,8 +1102,12 @@ def parse_mail(mail, list_id=None):
# we don't save the in-reply-to or references fields
# for a cover letter, as they can't refer to the same
# series
-SeriesReference.objects.get_or_create(series=series,
- msgid=msgid)
+try:
+SeriesReference.objects.get_or_create(series=series,
+ msgid=msgid)
+except SeriesReference.MultipleObjectsReturned:
+logger.error("Multiple SeriesReferences for %s"
+ " in project %s!" % (msgid, project.name))
cover_letter = CoverLetter(
msgid=msgid,
--
Andrew Donnellan OzLabs, ADL Canberra
andrew.donnel...@au1.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 8/9] parser: use Patch.objects.create instead of save()
On 22/02/18 01:17, Daniel Axtens wrote:
Attempts to do parallel parsing with MySQL threw the following errors:
_mysql_exceptions.OperationalError: (1213, 'Deadlock found when trying to get
lock; try restarting transaction')
Looking at the code, it was thrown when we created a patch like this:
patch = Patch(...)
patch.save()
The SQL statements that were being generated were weird:
UPDATE "patchwork_patch" SET ...
INSERT INTO "patchwork_patch" (...) VALUES (...)
As far as I can tell, the update could never work, because it was
trying to update a patch that didn't exist yet. My hypothesis is
that Django somehow didn't quite 'get' that because of the backend
complexity of the Patch model, so it tried to do an update, failed,
and then tried an insert.
Backend complexity... subclassing bug or something? Hmm.
Change the code to use Patch.objects.create, which makes the UPDATEs
and the weird MySQL errors go away.
Also move it up a bit earlier in the process so that if things go wrong
later at least we've committed the patch to the db.
Signed-off-by: Daniel Axtens <d...@axtens.net>
In any case, objects.create() is stylistically nicer imho.
I definitely think we should commit the patch to the database before
Series...
Reviewed-by: Andrew Donnellan <andrew.donnel...@au1.ibm.com>
---
patchwork/parser.py | 29 ++---
1 file changed, 14 insertions(+), 15 deletions(-)
diff --git a/patchwork/parser.py b/patchwork/parser.py
index 3d40b74375e0..0e53e6b9a3af 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -984,6 +984,20 @@ def parse_mail(mail, list_id=None):
filenames = find_filenames(diff)
delegate = find_delegate_by_filename(project, filenames)
+patch = Patch.objects.create(
+msgid=msgid,
+project=project,
+name=name[:255],
+date=date,
+headers=headers,
+submitter=author,
+content=message,
+diff=diff,
+pull_url=pull_url,
+delegate=delegate,
+state=find_state(mail))
+logger.debug('Patch saved')
+
# if we don't have a series marker, we will never have an existing
# series to match against.
series = None
@@ -1024,21 +1038,6 @@ def parse_mail(mail, list_id=None):
except SeriesReference.DoesNotExist:
SeriesReference.objects.create(series=series, msgid=ref)
-patch = Patch(
-msgid=msgid,
-project=project,
-name=name[:255],
-date=date,
-headers=headers,
-submitter=author,
-content=message,
-diff=diff,
-pull_url=pull_url,
-delegate=delegate,
-state=find_state(mail))
-patch.save()
-logger.debug('Patch saved')
-
# add to a series if we have found one, and we have a numbered
# patch. Don't add unnumbered patches (for example diffs sent
# in reply, or just messages with random refs/in-reply-tos)
--
Andrew Donnellan OzLabs, ADL Canberra
andrew.donnel...@au1.ibm.com IBM Australia Limited
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 7/9] parser: avoid an unnecessary UPDATE of Person
On 22/02/18 01:17, Daniel Axtens wrote: Analysis of SQL statements showed that when parsing an email, the row for the Person who sent the email was always getting updated. This is because the test for updating it only checks if the incoming mail has *a* name attached to the email address, and not if it has a new name. Django is not smart enough to figure that out, and so unconditionally UPDATEs the model when asked to save. Give it a hand - only update the model and save it if the new name is in fact different. Signed-off-by: Daniel Axtens <d...@axtens.net> Reviewed-by: Andrew Donnellan <andrew.donnel...@au1.ibm.com> Though, just the != test should be sufficient here without the not-null test --- patchwork/parser.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patchwork/parser.py b/patchwork/parser.py index 1f3b3dd8901f..3d40b74375e0 100644 --- a/patchwork/parser.py +++ b/patchwork/parser.py @@ -349,7 +349,7 @@ def get_or_create_author(mail): # we lost the race to create the person person = Person.objects.get(email__iexact=email) -if name: # use the latest provided name +if name and name != person.name: # use the latest provided name person.name = name person.save() -- Andrew Donnellan OzLabs, ADL Canberra andrew.donnel...@au1.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
