Re: [PATCH 2/2] api: allow filtering patches and covers by msgid
Daniel Axtens writes: > In the process of fixing the previous bug, I realised that: > > a) /api/patches/msgid is a perfectly reasonable thing to attempt > b) We have no way of finding a patch by message id in the API > > We can't actualy make /api/patches/msgid work because it may not > be unique, but we can add a filter. > > I'm shoehorning this into stable/2.2, even though it's technically > an API change: it's minor, not incompatible and in hindsight a > glaring hole. > > Cc: Michael Ellerman > Signed-off-by: Daniel Axtens Thanks, this is awesome. For reference here's my updated terrible script that I use to apply a series from a patch id or message id. cheers #!/usr/bin/python3 import sys import requests from subprocess import check_call session = requests.Session() arg = sys.argv[1] try: pid = int(arg) except ValueError: pid = None if pid is None: arg = arg.replace('id:', '') url = f'https://patchwork.ozlabs.org/api/patches/?msgid={arg}=linuxppc-dev' json = session.get(url).json()[0] else: url = f'https://patchwork.ozlabs.org/api/patches/{pid}/' json = session.get(url).json() sid = json['series'][0]['id'] cmd = f'git pw series apply {sid}' check_call(cmd.split()) print("OK") ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 2/2] api: allow filtering patches and covers by msgid
Stephen Finucane writes: > On Tue, 2020-04-14 at 16:21 +1000, Daniel Axtens wrote: >> In the process of fixing the previous bug, I realised that: >> >> a) /api/patches/msgid is a perfectly reasonable thing to attempt >> b) We have no way of finding a patch by message id in the API >> >> We can't actualy make /api/patches/msgid work because it may not >> be unique, but we can add a filter. >> >> I'm shoehorning this into stable/2.2, even though it's technically >> an API change: it's minor, not incompatible and in hindsight a >> glaring hole. >> >> Cc: Michael Ellerman >> Signed-off-by: Daniel Axtens > > I meant to do this before cutting v2.2.0 and forgot. Probably should > have had a list. > > It's not ideal putting this into an already released API version, but I > agree it's a huge oversight. Let's make sure we get v2.2.1 out nice and > quickly. Indeed, doing this now. Thanks! Regards, Daniel > > Reviewed-by: Stephen Finucane > > Stephen ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Patchwork v2.2.1 Available
We're pleased to announce the release of Patchwork v2.2.1. This release is part of the "Flannel" release series: https://github.com/getpatchwork/patchwork/releases/tag/v2.2.1 This release focuses on bugfixes, including an issue where Patchwork would attempt to construct a list of every single patch in the database, which is likely to lead to massive server load. As such, if you are running v2.2.0 we recommend upgrading immediately. If you are running a v2.1.x version, you should upgrade directly to v2.2.1 without going through v2.2.0 as this version also contains a fix to the database permission grant script that will ease your upgrade. Happy patchworking! --- Changes in Patchwork v2.2.0..v2.2.1 --- 123bfffe5187 Release 2.2.1 0a7a74a59bf9 api: allow filtering patches and covers by msgid 40f1fa9e9316 api: do not fetch every patch in a patch detail view 404 9c7bc7f8506d lib/sql: Update grant script for recent schema changes 2111f726e173 docs: Reference Patchwork 2.2 tarballs 6d959769f3a0 Post-release version bump ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 1/2] api: do not fetch every patch in a patch detail view 404
On Tue, 2020-04-14 at 16:21 +1000, Daniel Axtens wrote: > mpe and jk and sfr found that the OzLabs server was melting due > to some queries downloading every patch. > > Turns out if you 404 the patch detail view in the API, d-r-f attempts > to render a listbox with every single patch to fill in the 'related' > field. The bundle API also has a similar field. > > Replace the multiple selection box with a text field. You can still > (AIUI) populate the relevant patch IDs manually. > > Reported-by: Jeremy Kerr > Reported-by: Michael Ellerman > Reported-by: Stephen Rothwell > Signed-off-by: Daniel Axtens It's a crying shame that DRF doesn't provide any AJAX'y widgets out of the box, though I guess that would involve choosing a framework since JQuery is dying. Given they don't, this is the correct solution. Even the docs [1] say so! Reviewed-by: Stephen Finucane Stephen PS: It would be nice to include [1] in the commit message when merging. [1] https://www.django-rest-framework.org/topics/browsable-api/#handling-choicefield-with-large-numbers-of-items ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 2/2] api: allow filtering patches and covers by msgid
On Tue, 2020-04-14 at 16:21 +1000, Daniel Axtens wrote: > In the process of fixing the previous bug, I realised that: > > a) /api/patches/msgid is a perfectly reasonable thing to attempt > b) We have no way of finding a patch by message id in the API > > We can't actualy make /api/patches/msgid work because it may not > be unique, but we can add a filter. > > I'm shoehorning this into stable/2.2, even though it's technically > an API change: it's minor, not incompatible and in hindsight a > glaring hole. > > Cc: Michael Ellerman > Signed-off-by: Daniel Axtens I meant to do this before cutting v2.2.0 and forgot. Probably should have had a list. It's not ideal putting this into an already released API version, but I agree it's a huge oversight. Let's make sure we get v2.2.1 out nice and quickly. Reviewed-by: Stephen Finucane Stephen ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 2/2] api: allow filtering patches and covers by msgid
On 14/4/20 4:21 pm, Daniel Axtens wrote: In the process of fixing the previous bug, I realised that: a) /api/patches/msgid is a perfectly reasonable thing to attempt b) We have no way of finding a patch by message id in the API We can't actualy make /api/patches/msgid work because it may not be unique, but we can add a filter. I'm shoehorning this into stable/2.2, even though it's technically an API change: it's minor, not incompatible and in hindsight a glaring hole. Cc: Michael Ellerman Signed-off-by: Daniel Axtens This is indeed something we would expect to be able to do in the API now that we support lookup by message ID for the web. Reviewed-by: Andrew Donnellan --- docs/api/schemas/latest/patchwork.yaml | 16 docs/api/schemas/patchwork.j2 | 18 ++ docs/api/schemas/v1.2/patchwork.yaml | 16 patchwork/api/filters.py | 14 ++ patchwork/tests/api/test_cover.py | 12 patchwork/tests/api/test_patch.py | 12 .../rest-filter-msgid-41f693cd4e53cf93.yaml| 6 ++ 7 files changed, 90 insertions(+), 4 deletions(-) create mode 100644 releasenotes/notes/rest-filter-msgid-41f693cd4e53cf93.yaml diff --git a/docs/api/schemas/latest/patchwork.yaml b/docs/api/schemas/latest/patchwork.yaml index 13cdc9cd78fd..cc0d97e696b6 100644 --- a/docs/api/schemas/latest/patchwork.yaml +++ b/docs/api/schemas/latest/patchwork.yaml @@ -246,6 +246,14 @@ paths: schema: title: '' type: string +- in: query + name: msgid + description: > +The cover message-id as a case-sensitive string, without leading or +trailing angle brackets, to filter by. + schema: +title: '' +type: string responses: '200': description: '' @@ -474,6 +482,14 @@ paths: schema: title: '' type: string +- in: query + name: msgid + description: > +The patch message-id as a case-sensitive string, without leading or +trailing angle brackets, to filter by. + schema: +title: '' +type: string responses: '200': description: '' diff --git a/docs/api/schemas/patchwork.j2 b/docs/api/schemas/patchwork.j2 index bd714d5e7a2a..f5618d41faa0 100644 --- a/docs/api/schemas/patchwork.j2 +++ b/docs/api/schemas/patchwork.j2 @@ -251,6 +251,16 @@ paths: schema: title: '' type: string +{% if version >= (1, 2) %} +- in: query + name: msgid + description: > +The cover message-id as a case-sensitive string, without leading or +trailing angle brackets, to filter by. + schema: +title: '' +type: string +{% endif %} responses: '200': description: '' @@ -488,6 +498,14 @@ paths: schema: title: '' type: string +- in: query + name: msgid + description: > +The patch message-id as a case-sensitive string, without leading or +trailing angle brackets, to filter by. + schema: +title: '' +type: string {% endif %} responses: '200': diff --git a/docs/api/schemas/v1.2/patchwork.yaml b/docs/api/schemas/v1.2/patchwork.yaml index db2ed122eec2..7bdbe66997c0 100644 --- a/docs/api/schemas/v1.2/patchwork.yaml +++ b/docs/api/schemas/v1.2/patchwork.yaml @@ -246,6 +246,14 @@ paths: schema: title: '' type: string +- in: query + name: msgid + description: > +The cover message-id as a case-sensitive string, without leading or +trailing angle brackets, to filter by. + schema: +title: '' +type: string responses: '200': description: '' @@ -474,6 +482,14 @@ paths: schema: title: '' type: string +- in: query + name: msgid + description: > +The patch message-id as a case-sensitive string, without leading or +trailing angle brackets, to filter by. + schema: +title: '' +type: string responses: '200': description: '' diff --git a/patchwork/api/filters.py b/patchwork/api/filters.py index deb5ace11880..93e6281bf5e6 100644 --- a/patchwork/api/filters.py +++ b/patchwork/api/filters.py @@ -184,6 +184,10 @@ class SeriesFilterSet(TimestampMixin, BaseFilterSet): fields = ('submitter', 'project') +def msgid_filter(queryset, name, value): +return queryset.filter(**{name: '<' + value + '>'}) + + class
Re: [PATCH 1/2] api: do not fetch every patch in a patch detail view 404
Hi Daniel, > mpe and jk and sfr found that the OzLabs server was melting due > to some queries downloading every patch. > > Turns out if you 404 the patch detail view in the API, d-r-f attempts > to render a listbox with every single patch to fill in the 'related' > field. ... and the query for that includes all patch content and headers, so transferring gigabytes of data per access. > The bundle API also has a similar field. > > Replace the multiple selection box with a text field. You can still > (AIUI) populate the relevant patch IDs manually. Much better, the patchwork server is no longer on fire! Thanks for that. Tested-by: Jeremy Kerr Server-no-longer-on-fire-by: Jeremy Kerr Cheers, Jeremy ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH 2/2] api: allow filtering patches and covers by msgid
Hi Daniel, > In the process of fixing the previous bug, I realised that: > > a) /api/patches/msgid is a perfectly reasonable thing to attempt > b) We have no way of finding a patch by message id in the API > > We can't actualy make /api/patches/msgid work because it may not > be unique, but we can add a filter. LGTM: https://patchwork.ozlabs.org/api/patches/?msgid=20200414062102.6798-3-...@axtens.net Tested-by: Jeremy Kerr Cheers, Jeremy ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Patchwork v2.1.6 Available
We're pleased to announce the release of Patchwork v2.1.6. This release is part of the "Eolienne" release series: https://github.com/getpatchwork/patchwork/releases/tag/v2.1.6 This release is a PATCH release that focuses on bugfixes. For more details, please see below. Note that Patchwork 2.2 is also now available. It includes these fixes and many new features. Happy patchworking! --- Changes in Patchwork v2.1.5..v2.1.6 --- 2dd5d6c82d51 Release 2.1.6 eddea66d243d REST: Add release note for faster queries 90d85e24f120 REST: extend performance improvements to other parts of the API 4e3100fb3d26 REST: fix patch listing query 913f1957f934 REST: massively improve the patch counting query under filters f40bcd034010 REST: Fix duplicate project queries 58051431a33f Handle pull requests with random trailing space aa75400ff7cf parser: Fix style issues 86137cc1fa42 Improve pull request URL matching regex 37179103a0dd docs: Fix link to deployment guide da24345592cd templates: Fix mismatched close tags ea6a7bb00ec7 sql: Fix table lists fcac7790c5e3 Post-release version bump ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH 2/2] api: allow filtering patches and covers by msgid
In the process of fixing the previous bug, I realised that: a) /api/patches/msgid is a perfectly reasonable thing to attempt b) We have no way of finding a patch by message id in the API We can't actualy make /api/patches/msgid work because it may not be unique, but we can add a filter. I'm shoehorning this into stable/2.2, even though it's technically an API change: it's minor, not incompatible and in hindsight a glaring hole. Cc: Michael Ellerman Signed-off-by: Daniel Axtens --- docs/api/schemas/latest/patchwork.yaml | 16 docs/api/schemas/patchwork.j2 | 18 ++ docs/api/schemas/v1.2/patchwork.yaml | 16 patchwork/api/filters.py | 14 ++ patchwork/tests/api/test_cover.py | 12 patchwork/tests/api/test_patch.py | 12 .../rest-filter-msgid-41f693cd4e53cf93.yaml| 6 ++ 7 files changed, 90 insertions(+), 4 deletions(-) create mode 100644 releasenotes/notes/rest-filter-msgid-41f693cd4e53cf93.yaml diff --git a/docs/api/schemas/latest/patchwork.yaml b/docs/api/schemas/latest/patchwork.yaml index 13cdc9cd78fd..cc0d97e696b6 100644 --- a/docs/api/schemas/latest/patchwork.yaml +++ b/docs/api/schemas/latest/patchwork.yaml @@ -246,6 +246,14 @@ paths: schema: title: '' type: string +- in: query + name: msgid + description: > +The cover message-id as a case-sensitive string, without leading or +trailing angle brackets, to filter by. + schema: +title: '' +type: string responses: '200': description: '' @@ -474,6 +482,14 @@ paths: schema: title: '' type: string +- in: query + name: msgid + description: > +The patch message-id as a case-sensitive string, without leading or +trailing angle brackets, to filter by. + schema: +title: '' +type: string responses: '200': description: '' diff --git a/docs/api/schemas/patchwork.j2 b/docs/api/schemas/patchwork.j2 index bd714d5e7a2a..f5618d41faa0 100644 --- a/docs/api/schemas/patchwork.j2 +++ b/docs/api/schemas/patchwork.j2 @@ -251,6 +251,16 @@ paths: schema: title: '' type: string +{% if version >= (1, 2) %} +- in: query + name: msgid + description: > +The cover message-id as a case-sensitive string, without leading or +trailing angle brackets, to filter by. + schema: +title: '' +type: string +{% endif %} responses: '200': description: '' @@ -488,6 +498,14 @@ paths: schema: title: '' type: string +- in: query + name: msgid + description: > +The patch message-id as a case-sensitive string, without leading or +trailing angle brackets, to filter by. + schema: +title: '' +type: string {% endif %} responses: '200': diff --git a/docs/api/schemas/v1.2/patchwork.yaml b/docs/api/schemas/v1.2/patchwork.yaml index db2ed122eec2..7bdbe66997c0 100644 --- a/docs/api/schemas/v1.2/patchwork.yaml +++ b/docs/api/schemas/v1.2/patchwork.yaml @@ -246,6 +246,14 @@ paths: schema: title: '' type: string +- in: query + name: msgid + description: > +The cover message-id as a case-sensitive string, without leading or +trailing angle brackets, to filter by. + schema: +title: '' +type: string responses: '200': description: '' @@ -474,6 +482,14 @@ paths: schema: title: '' type: string +- in: query + name: msgid + description: > +The patch message-id as a case-sensitive string, without leading or +trailing angle brackets, to filter by. + schema: +title: '' +type: string responses: '200': description: '' diff --git a/patchwork/api/filters.py b/patchwork/api/filters.py index deb5ace11880..93e6281bf5e6 100644 --- a/patchwork/api/filters.py +++ b/patchwork/api/filters.py @@ -184,6 +184,10 @@ class SeriesFilterSet(TimestampMixin, BaseFilterSet): fields = ('submitter', 'project') +def msgid_filter(queryset, name, value): +return queryset.filter(**{name: '<' + value + '>'}) + + class CoverLetterFilterSet(TimestampMixin, BaseFilterSet): project = ProjectFilter(queryset=Project.objects.all(), distinct=False) @@ -192,6 +196,7 @@ class CoverLetterFilterSet(TimestampMixin, BaseFilterSet): series = BaseFilter(queryset=Project.objects.all(),
[PATCH 1/2] api: do not fetch every patch in a patch detail view 404
mpe and jk and sfr found that the OzLabs server was melting due to some queries downloading every patch. Turns out if you 404 the patch detail view in the API, d-r-f attempts to render a listbox with every single patch to fill in the 'related' field. The bundle API also has a similar field. Replace the multiple selection box with a text field. You can still (AIUI) populate the relevant patch IDs manually. Reported-by: Jeremy Kerr Reported-by: Michael Ellerman Reported-by: Stephen Rothwell Signed-off-by: Daniel Axtens --- patchwork/api/bundle.py | 3 ++- patchwork/api/embedded.py | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/patchwork/api/bundle.py b/patchwork/api/bundle.py index b8c0f1781786..54a9266e7d73 100644 --- a/patchwork/api/bundle.py +++ b/patchwork/api/bundle.py @@ -62,7 +62,8 @@ class BundleSerializer(BaseHyperlinkedModelSerializer): project = ProjectSerializer(read_only=True) mbox = SerializerMethodField() owner = UserSerializer(read_only=True) -patches = PatchSerializer(many=True, required=True) +patches = PatchSerializer(many=True, required=True, + style={'base_template': 'input.html'}) def get_web_url(self, instance): request = self.context.get('request') diff --git a/patchwork/api/embedded.py b/patchwork/api/embedded.py index 85a30cae1cdf..cb3f07e6b998 100644 --- a/patchwork/api/embedded.py +++ b/patchwork/api/embedded.py @@ -141,7 +141,8 @@ class PatchSerializer(SerializedRelatedField): class PatchRelationSerializer(BaseHyperlinkedModelSerializer): """Hide the PatchRelation model, just show the list""" -patches = PatchSerializer(many=True) +patches = PatchSerializer(many=True, + style={'base_template': 'input.html'}) def to_internal_value(self, data): if not isinstance(data, type([])): -- 2.20.1 ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[PATCH 0/2] v2.2 fixups for OzLabs
OzLabs found a way to get PW to try to load the entire patch db in one query, which was melting things. Fix that. In the process I realised there's no straightforward way to look up a patch by message id in the API, which is a bit more of a problem as the web UI moves to message-id URLs. I've written a simple filter which I'm just going to shoehorn into 2.2.1. Yes, it's a breach of semver conventions, but it's a pretty important feature that we should have had all along, so best to get it in now before its absence affects too many people. Daniel Axtens (2): api: do not fetch every patch in a patch detail view 404 api: allow filtering patches and covers by msgid docs/api/schemas/latest/patchwork.yaml | 16 docs/api/schemas/patchwork.j2 | 18 ++ docs/api/schemas/v1.2/patchwork.yaml | 16 patchwork/api/bundle.py| 3 ++- patchwork/api/embedded.py | 3 ++- patchwork/api/filters.py | 14 ++ patchwork/tests/api/test_cover.py | 12 patchwork/tests/api/test_patch.py | 12 .../rest-filter-msgid-41f693cd4e53cf93.yaml| 6 ++ 9 files changed, 94 insertions(+), 6 deletions(-) create mode 100644 releasenotes/notes/rest-filter-msgid-41f693cd4e53cf93.yaml -- 2.20.1 ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] lib/sql: Update grant script for recent schema changes
Hi Jeremy, Applied, thank you. I will backport it for 2.2.1 > This change fixes a few omissions in the grant scripts: > > - patchrelation is missing from both mysql and postgres scripts; it's > only needed for web user access. > > - event is missing from the web grants on postgres, and the mail grants > on mysql. > > Tested on postgres only. > > Fixes: 27c2acf56c ("models, templates: Add patch relations") > Fixes: 34e3c9c493 ("sql: Update 'grant-all.mysql' script with missing tables") > Fixes: 234bc7c316 ("lib/sql: fix permissions for v2.0.0 on postgres") > Signed-off-by: Jeremy Kerr > --- > lib/sql/grant-all.mysql.sql| 2 ++ > lib/sql/grant-all.postgres.sql | 4 > 2 files changed, 6 insertions(+) > > diff --git a/lib/sql/grant-all.mysql.sql b/lib/sql/grant-all.mysql.sql > index 0277077..100cd38 100644 > --- a/lib/sql/grant-all.mysql.sql > +++ b/lib/sql/grant-all.mysql.sql > @@ -23,6 +23,7 @@ GRANT SELECT, UPDATE, INSERT, DELETE ON > patchwork_emailoptout TO 'www-data'@loca > GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_event TO > 'www-data'@localhost; > GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_patch TO > 'www-data'@localhost; > GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_patchchangenotification TO > 'www-data'@localhost; > +GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_patchrelation TO > 'www-data'@localhost; > GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_patchtag TO > 'www-data'@localhost; > GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_person TO > 'www-data'@localhost; > GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_project TO > 'www-data'@localhost; > @@ -38,6 +39,7 @@ GRANT SELECT, UPDATE, INSERT, DELETE ON > patchwork_userprofile_maintainer_project > -- cover letters) and series > GRANT INSERT, SELECT ON patchwork_comment TO 'nobody'@localhost; > GRANT INSERT, SELECT ON patchwork_coverletter TO 'nobody'@localhost; > +GRANT INSERT, SELECT ON patchwork_event TO 'nobody'@localhost; > GRANT INSERT, SELECT ON patchwork_patch TO 'nobody'@localhost; > GRANT INSERT, SELECT ON patchwork_person TO 'nobody'@localhost; > GRANT INSERT, SELECT ON patchwork_series TO 'nobody'@localhost; > diff --git a/lib/sql/grant-all.postgres.sql b/lib/sql/grant-all.postgres.sql > index 10ec8d2..56a2486 100644 > --- a/lib/sql/grant-all.postgres.sql > +++ b/lib/sql/grant-all.postgres.sql > @@ -21,8 +21,10 @@ GRANT SELECT, UPDATE, INSERT, DELETE ON > patchwork_delegationrule, > patchwork_emailconfirmation, > patchwork_emailoptout, > + patchwork_event, > patchwork_patch, > patchwork_patchchangenotification, > + patchwork_patchrelation, > patchwork_patchtag, > patchwork_person, > patchwork_project, > @@ -50,7 +52,9 @@ GRANT SELECT, UPDATE ON > patchwork_comment_id_seq, > patchwork_delegationrule_id_seq, > patchwork_emailconfirmation_id_seq, > + patchwork_event_id_seq, > patchwork_patch_id_seq, > + patchwork_patchrelation_id_seq, > patchwork_patchtag_id_seq, > patchwork_person_id_seq, > patchwork_project_id_seq, > -- > 2.17.1 > > ___ > Patchwork mailing list > Patchwork@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/patchwork ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork