Re: [PATCH 2/4] migrations: 0043_merge_patch_submission: do less work

On Jul 16, 2021, at 10:20 AM, Daniel Axtens wrote: > >  - we do a lot of legwork to prevent there being a column with the same > name in a base class and subclass. This is purely for django's benefit > as databases don't see the class relationship and handle column names > being the same

[PATCH] docker: Switch to MariaDB for wider platform support

MySQL docker images only have amd64 arch support MariaDB images have amd64, arm64v8, and ppc64le Signed-off-by: Stewart Smith --- docker-compose.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker-compose.yml b/docker-compose.yml index 1762d4a..1d49c51 100644

Re: DB-murdering API query (index suggestions needed)

> On 15 Nov 2019, at 09:43, Konstantin Ryabitsev > wrote: > > On Sat, Nov 16, 2019 at 12:48:33AM +1100, Daniel Axtens wrote: >>> GET >>> /api/1.1/patches/?project=62=2019-11-01T00:00:00_page=100=6150 >>> The query behind this takes about 1 minute to run on a 20-core HT Xeon >>> system and

Re: DB-murdering API query (index suggestions needed)

> On 15 Nov 2019, at 05:48, Daniel Axtens wrote: > >  >> >> Today, the DB behind patchwork.kernel.org was in a semi-permanent state >> of suffering due to someone trying to suck down all patches in the >> linux-arm-kernel project. This is what the API request looked like: >> >> GET >>

Re: [RFC PATCH] docker: Add support for using eatmydata in the database

aintainer-of eatmydata, this 20x improvement in test execution time doesn't really surprise me. It turns out that properly flushing things to disk is *really* expensive. -- Stewart Smith OPAL Architect, IBM. ___ Patchwork mailing list Patchwork@lists.ozlabs.

Re: Updating a "patch" state via REST

my script I use to update patch status (from notmuch mail tags) and that seems to be the request type that works for me (with patchwork.ozlabs.org) -- Stewart Smith OPAL Architect, IBM. ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 10/11] Be sensible computing project patch counts

Stephen Finucane writes: > On Fri, 2018-08-31 at 15:16 +0100, Stephen Finucane wrote: >> On Fri, 2018-08-10 at 18:01 +1000, Stewart Smith wrote: >> > Django actively fights constructing a query that isn't insane. >> > >> > So, let's go and just execute

Re: [PATCH 10/11] Be sensible computing project patch counts

Stephen Finucane writes: > On Fri, 2018-08-10 at 18:01 +1000, Stewart Smith wrote: >> Django actively fights constructing a query that isn't insane. >> >> So, let's go and just execute a raw one. This is all very standard >> SQL so should execute e

Re: [PATCH 05/11] Add covering index for /list/ query

Stephen Finucane writes: > On Fri, 2018-08-10 at 18:01 +1000, Stewart Smith wrote: >> In constructing the list of patches for a project, there are two >> main queries that are executed: >> 1) get a count() of how many patches there are >> 2) Get the page

Re: [PATCH 03/11] Add index for patchwork_comment (submission_id, date)

Stephen Finucane writes: > On Fri, 2018-08-10 at 18:00 +1000, Stewart Smith wrote: >> This (at least theoretically) should speed up displaying comments >> on patches/cover letters. It's an index that will return rows >> in-order for the query that we always do (

Re: [PATCH 02/11] 4x performance improvement for viewing patch with many comments

Stephen Finucane writes: > On Fri, 2018-08-10 at 18:00 +1000, Stewart Smith wrote: >> Using the example of id:20180720035941.6844-1-khand...@linux.vnet.ibm.com >> with my test dataset of a chunk of a variety of mailing lists, has >> this cover letter have 67 comments fro

Re: [PATCH 01/11] Improve patch listing performance (~3x)

Stephen Finucane writes: > On Fri, 2018-08-10 at 18:00 +1000, Stewart Smith wrote: >> There's two main bits that are really expensive when composing the list >> of patches for a project: the query getting the list, and the query >> finding the series for each patch. >>

Re: [PATCH 01/11] Improve patch listing performance (~3x)

Stephen Finucane writes: > On Sat, 2018-08-11 at 04:28 +1000, Daniel Axtens wrote: >> Stewart Smith writes: >> So, further to our conversation with Konstantin, I tested this against >> Django 2.0. It still saves us some time - it means we no longer load th

Re: [PATCH 02/11] 4x performance improvement for viewing patch with many comments

Daniel Axtens writes: > Stewart Smith writes: > >> Using the example of id:20180720035941.6844-1-khand...@linux.vnet.ibm.com >> with my test dataset of a chunk of a variety of mailing lists, has >> this cover letter have 67 comments from a variety of people. Thus, &g

Re: [PATCH 01/11] Improve patch listing performance (~3x)

Daniel Axtens writes: > Stewart Smith writes: > >> There's two main bits that are really expensive when composing the list >> of patches for a project: the query getting the list, and the query >> finding the series for each patch. >> >> If we look at the q

[PATCH 03/11] Add index for patchwork_comment (submission_id,date)

is a good question. Signed-off-by: Stewart Smith --- .../migrations/0027_add_comment_date_index.py | 23 +++ patchwork/models.py | 4 2 files changed, 27 insertions(+) create mode 100644 patchwork/migrations/0027_add_comment_date_index.py diff --git a

[PATCH 06/11] check distinct(user) based on just user_id

, no join or prefetching users. This saves a couple of hundred queries on the linuxppc/list/ view and makes loading it about 4x faster in terms of time spent in the db. Signed-off-by: Stewart Smith --- patchwork/models.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git

[PATCH 05/11] Add covering index for /list/ query

) the query time goes from 28ms and 72ms down to 2ms and 33-40ms! Signed-off-by: Stewart Smith --- .../0028_add_list_covering_index.py | 19 +++ patchwork/models.py | 6 ++ 2 files changed, 25 insertions(+) create mode 100644 patchwork

[PATCH 10/11] Be sensible computing project patch counts

and 1500ms, with this patch it's down to 11 queries in ~250ms. For the dataset of the netdev list, it's down to 440ms from 1500ms. Signed-off-by: Stewart Smith --- patchwork/views/project.py | 23 --- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/patchwork/views

[PATCH 07/11] Be particular over check_set and series prefetch for /list/

At this point it shaves at most 1-2ms off the query time for /linuxppc-dev/list/ Signed-off-by: Stewart Smith --- patchwork/views/__init__.py | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/patchwork/views/__init__.py b/patchwork/views/__init__.py index 96fd0798af5a

[PATCH 01/11] Improve patch listing performance (~3x)

off-by: Stewart Smith --- patchwork/models.py | 1 - patchwork/views/__init__.py | 2 ++ 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/patchwork/models.py b/patchwork/models.py index 6268f5b72e3c..d2b88fc48c91 100644 --- a/patchwork/models.py +++ b/patchwork/models.py @@ -747,7 +74

[PATCH 04/11] Fetch all series for patch/cover viewing

for all the series. Since there just usually aren't *that* many series, fetch them all and take the first one if we need to. Signed-off-by: Stewart Smith --- patchwork/templates/patchwork/submission.html | 10 +- patchwork/views/cover.py | 2 +- patchwork/views

[PATCH 08/11] Add covering index to patchwork_submissions for /list/ queries

This gets PostgreSQL to generate *much* better query plans, gaining us about two orders of magnitude in performance on the /list/ query for the worst state project on the patchwork.ozlabs.org instance (qemu-devel). Signed-off-by: Stewart Smith --- .../0029_add_submission_covering_index.py

[PATCH 00/11] Performance for ALL THE THINGS!

set :) This series replaces: https://patchwork.ozlabs.org/patch/954921/ https://patchwork.ozlabs.org/patch/955321/ https://patchwork.ozlabs.org/patch/955322/ https://patchwork.ozlabs.org/patch/955342/ https://patchwork.ozlabs.org/patch/955347/ https://patchwork.ozlabs.org/patch/955392/ Stewart Smith

[PATCH 11/11] Fetch maintainer information in one query

the page. Realistically, this doesn't save us too much time as counting how many patches are there takes 99% of the database time for this page. Signed-off-by: Stewart Smith --- patchwork/views/project.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patchwork/views

[PATCH 02/11] 4x performance improvement for viewing patch with many comments

at the same time as getting all the comments rather than doing a one-by-one lookup after the fact. Signed-off-by: Stewart Smith --- patchwork/templates/patchwork/submission.html | 2 +- patchwork/views/cover.py | 5 + patchwork/views/patch.py | 5 + 3

[PATCH 09/11] Optimise fetching checks when displaying a patch

, so we do that. This brings a page load down to 10 queries in 12ms. Signed-off-by: Stewart Smith --- patchwork/templates/patchwork/submission.html | 4 ++-- patchwork/views/patch.py | 1 + 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/patchwork/templates

[PATCH] Add covering index for /list/ query

) the query time goes from 28ms and 72ms down to 2ms and 33-40ms! Signed-off-by: Stewart Smith --- .../0028_add_list_covering_index.py | 19 +++ patchwork/models.py | 6 ++ 2 files changed, 25 insertions(+) create mode 100644 patchwork

[PATCH] Fetch all series for patch/cover viewing

for all the series. Since there just usually aren't *that* many series, fetch them all and take the first one if we need to. Signed-off-by: Stewart Smith --- patchwork/templates/patchwork/submission.html | 10 +- patchwork/views/cover.py | 2 +- patchwork/views

[PATCH v2] 4x performance improvement for viewing patch with many comments

at the same time as getting all the comments rather than doing a one-by-one lookup after the fact. Signed-off-by: Stewart Smith --- patchwork/templates/patchwork/submission.html | 2 +- patchwork/views/cover.py | 5 + patchwork/views/patch.py | 5 + 3

[PATCH] Add index for patchwork_comment (submission_id,date)

is a good question. Signed-off-by: Stewart Smith --- .../migrations/0027_add_comment_date_index.py | 23 +++ patchwork/models.py | 4 2 files changed, 27 insertions(+) create mode 100644 patchwork/migrations/0027_add_comment_date_index.py diff --git a

[PATCH] 4x performance improvement for viewing patch with many comments

at the same time as getting all the comments rather than doing a one-by-one lookup after the fact. Signed-off-by: Stewart Smith --- patchwork/templates/patchwork/submission.html | 2 +- patchwork/views/cover.py | 5 + patchwork/views/patch.py | 5 + 3

Re: [RFC PATCH] 3x improvement in patch listing

Daniel Axtens writes: > Daniel Axtens writes: > >> Konstantin Ryabitsev writes: >> >>> On Wed, Aug 08, 2018 at 05:40:06PM +1000, Stewart Smith wrote: >>>>There's two main bits that are really expensive when composing the list >>>>o

Re: [RFC PATCH] 3x improvement in patch listing

Konstantin Ryabitsev writes: > On Wed, Aug 08, 2018 at 05:40:06PM +1000, Stewart Smith wrote: >>There's two main bits that are really expensive when composing the list >>of patches for a project: the query getting the list, and the query >>finding the series for ea

[RFC PATCH] 3x improvement in patch listing

off-by: Stewart Smith --- patchwork/models.py | 1 - patchwork/views/__init__.py | 2 ++ 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/patchwork/models.py b/patchwork/models.py index 6268f5b72e3c..d2b88fc48c91 100644 --- a/patchwork/models.py +++ b/patchwork/models.py @@ -747,7 +74

Re: [RFC PATCH] REST: Add new setting for maximum API page size

set I could load up locally and experiment with? > * Yes, I probably should have realised this at the time. something > something unpaid volunteers something something patch review something > something. :) pretty much neglecting some piece of software that ends up being vital open source infrastructure is a grand tradition, welcome! -- Stewart Smith OPAL Architect, IBM. ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork

Re: [RFC PATCH] REST: Add new setting for maximum API page size

t;>>> >>>> Add a new setting, MAX_REST_RESULTS_PER_PAGE, to set the maximum page size. >>>> >>>> Closes: #202 ("Separate max API page size and default API page size into >>>> different settings") >>>> Suggested-by: Stewart S

Re: [RFC PATCH] REST: Add new setting for maximum API page size

page size and default API page size into >> different settings") >> Suggested-by: Stewart Smith >> Suggested-by: Joel Stanley >> Signed-off-by: Andrew Donnellan > > FWIW we now have this applied on patchwork.ozlabs.org and it appears to > be working. Would l