Re: [PATCH 01/11] Improve patch listing performance (~3x)

Stewart Smith writes: > There's two main bits that are really expensive when composing the list > of patches for a project: the query getting the list, and the query > finding the series for each patch. > > If we look at the query getting the list, it gets a lot of unnesseccary > fields such as

Re: [PATCH 08/11] requirements: Start using fixed versions

Stephen Finucane writes: > Given that 'tox' doesn't actually read any of these, there's no reason > to use ranges of requirements. Instead, use the latest and greatest for > live instances and rely on tox to validate behavior with older versions. > I don't know if it's this patch or one of the

Re: [PATCH 04/11] Remove support for Django 1.8, 1.9, 1.10

Stephen Finucane writes: > These are now all EOL and Debian Testing supports Django 1.11 (LTS). We > can and should drop them. > > This change does not remove the many compat wrappers. These will be > removed separately (there are a lot of them). > > This leaves 1.11 as the only supported

Re: [PATCH v2 2/3] Default dev settings, set host to empty (unix socket) on postgresql

Daniel Black writes: > On Fri, 10 Aug 2018 01:08:48 +1000 > Daniel Axtens wrote: > >> Daniel Black writes: >> >> > An empty environment variable resulted in localhost, meaning >> > posgresql connecting to domain sockets wasn't available. >> >> It took me a long time to understand how this

[PATCH 03/11] Add index for patchwork_comment (submission_id,date)

This (at least theoretically) should speed up displaying comments on patches/cover letters. It's an index that will return rows in-order for the query that we always do ("give me the comments on this submission in date order"), rather than having to have the database server do a sort for us. I

[PATCH 06/11] check distinct(user) based on just user_id

The logic to display the Check(s) on the patch list wants to really do a DISTINCT(user_id,context) ORDER BY DATE query, but with Django that is currently a bit Too Hard (at least for me). But what we can do is from python just use the user_id rather than the user object itself. Same

[PATCH 05/11] Add covering index for /list/ query

In constructing the list of patches for a project, there are two main queries that are executed: 1) get a count() of how many patches there are 2) Get the page of results being displayed In a test dataset of ~11500 LKML patches and ~4000 others, the existing code would take around 585ms and 858ms

[PATCH 10/11] Be sensible computing project patch counts

Django actively fights constructing a query that isn't insane. So, let's go and just execute a raw one. This is all very standard SQL so should execute everywhere without a problem. With the dataset of patchwork.ozlabs.org, looking at the /project/ page for qemu-devel would take 13 queries and

[PATCH 07/11] Be particular over check_set and series prefetch for /list/

At this point it shaves at most 1-2ms off the query time for /linuxppc-dev/list/ Signed-off-by: Stewart Smith --- patchwork/views/__init__.py | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/patchwork/views/__init__.py b/patchwork/views/__init__.py index

[PATCH 01/11] Improve patch listing performance (~3x)

There's two main bits that are really expensive when composing the list of patches for a project: the query getting the list, and the query finding the series for each patch. If we look at the query getting the list, it gets a lot of unnesseccary fields such as 'headers' and 'content', even

[PATCH 04/11] Fetch all series for patch/cover viewing

e.g. a 10 comment patch goes from 26 queries in 17-20ms down to 20 queries in 12ms. A 67 comment cover letter goes from 14 queries in 16ms down to 8 queries in 8ms. So, effectively, a near 2x perf improvement. Previously, at several points we were asking for the latest series and then asking

[PATCH 08/11] Add covering index to patchwork_submissions for /list/ queries

This gets PostgreSQL to generate *much* better query plans, gaining us about two orders of magnitude in performance on the /list/ query for the worst state project on the patchwork.ozlabs.org instance (qemu-devel). Signed-off-by: Stewart Smith --- .../0029_add_submission_covering_index.py |

[PATCH 00/11] Performance for ALL THE THINGS!

This is an updated and extended patch series of the few patches I've sent out over the past few days. I've had a pretty major attack at performance of as many patchwork pages and bits of functionality that I could find. I've been using a mixture of test environments for this, initially a small

[PATCH 11/11] Fetch maintainer information in one query

Viewing the /project/ page lists maintainers. Prior to this patch, this was done in one query to fetch the maintainer IDs, and then one query per mainatiner to get the name/email address. Now, with this patch, it's all in one query (yay joins) and saves a few ms of database queries for displaying

[PATCH 02/11] 4x performance improvement for viewing patch with many comments

Using the example of id:20180720035941.6844-1-khand...@linux.vnet.ibm.com with my test dataset of a chunk of a variety of mailing lists, has this cover letter have 67 comments from a variety of people. Thus, it's on the larger side of things. Originally, displaying the /patch/550/ for this

[PATCH 09/11] Optimise fetching checks when displaying a patch

Prior to this patch, a typical /patch// query for linuxppc-dev (which has about half a dozen checks per patch) took around 20 queries and 16.5ms in the database. About half of those queries were fetching the checks and who did the check. We can just do one query to get all that needed