Re: [PATCH 2/4] migrations: 0043_merge_patch_submission: do less work

2021-07-16 Thread Stewart Smith
On Jul 16, 2021, at 10:20 AM, Daniel Axtens wrote: > >  - we do a lot of legwork to prevent there being a column with the same > name in a base class and subclass. This is purely for django's benefit > as databases don't see the class relationship and handle column names > being the same

[PATCH 4/4] RFC: migrations: 0043: Perform ALTER TABLE in one big batch

2021-07-16 Thread Daniel Axtens
This saves us doing a whole bunch of rewrites sequentially - instead we just do one big rewrite. Currently breaks postgres and sqlite, which is annoying because they don't need this to go faster. OTOH it makes mysql a **lot** faster - from around 220s to ~45s, and that will scale with table size.

[PATCH 3/4] migrations: 0043: split the data migration into chunks

2021-07-16 Thread Daniel Axtens
Migrate 1 rows at a time. This: - provides a view on progress - means replication happens in manageable chunks - hopefully prevents db lockups Signed-off-by: Daniel Axtens --- .../migrations/0043_merge_patch_submission.py | 146 ++ 1 file changed, 82 insertions(+), 64

[PATCH 2/4] migrations: 0043_merge_patch_submission: do less work

2021-07-16 Thread Daniel Axtens
- we do a lot of legwork to prevent there being a column with the same name in a base class and subclass. This is purely for django's benefit as databases don't see the class relationship and handle column names being the same across tables just fine. Unfortunately renaming columns in

[PATCH 1/4] migrations: don't go to the db for 0041_python3 migration

2021-07-16 Thread Daniel Axtens
When we moved to Python 3, makemigrations wanted to change a bunch of things, see commit 4ad87ed72aec ("migrations: Add the Python 3 patch"). The change is, as observed then, just to make django happy; it's not supposed to change the database at all. So make the migration change the state as

[PATCH 0/4] RFC: Patchwork 2.2->3.0 migration improvements

2021-07-16 Thread Daniel Axtens
Larger deployments have noticed that our migrations are a bit suboptimal, especially on clustered MySQL deployments. Both our larger deployments are also on 2.2 still. This series speeds up migrations from ~7-8 mins to 1-1.5 mins for my 558 MB MariaDB test database. It also does the data