ktmud opened a new pull request #11209: URL: https://github.com/apache/incubator-superset/pull/11209
### SUMMARY We have tens of thousands of dashboards and more than 1 million slices in our Superset deployment. It takes forever to run the db migration for #11098 . This PR tries to speed up the migration process by adding pagination to db queries. I also tried to utilize `ThreadPoolExecutor` to parallelize the db operations and uuid generations, but it doesn't seem to help the overall speed much. Also fixes certain API errors while downgrading in MySQL. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF N/A ### TEST PLAN Make a copy of your very large database, then try: ``` superset db downgrade superset db upgrade ``` ### ADDITIONAL INFORMATION <!--- Check any relevant boxes with "x" --> <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue --> - [x] Has associated issue: - [ ] Changes UI - [ ] Requires DB Migration. - [x] Confirm DB Migration upgrade and downgrade tested. - [ ] Introduces new feature or API - [ ] Removes existing feature or API ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
