Re: Making tests fast

[wd]

On Wed, Sep 4, 2019 at 4:52 AM Dan Davis  wrote:

> I'm wondering how to make my tests fast while also having a lot of data
> defined in the models.
> This project is already using sqlite3 and the tests are single-threaded.
>
>
Django support parallel, you can use multi process to save time.

>   --parallel [N]Run tests using up to N parallel processes.
>

 And also I don't recommend to use sqlite, I think it's not as fast as
PostgreSQL, you also can use memory database for PostgreSQL as well.

Only use the minimal required fixtures for each test, because setup and
delete the data are slow. Also use TestCase instead of TranscationTestCase
if you can, via
https://adamj.eu/tech/2019/07/15/djangos-test-case-classes-and-a-three-times-speed-up/
.


> My ideas are:
>
>- Specify a different sqlite3 file and keepdb for different tests -
>presumably with @override_settings
>- Do something to copy this data into memory (see
>
> https://stackoverflow.com/questions/3850022/how-to-load-existing-db-file-to-memory-in-python-sqlite3)
>- I suspect Django already supports this, but I don't see where
>_clone_test_db is called.
>
> I am loathe to use modelmommy and/or factoryboy because they are simply a
> bit harder to use.   I'd rather use dump data/load data with :memory
> database.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/b22204f4-85db-4eb9-9a53-adca0dea269c%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CABexzmiyOPwZ8X3-5WySF-Rocr-RZ_hCMQm0udijLu-10o7Dew%40mail.gmail.com.

Re: Making tests fast

[Mike Dewhirst]

On 4/09/2019 6:51 am, Dan Davis wrote:
I'm wondering how to make my tests fast while also having a lot of 
data defined in the models.

This project is already using sqlite3 and the tests are single-threaded.

My ideas are:

  * Specify a different sqlite3 file and keepdb for different tests -
presumably with @override_settings
  * Do something to copy this data into memory
(see 
https://stackoverflow.com/questions/3850022/how-to-load-existing-db-file-to-memory-in-python-sqlite3)
- I suspect Django already supports this, but I don't see where
_clone_test_db is called.

I am loathe to use modelmommy and/or factoryboy because they are 
simply a bit harder to use.   I'd rather use dump data/load data with 
:memory database.


I agree. I use fixtures a lot and things go slow. Like the little green 
frog in the pot getting warmer I've just been getting used to it. 
However, you have inspired me to think again ...


1. I'm going out now to buy a SSD. I think that is a reasonable 
compromise between a spinning disk and a RAM disk. I run about 1400 
tests in around 40 minutes.


2. I made a BaseTest class which uses TestCase and sets up minimal model 
instances but DOES NOT use fixtures


3. All tests inherit from BaseTest but only those which actually require 
data need to use the fixtures


4. I have decided to give up SQLite3 for tests and stick with PostgreSQL 
because I discovered the hard way that some tests pass on one but not 
the other.


5. I'm going to set up a separate machine to do CI. Real soon now.

Cheers

Mike




--
You received this message because you are subscribed to the Google 
Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to django-users+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b22204f4-85db-4eb9-9a53-adca0dea269c%40googlegroups.com 
.


--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/8c2d58b1-ba91-f83d-e36f-12beaa7b109b%40dewhirst.com.au.

Re: Making tests fast

[Dan Davis]

I will just use fixtures, natural-keys, and an in-memory database.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/a4601657-35bf-445a-a972-2abda5ddabc3%40googlegroups.com.

Making tests fast

[Dan Davis]

I'm wondering how to make my tests fast while also having a lot of data 
defined in the models.  
This project is already using sqlite3 and the tests are single-threaded.

My ideas are:

   - Specify a different sqlite3 file and keepdb for different tests - 
   presumably with @override_settings
   - Do something to copy this data into memory 
   (see 
https://stackoverflow.com/questions/3850022/how-to-load-existing-db-file-to-memory-in-python-sqlite3)
 
   - I suspect Django already supports this, but I don't see where 
   _clone_test_db is called.

I am loathe to use modelmommy and/or factoryboy because they are simply a 
bit harder to use.   I'd rather use dump data/load data with :memory 
database.

   

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b22204f4-85db-4eb9-9a53-adca0dea269c%40googlegroups.com.