Re: [RFC PATCH] docker: Add support for using eatmydata in the database
On 17/9/19 11:36 pm, Stephen Finucane wrote: On Fri, 2019-05-03 at 17:33 +1000, Russell Currey wrote: When running tox on a VM with presumably pretty busy spinning disks, using eatmydata with the database took running one configuration's test suite from (no exaggeration) 20 minutes down to 60 seconds. It makes a huge difference to test speed, so we should make it easily available for developers. The primary motivation here was to automatically test each patch in a timeframe that isn't insane. Open to ideas on how to organise this, whether we do it for MySQL too (which we probably should), whether the base directory should have these files in it, what to call the Dockerfile, etc. I think it's a good thing to have in the repo, though. Signed-off-by: Russell Currey What are the implications of doing this _from a development perspective_ and can we do this by default in the two existing docker- compose files? Given that we're only talking about a development environmnent where information presumably isn't that important allied to the fact that we have the ability to back up and restore from known good points (the dbbackup, dbrestore management commands, respectively), do we need to be seriously concerned about dataloss? We don't need to worry about dataloss in a development environment, though perhaps some people are borrowing our dockerfiles for a production deployment? -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [RFC PATCH] docker: Add support for using eatmydata in the database
On Fri, 2019-05-03 at 17:33 +1000, Russell Currey wrote: > When running tox on a VM with presumably pretty busy spinning disks, > using eatmydata with the database took running one configuration's test > suite from (no exaggeration) 20 minutes down to 60 seconds. > > It makes a huge difference to test speed, so we should make it easily > available for developers. The primary motivation here was to > automatically test each patch in a timeframe that isn't insane. > > Open to ideas on how to organise this, whether we do it for MySQL too > (which we probably should), whether the base directory should have these > files in it, what to call the Dockerfile, etc. I think it's a good > thing to have in the repo, though. > > Signed-off-by: Russell Currey What are the implications of doing this _from a development perspective_ and can we do this by default in the two existing docker- compose files? Given that we're only talking about a development environmnent where information presumably isn't that important allied to the fact that we have the ability to back up and restore from known good points (the dbbackup, dbrestore management commands, respectively), do we need to be seriously concerned about dataloss? Stephen ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [RFC PATCH] docker: Add support for using eatmydata in the database
Russell Currey writes: > When running tox on a VM with presumably pretty busy spinning disks, > using eatmydata with the database took running one configuration's test > suite from (no exaggeration) 20 minutes down to 60 seconds. > > It makes a huge difference to test speed, so we should make it easily > available for developers. The primary motivation here was to > automatically test each patch in a timeframe that isn't insane. > > Open to ideas on how to organise this, whether we do it for MySQL too > (which we probably should), whether the base directory should have these > files in it, what to call the Dockerfile, etc. I think it's a good > thing to have in the repo, though. So I really want to do this but I don't really like the idea of docker-compose-eatmydata, it's just a bit ugly. I am hoping that we can get the tox-docker thing working and integrate from there, depending on how that ends up working. Regards, Daniel > > Signed-off-by: Russell Currey > --- > docker-compose-eatmydata.yml | 32 +++ > tools/docker/Dockerfile.eatmydata | 9 + > 2 files changed, 41 insertions(+) > create mode 100644 docker-compose-eatmydata.yml > create mode 100644 tools/docker/Dockerfile.eatmydata > > diff --git a/docker-compose-eatmydata.yml b/docker-compose-eatmydata.yml > new file mode 100644 > index 000..27d1604 > --- /dev/null > +++ b/docker-compose-eatmydata.yml > @@ -0,0 +1,32 @@ > +version: "3" > +services: > + db: > +build: > + context: . > + dockerfile: ./tools/docker/Dockerfile.eatmydata > +volumes: > + - ./tools/docker/db/postdata:/var/lib/postgresql/data > +environment: > + - POSTGRES_PASSWORD=password > + > + web: > +build: > + context: . > + dockerfile: ./tools/docker/Dockerfile > + args: > +- UID > +depends_on: > + - db > +command: python3 manage.py runserver 0.0.0.0:8000 > +volumes: > + - .:/home/patchwork/patchwork/ > +ports: > + - "8000:8000" > +environment: > + - UID > + - PW_TEST_DB_HOST=db > + - PW_TEST_DB_PORT=5432 > + - PW_TEST_DB_TYPE=postgres > + - PW_TEST_DB_USER=postgres > + - PW_TEST_DB_PASS=password > + - PGPASSWORD=password > diff --git a/tools/docker/Dockerfile.eatmydata > b/tools/docker/Dockerfile.eatmydata > new file mode 100644 > index 000..693cbb3 > --- /dev/null > +++ b/tools/docker/Dockerfile.eatmydata > @@ -0,0 +1,9 @@ > +FROM postgres:9.6 > + > +RUN apt-get update \ > + && apt-get install -y eatmydata \ > + && apt-get autoremove -y \ > + && rm -rf /var/lib/apt/lists/* > + > +ENTRYPOINT [ "/usr/bin/eatmydata", "/usr/local/bin/docker-entrypoint.sh" ] > +CMD ["postgres"] > -- > 2.21.0 > > ___ > Patchwork mailing list > Patchwork@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/patchwork ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [RFC PATCH] docker: Add support for using eatmydata in the database
On 3/5/19 5:33 pm, Russell Currey wrote: When running tox on a VM with presumably pretty busy spinning disks, using eatmydata with the database took running one configuration's test suite from (no exaggeration) 20 minutes down to 60 seconds. It makes a huge difference to test speed, so we should make it easily available for developers. The primary motivation here was to automatically test each patch in a timeframe that isn't insane. Open to ideas on how to organise this, whether we do it for MySQL too (which we probably should), whether the base directory should have these files in it, what to call the Dockerfile, etc. I think it's a good thing to have in the repo, though. Signed-off-by: Russell Currey Reviewed-by: Andrew Donnellan --- docker-compose-eatmydata.yml | 32 +++ tools/docker/Dockerfile.eatmydata | 9 + 2 files changed, 41 insertions(+) create mode 100644 docker-compose-eatmydata.yml create mode 100644 tools/docker/Dockerfile.eatmydata diff --git a/docker-compose-eatmydata.yml b/docker-compose-eatmydata.yml new file mode 100644 index 000..27d1604 --- /dev/null +++ b/docker-compose-eatmydata.yml @@ -0,0 +1,32 @@ +version: "3" +services: + db: +build: + context: . + dockerfile: ./tools/docker/Dockerfile.eatmydata +volumes: + - ./tools/docker/db/postdata:/var/lib/postgresql/data +environment: + - POSTGRES_PASSWORD=password + + web: +build: + context: . + dockerfile: ./tools/docker/Dockerfile + args: +- UID +depends_on: + - db +command: python3 manage.py runserver 0.0.0.0:8000 +volumes: + - .:/home/patchwork/patchwork/ +ports: + - "8000:8000" +environment: + - UID + - PW_TEST_DB_HOST=db + - PW_TEST_DB_PORT=5432 + - PW_TEST_DB_TYPE=postgres + - PW_TEST_DB_USER=postgres + - PW_TEST_DB_PASS=password + - PGPASSWORD=password diff --git a/tools/docker/Dockerfile.eatmydata b/tools/docker/Dockerfile.eatmydata new file mode 100644 index 000..693cbb3 --- /dev/null +++ b/tools/docker/Dockerfile.eatmydata @@ -0,0 +1,9 @@ +FROM postgres:9.6 + +RUN apt-get update \ + && apt-get install -y eatmydata \ + && apt-get autoremove -y \ + && rm -rf /var/lib/apt/lists/* + +ENTRYPOINT [ "/usr/bin/eatmydata", "/usr/local/bin/docker-entrypoint.sh" ] +CMD ["postgres"] -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [RFC PATCH] docker: Add support for using eatmydata in the database
Russell Currey writes: > When running tox on a VM with presumably pretty busy spinning disks, > using eatmydata with the database took running one configuration's test > suite from (no exaggeration) 20 minutes down to 60 seconds. As the author and been-attempting-to-no-longer-be-maintainer-of eatmydata, this 20x improvement in test execution time doesn't really surprise me. It turns out that properly flushing things to disk is *really* expensive. -- Stewart Smith OPAL Architect, IBM. ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
[RFC PATCH] docker: Add support for using eatmydata in the database
When running tox on a VM with presumably pretty busy spinning disks, using eatmydata with the database took running one configuration's test suite from (no exaggeration) 20 minutes down to 60 seconds. It makes a huge difference to test speed, so we should make it easily available for developers. The primary motivation here was to automatically test each patch in a timeframe that isn't insane. Open to ideas on how to organise this, whether we do it for MySQL too (which we probably should), whether the base directory should have these files in it, what to call the Dockerfile, etc. I think it's a good thing to have in the repo, though. Signed-off-by: Russell Currey --- docker-compose-eatmydata.yml | 32 +++ tools/docker/Dockerfile.eatmydata | 9 + 2 files changed, 41 insertions(+) create mode 100644 docker-compose-eatmydata.yml create mode 100644 tools/docker/Dockerfile.eatmydata diff --git a/docker-compose-eatmydata.yml b/docker-compose-eatmydata.yml new file mode 100644 index 000..27d1604 --- /dev/null +++ b/docker-compose-eatmydata.yml @@ -0,0 +1,32 @@ +version: "3" +services: + db: +build: + context: . + dockerfile: ./tools/docker/Dockerfile.eatmydata +volumes: + - ./tools/docker/db/postdata:/var/lib/postgresql/data +environment: + - POSTGRES_PASSWORD=password + + web: +build: + context: . + dockerfile: ./tools/docker/Dockerfile + args: +- UID +depends_on: + - db +command: python3 manage.py runserver 0.0.0.0:8000 +volumes: + - .:/home/patchwork/patchwork/ +ports: + - "8000:8000" +environment: + - UID + - PW_TEST_DB_HOST=db + - PW_TEST_DB_PORT=5432 + - PW_TEST_DB_TYPE=postgres + - PW_TEST_DB_USER=postgres + - PW_TEST_DB_PASS=password + - PGPASSWORD=password diff --git a/tools/docker/Dockerfile.eatmydata b/tools/docker/Dockerfile.eatmydata new file mode 100644 index 000..693cbb3 --- /dev/null +++ b/tools/docker/Dockerfile.eatmydata @@ -0,0 +1,9 @@ +FROM postgres:9.6 + +RUN apt-get update \ + && apt-get install -y eatmydata \ + && apt-get autoremove -y \ + && rm -rf /var/lib/apt/lists/* + +ENTRYPOINT [ "/usr/bin/eatmydata", "/usr/local/bin/docker-entrypoint.sh" ] +CMD ["postgres"] -- 2.21.0 ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
