[PATCH 4/4] docker-compose: Massively simplify

[Stephen Finucane]

Now that we can run tox on the localhost, we no longer need to install
the variety of Python versions we previously did and can keep this
purely for the 'manage.py' use cases.

Note that we're also able to remove the 'libmysqlclient-dev' and
'mysql-client' packages thanks to our switch to 'PyMySQL', which is pure
Python.

Signed-off-by: Stephen Finucane 
---
 tools/docker/Dockerfile| 22 ++
 tools/docker/entrypoint.sh |  4 ++--
 tools/docker/trusty-ports.list |  3 ---
 tools/docker/trusty.list   |  3 ---
 tools/docker/xenial-ports.list |  3 ---
 tools/docker/xenial.list   |  3 ---
 tox.ini| 15 ++-
 7 files changed, 18 insertions(+), 35 deletions(-)
 delete mode 100644 tools/docker/trusty-ports.list
 delete mode 100644 tools/docker/trusty.list
 delete mode 100644 tools/docker/xenial-ports.list
 delete mode 100644 tools/docker/xenial.list

diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile
index 76bb6b2b..5653ec89 100644
--- a/tools/docker/Dockerfile
+++ b/tools/docker/Dockerfile
@@ -17,26 +17,10 @@ ENV PYTHONUNBUFFERED 1
 
 
 # System
-# trusty and findutils is for python3.4; xenial is for python3.5
-# TODO(stephenfin): Are curl, unzip required?
-COPY tools/docker/*.list /etc/apt/sources.list.d/
-
-RUN cd /etc/apt/sources.list.d; \
-echo $(uname -m) > /tmp/arch; \
-if [ $(cat /tmp/arch) != 'x86_64' ] && grep -q -v "i.86" /tmp/arch; then \
-mv trusty-ports.list trusty.list; \
-mv xenial-ports.list xenial.list; \
-else \
-rm *-ports.list; \
-fi
-
 RUN apt-get update -qq && \
 apt-get install -y --no-install-recommends --allow-downgrades \
-python-dev python-pip python-setuptools python-wheel \
-python3.5-dev python3-pip python3-setuptools python3-wheel \
-python3.4-dev findutils=4.4.2-7 python3.6-dev \
-libmysqlclient-dev mysql-client curl unzip build-essential \
-git postgresql-client tzdata libpq-dev
+python3-dev python3-pip python3-setuptools python3-wheel \
+postgresql-client tzdata
 
 # User
 RUN useradd --uid=$UID --create-home patchwork
@@ -51,8 +35,6 @@ RUN rm /etc/localtime; ln -s /usr/share/zoneinfo/$TZ 
/etc/localtime
 COPY requirements-*.txt /tmp/
 RUN pip3 install virtualenv tox && \
 pip3 install -r /tmp/requirements-dev.txt
-RUN pip2 install virtualenv tox && \
-pip2 install -r /tmp/requirements-dev.txt
 # we deliberately leave the requirements files in tmp so we can
 # ping the user in entrypoint.sh if the change them!
 
diff --git a/tools/docker/entrypoint.sh b/tools/docker/entrypoint.sh
index 32f7132a..b28de5a0 100755
--- a/tools/docker/entrypoint.sh
+++ b/tools/docker/entrypoint.sh
@@ -124,8 +124,8 @@ elif [ "$1" == "--test" ] || [ "$1" == "--quick-test" ]; 
then
 shift
 python3 manage.py test $@
 elif [ "$1" == "--tox" ] || [ "$1" == "--quick-tox" ]; then
-shift
-tox $@
+echo "tox is no longer installed here; use e.g. 'tox -e 
py36-django21-mysql' instead"
+exit 0
 else # run whatever CMD is set to
 $@
 fi
diff --git a/tools/docker/trusty-ports.list b/tools/docker/trusty-ports.list
deleted file mode 100644
index ebcf4fa4..
--- a/tools/docker/trusty-ports.list
+++ /dev/null
@@ -1,3 +0,0 @@
-deb http://ports.ubuntu.com/ubuntu-ports/ trusty main
-deb http://ports.ubuntu.com/ubuntu-ports/ trusty-updates main
-deb http://ports.ubuntu.com/ubuntu-ports/ trusty-security main
diff --git a/tools/docker/trusty.list b/tools/docker/trusty.list
deleted file mode 100644
index 8bb92c09..
--- a/tools/docker/trusty.list
+++ /dev/null
@@ -1,3 +0,0 @@
-deb http://archive.ubuntu.com/ubuntu/ trusty main
-deb http://archive.ubuntu.com/ubuntu/ trusty-updates main
-deb http://security.ubuntu.com/ubuntu trusty-security main
diff --git a/tools/docker/xenial-ports.list b/tools/docker/xenial-ports.list
deleted file mode 100644
index d84641fa..
--- a/tools/docker/xenial-ports.list
+++ /dev/null
@@ -1,3 +0,0 @@
-deb http://ports.ubuntu.com/ubuntu-ports/ xenial main
-deb http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main
-deb http://ports.ubuntu.com/ubuntu-ports/ xenial-security main
diff --git a/tools/docker/xenial.list b/tools/docker/xenial.list
deleted file mode 100644
index a70ff56a..
--- a/tools/docker/xenial.list
+++ /dev/null
@@ -1,3 +0,0 @@
-deb http://archive.ubuntu.com/ubuntu/ xenial main
-deb http://archive.ubuntu.com/ubuntu/ xenial-updates main
-deb http://security.ubuntu.com/ubuntu xenial-security main
diff --git a/tox.ini b/tox.ini
index 86fb62e1..22291748 100644
--- a/tox.ini
+++ b/tox.ini
@@ -90,7 +90,20 @@ deps =
 commands = pylint patchwork --rcfile=pylint.rc
 
 [testenv:venv]
-commands = {posargs}
+deps =
+-rrequirements-dev.txt
+docker =
+postgres:9.6
+setenv =
+PW_TEST_DB_TYPE = postgres
+PW_TEST_DB_USER = postgres
+PW_TEST_DB_PASS = password
+PGPASSWORD = password
+dockerenv =
+POSTGRES_PASSWORD=password
+commands =
+sleep 5
+

[PATCH 2/4] tox: Integrate tox-docker

[Stephen Finucane]

This eliminates the need to use docker-compose for most use cases.
Instead, we can now do:

tox -e py27-django111-postgres

If you're using a locally configured PostgreSQL or MySQL instance, you
simply omit the last factor and things behave as before:

tox -e py27-django111

Signed-off-by: Stephen Finucane 
Cc: Daniel Axtens 
---
 docs/development/contributing.rst | 28 +++-
 docs/development/installation.rst |  6 --
 patchwork/settings/dev.py | 12 
 requirements-dev.txt  |  2 ++
 requirements-test.txt |  2 --
 tox.ini   | 27 ++-
 6 files changed, 47 insertions(+), 30 deletions(-)

diff --git a/docs/development/contributing.rst 
b/docs/development/contributing.rst
index 5089bba8..f713f872 100644
--- a/docs/development/contributing.rst
+++ b/docs/development/contributing.rst
@@ -31,29 +31,15 @@ Testing
 ---
 
 Patchwork includes a `tox`_ script to automate testing. This requires a
-functional database and some Python requirements like *tox*. Refer to
-:doc:`installation` for information on how to configure these.
-
-You may also need to install *tox*. If so, do this now:
+functional database and some Python requirements like *tox*. These can be
+installed using :command:`pip`:
 
 .. code-block:: shell
 
-   $ pip install --user tox
-
-.. tip::
-
-   If you're using Docker, you may not need to install *tox*
-   locally. Instead, it will already be installed inside the
-   container. For Docker, you can run *tox* like so:
-
-   .. code-block:: shell
-
-  $ docker-compose run --rm web tox [ARGS...]
-
-   For more information, refer to :ref:`installation-docker`.
+   $ pip install --user tox tox-docker
 
-Assuming these requirements are met, actually testing Patchwork is quite easy
-to do. To start, you can show the default targets like so:
+Once installed, actually testing Patchwork is quite easy to do. To start, you
+can show the default targets like so:
 
 .. code-block:: shell
 
@@ -66,7 +52,7 @@ parameter:
 
 .. code-block:: shell
 
-   $ tox -e py27-django18
+   $ tox -e py36-django21-mysql
 
 In the case of the unit tests targets, you can also run specific tests by
 passing the fully qualified test name as an additional argument to this
@@ -74,7 +60,7 @@ command:
 
 .. code-block:: shell
 
-   $ tox -e py27-django18 patchwork.tests.SubjectCleanUpTest
+   $ tox -e py36-django21-mysql patchwork.tests.SubjectCleanUpTest
 
 Because Patchwork support multiple versions of Django, it's very important that
 you test against all supported versions. When run without argument, tox will do
diff --git a/docs/development/installation.rst 
b/docs/development/installation.rst
index 0ab755f4..433c3a41 100644
--- a/docs/development/installation.rst
+++ b/docs/development/installation.rst
@@ -86,12 +86,6 @@ To run unit tests against the system Python packages, run:
 
$ docker-compose run --rm web python manage.py test
 
-To run unit tests for multiple versions using ``tox``, run:
-
-.. code-block:: shell
-
-   $ docker-compose run --rm web tox
-
 To reset the database before any of these commands, add ``--reset`` to the
 command line after ``web`` and before any other arguments:
 
diff --git a/patchwork/settings/dev.py b/patchwork/settings/dev.py
index 53fa58f6..400a8d70 100644
--- a/patchwork/settings/dev.py
+++ b/patchwork/settings/dev.py
@@ -19,6 +19,18 @@ try:
 except ImportError:
 debug_toolbar = None
 
+#
+# tox-docker settings
+#
+
+if 'POSTGRES_5432_TCP' in os.environ:
+os.environ['PW_TEST_DB_HOST'] = 'localhost'
+os.environ['PW_TEST_DB_PORT'] = os.environ['POSTGRES_5432_TCP']
+elif 'MYSQL_3306_TCP' in os.environ:
+os.environ['PW_TEST_DB_HOST'] = 'localhost'
+os.environ['PW_TEST_DB_PORT'] = os.environ['MYSQL_3306_TCP']
+
+
 #
 # Core settings
 # https://docs.djangoproject.com/en/1.11/ref/settings/#core-settings
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 563e24e2..0efdf886 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -5,4 +5,6 @@ django-filter==2.1.0; python_version >= '3.4'
 django-filter==1.1.0; python_version < '3.0'  # pyup: ignore
 django-debug-toolbar==1.11
 django-dbbackup==3.2.0
+psycopg2-binary==2.8.2
+mysqlclient==1.3.14
 -r requirements-test.txt
diff --git a/requirements-test.txt b/requirements-test.txt
index 0ad3c8aa..b05bc15b 100644
--- a/requirements-test.txt
+++ b/requirements-test.txt
@@ -1,5 +1,3 @@
-mysqlclient==1.3.14
-psycopg2-binary==2.8.2
 sqlparse==0.3.0
 python-dateutil==2.8.0
 openapi-core==0.8.0
diff --git a/tox.ini b/tox.ini
index 78b57f78..ba5d5104 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,6 +1,6 @@
 [tox]
 minversion = 2.0
-envlist = pep8,docs,py{27,34}-django111,py{35,36}-django{111,20,21}
+envlist = 
pep8,docs,py{27,34}-django111-{mysql,postgres},py{35,36}-django{111,20,21}-{mysql,postgres}
 skipsdist = True
 
 [testenv]
@@ -14,6 +14,11 @@ deps =
 django21: django>=2.1,<2.2
 

[PATCH 3/4] requirements: Switch to PyMySQL

[Stephen Finucane]

This is pure Python, which means there's no need for those pesky MySQL
development libs.

Signed-off-by: Stephen Finucane 
---
 manage.py| 12 ++--
 requirements-dev.txt |  2 +-
 tox.ini  |  2 +-
 3 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/manage.py b/manage.py
index fe6189ee..86bd67e9 100755
--- a/manage.py
+++ b/manage.py
@@ -2,9 +2,17 @@
 import os
 import sys
 
+try:
+# From https://github.com/PyMySQL/PyMySQL/wiki/WhyPyMySQL
+import pymysql
+pymysql.install_as_MySQLdb()
+except ImportError:
+pass
+
+
 if __name__ == "__main__":
-os.environ.setdefault("DJANGO_SETTINGS_MODULE",
-"patchwork.settings.production")
+os.environ.setdefault(
+"DJANGO_SETTINGS_MODULE", "patchwork.settings.production")
 
 from django.core.management import execute_from_command_line
 
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 0efdf886..06eefd99 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -6,5 +6,5 @@ django-filter==1.1.0; python_version < '3.0'  # pyup: ignore
 django-debug-toolbar==1.11
 django-dbbackup==3.2.0
 psycopg2-binary==2.8.2
-mysqlclient==1.3.14
+PyMySQL==0.9.3
 -r requirements-test.txt
diff --git a/tox.ini b/tox.ini
index ba5d5104..86fb62e1 100644
--- a/tox.ini
+++ b/tox.ini
@@ -15,7 +15,7 @@ deps =
 django{20,21}: djangorestframework>=3.7,<3.10
 django{20,21}: django-filter>=2.0,<3.0
 postgres: psycopg2-binary==2.7.7
-mysql: mysqlclient==1.3.14
+mysql: PyMySQL==0.9.3
 docker =
 postgres: postgres:9.6
 mysql: mysql:5.7
-- 
2.20.1

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH 1/4] forms: Don't attempt to evaluate State at startup

[Stephen Finucane]

As was designed, starting the interpreter would cause the State model
and its entries to be evaluated. This was an issue if, for example, the
model had been modified and you were attempting to apply the migration.

  Traceback (most recent call last):
File "manage.py", line 11, in 
  execute_from_command_line(sys.argv)
...
File "/usr/local/lib/python2.7/dist-packages/django/forms/models.py", line 
1199, in _set_queryset
  self.widget.choices = self.choices
File "/home/patchwork/patchwork/patchwork/forms.py", line 157, in 
_get_choices
  super(OptionalModelChoiceField, self)._get_choices())
File "/usr/local/lib/python2.7/dist-packages/django/forms/models.py", line 
1143, in __len__
  return (len(self.queryset) + (1 if self.field.empty_label is not None 
else 0))
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", 
line 232, in __len__
  self._fetch_all()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", 
line 1118, in _fetch_all
  self._result_cache = list(self._iterable_class(self))
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", 
line 53, in __iter__
  results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File 
"/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 
899, in execute_sql
  raise original_exception
  django.db.utils.OperationalError: (1054, "Unknown column 
'patchwork_state.slug' in 'field list'")

Resolve this by moving the evaluation into '__init__', meaning it will
only occur when a new form is created.

Signed-off-by: Stephen Finucane 
---
 patchwork/forms.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/patchwork/forms.py b/patchwork/forms.py
index 5d4c920a..5690eb01 100644
--- a/patchwork/forms.py
+++ b/patchwork/forms.py
@@ -165,7 +165,6 @@ class OptionalBooleanField(forms.TypedChoiceField):
 
 class MultiplePatchForm(forms.Form):
 action = 'update'
-state = OptionalModelChoiceField(queryset=State.objects.all())
 archived = OptionalBooleanField(
 choices=[('*', 'no change'), ('True', 'Archived'),
  ('False', 'Unarchived')],
@@ -176,6 +175,8 @@ class MultiplePatchForm(forms.Form):
 super(MultiplePatchForm, self).__init__(*args, **kwargs)
 self.fields['delegate'] = OptionalModelChoiceField(
 queryset=_get_delegate_qs(project=project), required=False)
+self.fields['state'] = OptionalModelChoiceField(
+queryset=State.objects.all())
 
 def save(self, instance, commit=True):
 opts = instance.__class__._meta
-- 
2.20.1

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH 0/4] Integrate tox-docker

[Stephen Finucane]

Testing Patchwork is still way more difficult than it should be. Docker
has massively simplified things but we're still unable to do things like
run 'tox -e py27' like we can most other Python projects. Take another
step towards this by integrating 'tox-docker' so we can do e.g. 'tox -e
py27-django11-mysql'

Stephen Finucane (4):
  forms: Don't attempt to evaluate State at startup
  tox: Integrate tox-docker
  requirements: Switch to PyMySQL
  docker-compose: Massively simplify

 docs/development/contributing.rst | 28 ++---
 docs/development/installation.rst |  6 -
 manage.py | 12 +++--
 patchwork/forms.py|  3 ++-
 patchwork/settings/dev.py | 12 +
 requirements-dev.txt  |  2 ++
 requirements-test.txt |  2 --
 tools/docker/Dockerfile   | 22 ++--
 tools/docker/entrypoint.sh|  4 +--
 tools/docker/trusty-ports.list|  3 ---
 tools/docker/trusty.list  |  3 ---
 tools/docker/xenial-ports.list|  3 ---
 tools/docker/xenial.list  |  3 ---
 tox.ini   | 42 +--
 14 files changed, 77 insertions(+), 68 deletions(-)
 delete mode 100644 tools/docker/trusty-ports.list
 delete mode 100644 tools/docker/trusty.list
 delete mode 100644 tools/docker/xenial-ports.list
 delete mode 100644 tools/docker/xenial.list

-- 
2.20.1

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] docker: Install libpq-dev to fix psycopg2-binary build

[Stephen Finucane]

On Fri, 2019-05-03 at 16:14 +1000, Daniel Axtens wrote:
> Andrew Donnellan  writes:
> 
> > On 1/5/19 2:35 pm, Russell Currey wrote:
> > > psycopg2-binary fails if pg_config isn't installed, which is provided by
> > > libpq-dev.
> > > 
> > > This seems strange to me since psycopg2-binary suggests that
> > > you use psycopg2-binary instead (of itself) if you don't want to build
> > > psycopg2 so you wouldn't need pg_config, which is very confusing.
> > > 
> > > It's possible that psycopg2-binary only needs to compile itself on
> > > non-x86 platforms, since I hit this on ppc64le.
> > > 
> > > Anyway, it works when this is added.
> > 
> > I don't think there is a binary psycopg2-binary build for anything other 
> > than i686/x86_64?
> 
> Right, yeah that would probably explain it. Given that libpq-dev just
> lives in the docker image, I don't think there's any harm here other
> than a slightly larger download when doing your first image build. I
> think we can live with that for cross-platform support.
> 
> Applied.

Given that we've done this, we should probably switch back to the non-
binary package since this isn't necessary now.

Stephen

> Daniel
> 
> 
> > Reviewed-by: Andrew Donnellan 
> > 
> > > Signed-off-by: Russell Currey 
> > > ---
> > >   tools/docker/Dockerfile | 2 +-
> > >   1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile
> > > index b9ecdb5..eef40e4 100644
> > > --- a/tools/docker/Dockerfile
> > > +++ b/tools/docker/Dockerfile
> > > @@ -26,7 +26,7 @@ RUN apt-get update -qq && \
> > >   python3.5-dev python3-pip python3-setuptools python3-wheel \
> > >   python3.4-dev findutils=4.4.2-7 python3.6-dev \
> > >   libmysqlclient-dev mysql-client curl unzip build-essential \
> > > -git postgresql-client tzdata
> > > +git postgresql-client tzdata libpq-dev
> > >   
> > >   # User
> > >   RUN useradd --uid=$UID --create-home patchwork
> > > 
> > 
> > -- 
> > Andrew Donnellan  OzLabs, ADL Canberra
> > andrew.donnel...@au1.ibm.com  IBM Australia Limited
> > 
> > ___
> > Patchwork mailing list
> > Patchwork@lists.ozlabs.org
> > https://lists.ozlabs.org/listinfo/patchwork
> ___
> Patchwork mailing list
> Patchwork@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/patchwork


___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 07/10] Add support for Django REST Framework 3.7, 3.8

[Stephen Finucane]

On Fri, 2019-05-03 at 17:01 +1000, Daniel Axtens wrote:
> Daniel Axtens  writes:
> 
> > Stephen Finucane  writes:
> > 
> > > On Tue, 2019-04-30 at 16:03 +1000, Daniel Axtens wrote:
> > > > From: Stephen Finucane 
> > > > 
> > > > No breaking changes that concern us here.
> > > > 
> > > > Signed-off-by: Stephen Finucane 
> > > > Reviewed-by: Daniel Axtens 
> > > > Signed-off-by: Daniel Axtens 
> > > > (cherry picked from commit a27f36fb784867e434527b8e65064ae3bdb12c82)
> > > > Signed-off-by: Daniel Axtens 
> > > 
> > > I'm conflicted on this. This introduces a change in requirements which
> > > is not something we should typically do on a stable branch. The only
> > > thing that could save us is that we don't *drop* support for something
> > > (I'd be a hard -1 if we did), but is there any reason we couldn't just
> > > cut a 2.2 release instead?
> > 
> > So:
> > 
> >  - Production systems may legitimately want to run with the most recent
> >minor version of dependencies, because they often contain security
> >fixes.
> > 
> >  - Production systems may also want to stick to stable versions of PW
> >while running recent dependencies, especially since a 2.1->2.2
> >upgrade will require a schema migration and downtime.
> > 
> >  - Therefore, stable versions of PW should support the most recent
> >versions of our dependencies.
> > 
> > In this case, the only patch we _really_ need is the django-filters one,
> > as there's a change between 1.0 and 1.1 that we need support for. The
> > DRF changes just allow us to use tox and friends to test that we support
> > this usecase.
> > 
> > If you wanted, I'd be happy to only pick the changes to tox.ini and drop
> > the changes to requirements-* - then I can use tox to satisfy myself
> > that the OzLabs setup is likely to work but we don't change the default
> > versions.
> > 
> 
> I've adopted this approach (only change tox.ini), applied the patches,
> and tagged v2.1.2.

Yup, that's better. Thanks.

Stephen

> Regards,
> Daniel
> 
> > We also should cut 2.2 soon, if for no other reason than to get the db
> > speedups, but I don't think it provides a solution for this usecase.
> > 
> > Regards,
> > Daniel
> > 
> > > Stephen
> > > 
> > > > ---
> > > >  README.rst  | 2 +-
> > > >  .../notes/django-rest-framework-3-7-bc6ad5df8bc54afc.yaml   | 6 ++
> > > >  .../notes/django-rest-framework-3-8-23865db833b4d188.yaml   | 6 ++
> > > >  requirements-dev.txt| 2 +-
> > > >  requirements-prod.txt   | 2 +-
> > > >  tox.ini | 5 +++--
> > > >  6 files changed, 18 insertions(+), 5 deletions(-)
> > > >  create mode 100644 
> > > > releasenotes/notes/django-rest-framework-3-7-bc6ad5df8bc54afc.yaml
> > > >  create mode 100644 
> > > > releasenotes/notes/django-rest-framework-3-8-23865db833b4d188.yaml
> > > > 
> > > > diff --git a/README.rst b/README.rst
> > > > index 4a970ec5a9ff..ddc4b98125d1 100644
> > > > --- a/README.rst
> > > > +++ b/README.rst
> > > > @@ -45,7 +45,7 @@ Requirements
> > > >  
> > > >  - Django (1.8 - 1.11)
> > > >  
> > > > -- Django REST Framework (3.2 - 3.6)
> > > > +- Django REST Framework (3.4 - 3.8)
> > > >  
> > > >  - Django Filters (1.0)
> > > >  
> > > > diff --git 
> > > > a/releasenotes/notes/django-rest-framework-3-7-bc6ad5df8bc54afc.yaml 
> > > > b/releasenotes/notes/django-rest-framework-3-7-bc6ad5df8bc54afc.yaml
> > > > new file mode 100644
> > > > index ..4bf92c999df5
> > > > --- /dev/null
> > > > +++ b/releasenotes/notes/django-rest-framework-3-7-bc6ad5df8bc54afc.yaml
> > > > @@ -0,0 +1,6 @@
> > > > +---
> > > > +upgrade:
> > > > +  - |
> > > > +`Django REST Framework 3.7
> > > > +
> > > > `_ is 
> > > > now
> > > > +supported.
> > > > diff --git 
> > > > a/releasenotes/notes/django-rest-framework-3-8-23865db833b4d188.yaml 
> > > > b/releasenotes/notes/django-rest-framework-3-8-23865db833b4d188.yaml
> > > > new file mode 100644
> > > > index ..dc2d2c8f8bfa
> > > > --- /dev/null
> > > > +++ b/releasenotes/notes/django-rest-framework-3-8-23865db833b4d188.yaml
> > > > @@ -0,0 +1,6 @@
> > > > +---
> > > > +upgrade:
> > > > +  - |
> > > > +`Django REST Framework 3.8
> > > > +
> > > > `_ is 
> > > > now
> > > > +supported.
> > > > diff --git a/requirements-dev.txt b/requirements-dev.txt
> > > > index 7a8fdb9c8851..b12246dd1b89 100644
> > > > --- a/requirements-dev.txt
> > > > +++ b/requirements-dev.txt
> > > > @@ -1,4 +1,4 @@
> > > >  Django>=1.8,<2.0
> > > > -djangorestframework>=3.4,<3.7
> > > > +djangorestframework>=3.4,<3.9
> > > >  django-filter>=1.0,<1.1
> > > >  -r requirements-test.txt
> > > > diff --git a/requirements-prod.txt b/requirements-prod.txt
> > > > index 

[RFC PATCH] docker: Add support for using eatmydata in the database

[Russell Currey]

When running tox on a VM with presumably pretty busy spinning disks,
using eatmydata with the database took running one configuration's test
suite from (no exaggeration) 20 minutes down to 60 seconds.

It makes a huge difference to test speed, so we should make it easily
available for developers.  The primary motivation here was to
automatically test each patch in a timeframe that isn't insane.

Open to ideas on how to organise this, whether we do it for MySQL too
(which we probably should), whether the base directory should have these
files in it, what to call the Dockerfile, etc.  I think it's a good
thing to have in the repo, though.

Signed-off-by: Russell Currey 
---
 docker-compose-eatmydata.yml  | 32 +++
 tools/docker/Dockerfile.eatmydata |  9 +
 2 files changed, 41 insertions(+)
 create mode 100644 docker-compose-eatmydata.yml
 create mode 100644 tools/docker/Dockerfile.eatmydata

diff --git a/docker-compose-eatmydata.yml b/docker-compose-eatmydata.yml
new file mode 100644
index 000..27d1604
--- /dev/null
+++ b/docker-compose-eatmydata.yml
@@ -0,0 +1,32 @@
+version: "3"
+services:
+  db:
+build:
+  context: .
+  dockerfile: ./tools/docker/Dockerfile.eatmydata
+volumes:
+  - ./tools/docker/db/postdata:/var/lib/postgresql/data
+environment:
+  - POSTGRES_PASSWORD=password
+
+  web:
+build:
+  context: .
+  dockerfile: ./tools/docker/Dockerfile
+  args:
+- UID
+depends_on:
+  - db
+command: python3 manage.py runserver 0.0.0.0:8000
+volumes:
+  - .:/home/patchwork/patchwork/
+ports:
+  - "8000:8000"
+environment:
+  - UID
+  - PW_TEST_DB_HOST=db
+  - PW_TEST_DB_PORT=5432
+  - PW_TEST_DB_TYPE=postgres
+  - PW_TEST_DB_USER=postgres
+  - PW_TEST_DB_PASS=password
+  - PGPASSWORD=password
diff --git a/tools/docker/Dockerfile.eatmydata 
b/tools/docker/Dockerfile.eatmydata
new file mode 100644
index 000..693cbb3
--- /dev/null
+++ b/tools/docker/Dockerfile.eatmydata
@@ -0,0 +1,9 @@
+FROM postgres:9.6
+
+RUN apt-get update \
+ && apt-get install -y eatmydata \
+ && apt-get autoremove -y \
+ && rm -rf /var/lib/apt/lists/*
+
+ENTRYPOINT [ "/usr/bin/eatmydata", "/usr/local/bin/docker-entrypoint.sh" ]
+CMD ["postgres"]
-- 
2.21.0

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 07/10] Add support for Django REST Framework 3.7, 3.8

[Daniel Axtens]

Daniel Axtens  writes:

> Stephen Finucane  writes:
>
>> On Tue, 2019-04-30 at 16:03 +1000, Daniel Axtens wrote:
>>> From: Stephen Finucane 
>>> 
>>> No breaking changes that concern us here.
>>> 
>>> Signed-off-by: Stephen Finucane 
>>> Reviewed-by: Daniel Axtens 
>>> Signed-off-by: Daniel Axtens 
>>> (cherry picked from commit a27f36fb784867e434527b8e65064ae3bdb12c82)
>>> Signed-off-by: Daniel Axtens 
>>
>> I'm conflicted on this. This introduces a change in requirements which
>> is not something we should typically do on a stable branch. The only
>> thing that could save us is that we don't *drop* support for something
>> (I'd be a hard -1 if we did), but is there any reason we couldn't just
>> cut a 2.2 release instead?
>
> So:
>
>  - Production systems may legitimately want to run with the most recent
>minor version of dependencies, because they often contain security
>fixes.
>
>  - Production systems may also want to stick to stable versions of PW
>while running recent dependencies, especially since a 2.1->2.2
>upgrade will require a schema migration and downtime.
>
>  - Therefore, stable versions of PW should support the most recent
>versions of our dependencies.
>
> In this case, the only patch we _really_ need is the django-filters one,
> as there's a change between 1.0 and 1.1 that we need support for. The
> DRF changes just allow us to use tox and friends to test that we support
> this usecase.
>
> If you wanted, I'd be happy to only pick the changes to tox.ini and drop
> the changes to requirements-* - then I can use tox to satisfy myself
> that the OzLabs setup is likely to work but we don't change the default
> versions.
>

I've adopted this approach (only change tox.ini), applied the patches,
and tagged v2.1.2.

Regards,
Daniel

> We also should cut 2.2 soon, if for no other reason than to get the db
> speedups, but I don't think it provides a solution for this usecase.
>
> Regards,
> Daniel
>
>>
>> Stephen
>>
>>> ---
>>>  README.rst  | 2 +-
>>>  .../notes/django-rest-framework-3-7-bc6ad5df8bc54afc.yaml   | 6 ++
>>>  .../notes/django-rest-framework-3-8-23865db833b4d188.yaml   | 6 ++
>>>  requirements-dev.txt| 2 +-
>>>  requirements-prod.txt   | 2 +-
>>>  tox.ini | 5 +++--
>>>  6 files changed, 18 insertions(+), 5 deletions(-)
>>>  create mode 100644 
>>> releasenotes/notes/django-rest-framework-3-7-bc6ad5df8bc54afc.yaml
>>>  create mode 100644 
>>> releasenotes/notes/django-rest-framework-3-8-23865db833b4d188.yaml
>>> 
>>> diff --git a/README.rst b/README.rst
>>> index 4a970ec5a9ff..ddc4b98125d1 100644
>>> --- a/README.rst
>>> +++ b/README.rst
>>> @@ -45,7 +45,7 @@ Requirements
>>>  
>>>  - Django (1.8 - 1.11)
>>>  
>>> -- Django REST Framework (3.2 - 3.6)
>>> +- Django REST Framework (3.4 - 3.8)
>>>  
>>>  - Django Filters (1.0)
>>>  
>>> diff --git 
>>> a/releasenotes/notes/django-rest-framework-3-7-bc6ad5df8bc54afc.yaml 
>>> b/releasenotes/notes/django-rest-framework-3-7-bc6ad5df8bc54afc.yaml
>>> new file mode 100644
>>> index ..4bf92c999df5
>>> --- /dev/null
>>> +++ b/releasenotes/notes/django-rest-framework-3-7-bc6ad5df8bc54afc.yaml
>>> @@ -0,0 +1,6 @@
>>> +---
>>> +upgrade:
>>> +  - |
>>> +`Django REST Framework 3.7
>>> +`_ is 
>>> now
>>> +supported.
>>> diff --git 
>>> a/releasenotes/notes/django-rest-framework-3-8-23865db833b4d188.yaml 
>>> b/releasenotes/notes/django-rest-framework-3-8-23865db833b4d188.yaml
>>> new file mode 100644
>>> index ..dc2d2c8f8bfa
>>> --- /dev/null
>>> +++ b/releasenotes/notes/django-rest-framework-3-8-23865db833b4d188.yaml
>>> @@ -0,0 +1,6 @@
>>> +---
>>> +upgrade:
>>> +  - |
>>> +`Django REST Framework 3.8
>>> +`_ is 
>>> now
>>> +supported.
>>> diff --git a/requirements-dev.txt b/requirements-dev.txt
>>> index 7a8fdb9c8851..b12246dd1b89 100644
>>> --- a/requirements-dev.txt
>>> +++ b/requirements-dev.txt
>>> @@ -1,4 +1,4 @@
>>>  Django>=1.8,<2.0
>>> -djangorestframework>=3.4,<3.7
>>> +djangorestframework>=3.4,<3.9
>>>  django-filter>=1.0,<1.1
>>>  -r requirements-test.txt
>>> diff --git a/requirements-prod.txt b/requirements-prod.txt
>>> index d249ad84c10c..42ff8ecd82f1 100644
>>> --- a/requirements-prod.txt
>>> +++ b/requirements-prod.txt
>>> @@ -1,5 +1,5 @@
>>>  Django>=1.8,<2.0
>>> -djangorestframework>=3.4,<3.7
>>> +djangorestframework>=3.4,<3.9
>>>  django-filter>=1.0,<1.1
>>>  psycopg2>=2.7,<2.8
>>>  sqlparse==0.2.4
>>> diff --git a/tox.ini b/tox.ini
>>> index 327b9e7c14a8..5c741d374a64 100644
>>> --- a/tox.ini
>>> +++ b/tox.ini
>>> @@ -10,8 +10,9 @@ deps =
>>>  django19: django>=1.9,<1.10
>>>  django110: django>=1.10,<1.11
>>>  django111: django>=1.11,<2.0

Patchwork v2.1.2 Available

[Daniel Axtens]

We're pleased to announce the release of Patchwork v2.1.2. This release
is part of the "Eolienne" release series:

https://github.com/getpatchwork/patchwork/releases/tag/v2.1.2

This release is a PATCH release that focuses on bugfixes. For more
details, please see below.

Happy patchworking!

---

Changes in Patchwork v2.1.1..v2.1.2
---

bc4c011e8947 Release 2.1.2
9b4d7a2cf4b8 Add support for Django REST Framework 3.9
4a4862f6d187 Add support for django-filter 1.1
b99531224fcb Add support for Django REST Framework 3.7, 3.8
233e6849f636 REST: A check must specify a state
d963655ccd89 REST: Handle regular form data requests for checks
b7aa6f1f62c5 REST: Handle JSON requests
ea9301b76d0f notifications: fix notification expiry when no user is associated
ca544ba9e197 REST: Add new setting for maximum API page size
641b12321690 REST: Check.user is not read-only
0c60d688d0ac parser: recognise git commit consisting only of empty new file
b00cb3cb8a67 tests: Add tests for 'series-completed' event
aa6f4f0385a3 parser: Ensure whitespace is stripped for long headers
413bd043df90 models: Ensure UserProfile.user is configured
3a98b6b10ae8 REST: Fix typo in embedded serializers
62b0f0406895 REST: Ensure submission exists for comment listing
63165ec62db3 REST: Ensure patch exists for check creation
85c31c5a3222 REST: Show 'web_url' in embedded series responses
4f1dc4febf7b Don't passthrough 'Content-Type: multipart/signed' header
7d0b1ddef172 Resolve Python 3.6 warnings
877a52413167 tox: Disable W504 ("line break after binary operator")
32248ce9c4ac Post-release version bump
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 1/2] docker: Use Ubuntu ports repositories on non-x86 architectures

[Daniel Axtens]

Russell Currey  writes:

> This should allow Patchwork to run "out of the box" in Docker on any
> architecture with a) an Ubuntu port and b) support in the Postgres
> multiarch Docker image, which includes at least arm64 and ppc64le.
>
> It's a little gross hacking the Dockerfile like this, but I'm not sure
> there's a more elegant way to do it.  Unfortunately it doesn't seem like
> there's any way to do conditional COPY, and anything in RUN is plain
> /bin/sh, so that's why it looks like it does.
>
> Tested on ppc64le and on x86_64.
>
> Signed-off-by: Russell Currey 
> ---
>  tools/docker/Dockerfile| 14 --
>  tools/docker/trusty-ports.list |  3 +++
>  tools/docker/xenial-ports.list |  3 +++
>  3 files changed, 18 insertions(+), 2 deletions(-)
>  create mode 100644 tools/docker/trusty-ports.list
>  create mode 100644 tools/docker/xenial-ports.list
>
> diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile
> index eef40e4..76bb6b2 100644
> --- a/tools/docker/Dockerfile
> +++ b/tools/docker/Dockerfile
> @@ -15,11 +15,21 @@ ENV DJANGO_SETTINGS_MODULE patchwork.settings.dev
>  ENV DEBIAN_FRONTEND noninteractive
>  ENV PYTHONUNBUFFERED 1
>  
> +
>  # System
>  # trusty and findutils is for python3.4; xenial is for python3.5
>  # TODO(stephenfin): Are curl, unzip required?
> -COPY tools/docker/trusty.list /etc/apt/sources.list.d/trusty.list
> -COPY tools/docker/xenial.list /etc/apt/sources.list.d/xenial.list
> +COPY tools/docker/*.list /etc/apt/sources.list.d/
> +
> +RUN cd /etc/apt/sources.list.d; \
> +echo $(uname -m) > /tmp/arch; \
> +if [ $(cat /tmp/arch) != 'x86_64' ] && grep -q -v "i.86" /tmp/arch; then 
> \
> +mv trusty-ports.list trusty.list; \
> +mv xenial-ports.list xenial.list; \
> +else \
> +rm *-ports.list; \
> +fi
> +

Yeah, this is pretty gross, but oh well. I'm not in general super
bothered by gross Dockerfiles, especially where it keeps mess out of the
code proper.

Applied the series, thanks for updating the docs in the next patch.

As an aside, I wonder when we can drop py34 - I don't think django2
supports it so hopefully soon...

Regards,
Daniel


>  RUN apt-get update -qq && \
>  apt-get install -y --no-install-recommends --allow-downgrades \
>  python-dev python-pip python-setuptools python-wheel \
> diff --git a/tools/docker/trusty-ports.list b/tools/docker/trusty-ports.list
> new file mode 100644
> index 000..ebcf4fa
> --- /dev/null
> +++ b/tools/docker/trusty-ports.list
> @@ -0,0 +1,3 @@
> +deb http://ports.ubuntu.com/ubuntu-ports/ trusty main
> +deb http://ports.ubuntu.com/ubuntu-ports/ trusty-updates main
> +deb http://ports.ubuntu.com/ubuntu-ports/ trusty-security main
> diff --git a/tools/docker/xenial-ports.list b/tools/docker/xenial-ports.list
> new file mode 100644
> index 000..d84641f
> --- /dev/null
> +++ b/tools/docker/xenial-ports.list
> @@ -0,0 +1,3 @@
> +deb http://ports.ubuntu.com/ubuntu-ports/ xenial main
> +deb http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main
> +deb http://ports.ubuntu.com/ubuntu-ports/ xenial-security main
> -- 
> 2.21.0
>
> ___
> Patchwork mailing list
> Patchwork@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/patchwork
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] README: add .env file to installation instructions

[Daniel Axtens]

Sounds good, applied.

Russell Currey  writes:

> Creating the .env file is mentioned in the installation documentation
> but not in the README, so following only the steps mentioned there will
> fail.  Add this and add a `cd patchwork` in there for good measure so
> you could straight up copy paste the steps.
>
> Signed-off-by: Russell Currey 
> ---
>  README.rst | 9 +++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/README.rst b/README.rst
> index 38c1847..b45c3e6 100644
> --- a/README.rst
> +++ b/README.rst
> @@ -61,11 +61,16 @@ environment. To install Patchwork:
>  
> $ git clone https://github.com/getpatchwork/patchwork.git
>  
> -3. Build the images. This will download over 200MB from the internet::
> +3. Create a ``.env`` file in the root directory of the project and store your
> +   ``UID`` attribute there::
> +
> +   $ cd patchwork && echo "UID=$UID" > .env
> +
> +4. Build the images. This will download over 200MB from the internet::
>  
> $ docker-compose build
>  
> -4. Run `docker-compose up`::
> +5. Run `docker-compose up`::
>  
> $ docker-compose up
>  
> -- 
> 2.21.0
>
> ___
> Patchwork mailing list
> Patchwork@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/patchwork
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork