I have updated the Docker image again. I actually had to embed the knowledge of
dealing with a wheel directory into the image as there is no way for a user to
override environment variables during the build phase except by not using the
onbuild image and creating a manual one again, but want to avoid that for
various reasons.
With the very latest images what you can now do is.
1. Build the image.
$ time docker build -t myapp .Sending build context to Docker daemon 574 kB
Sending build context to Docker daemon
Step 0 : FROM grahamdumpleton/mod-wsgi-docker:python-2.7-onbuild
# Executing 2 build triggers
Trigger 0, COPY . /app
Step 0 : COPY . /app
Trigger 1, RUN mod_wsgi-docker-build
Step 0 : RUN mod_wsgi-docker-build
---> Running in fa3adb991c09
-----> Installing dependencies with pip
Collecting Django<1.8 (from -r requirements.txt (line 1))
Downloading Django-1.7.8-py2.py3-none-any.whl (7.4MB)
Installing collected packages: Django
Successfully installed Django-1.7.8
-----> Running .whiskey/action_hooks/build
….
61 static files copied to '/app/example/htdocs'.
---> c72e831182e0
Removing intermediate container cc5bdce4213b
Removing intermediate container fa3adb991c09
Step 1 : CMD --working-directory example --url-alias /static example/htdocs
--application-type module example.wsgi
---> Running in 0e70fe87ecc8
---> 5da3bdf7d5e1
Removing intermediate container 0e70fe87ecc8
Successfully built 5da3bdf7d5e1
real 0m15.185s
user 0m0.181s
sys 0m0.014s
So this is like what you would normally be doing now.
2. Run the image but don't start the WSGI server, just enter into the shell by
setting the entry point. Using a special shell command here so pick up same
environment as will exist when deploying. Also mount the current directory
source as well so it actually overrides what is in the image. As it is a built
image you can use 'pip freeze' to see that packages were installed.
$ docker run -it --rm --entrypoint mod_wsgi-docker-shell -v `pwd`:/app myapp
Django==1.7.8
mod-wsgi==4.4.13
virtualenv==13.0.3
wheel==0.24.0
3. We are now going to create wheels for the packages that we want installed
each time. In this case what is in the requirements.txt file. We will store the
wheels in the .whiskey/wheelhouse directory. You might have to create the
.whiskey directory if it doesn't already exist.
root@095b6ad1ff25:/app# pip wheel --wheel-dir .whiskey/wheelhouse -r
requirements.txt
Collecting Django<1.8 (from -r requirements.txt (line 1))
Downloading Django-1.7.8-py2.py3-none-any.whl (7.4MB)
100% |████████████████████████████████| 7.4MB 60kB/s
Saved ./.whiskey/wheelhouse/Django-1.7.8-py2.py3-none-any.whl
Skipping Django, due to already being wheel.
root@095b6ad1ff25:/app# ls -las .whiskey/wheelhouse/
total 7252
0 drwxr-xr-x 1 1000 staff 102 Jun 20 06:12 .
0 drwxr-xr-x 1 1000 staff 136 Jun 20 06:12 ..
7252 -rw-r--r-- 1 1000 staff 7423353 Jun 20 06:12
Django-1.7.8-py2.py3-none-any.whl
root@095b6ad1ff25:/app# exit
Because we mounted the original source directory, this has actually written it
back to your host. Once done we can exit.
>From the host you can see that files are now present.
$ ls -las .whiskey/wheelhouse/
total 14504
0 drwxr-xr-x 3 graham staff 102 20 Jun 16:12 .
0 drwxr-xr-x 4 graham staff 136 20 Jun 16:12 ..
14504 -rw-r--r-- 1 graham staff 7423353 20 Jun 16:12
Django-1.7.8-py2.py3-none-any.whl
4. Build the image again. This time it the wheelhouse directory will be copied
into the image and the wheel files there will be used to do pip installation.
$ time docker build -t myapp .
Sending build context to Docker daemon 7.998 MB
Sending build context to Docker daemon
Step 0 : FROM grahamdumpleton/mod-wsgi-docker:python-2.7-onbuild
# Executing 2 build triggers
Trigger 0, COPY . /app
Step 0 : COPY . /app
Trigger 1, RUN mod_wsgi-docker-build
Step 0 : RUN mod_wsgi-docker-build
---> Running in aed306a4db30
-----> Detected wheelhouse for pip
-----> Installing dependencies with pip
Collecting Django<1.8 (from -r requirements.txt (line 1))
Installing collected packages: Django
Successfully installed Django-1.7.8
-----> Running .whiskey/action_hooks/build
….
61 static files copied to '/app/example/htdocs'.
---> 9f4681e0d152
Removing intermediate container aed306a4db30
Removing intermediate container 1f9033a7887e
Step 1 : CMD --working-directory example --url-alias /static example/htdocs
--application-type module example.wsgi
---> Running in d56665f7d8be
---> 6d7ee644df22
Removing intermediate container d56665f7d8be
Successfully built 6d7ee644df22
real 0m9.264s
user 0m0.256s
sys 0m0.040s
For just Django, this time it is 6 seconds quicker. For packages which need
code compilation, would be a lot quicker.
So the general idea is that keep doing builds and will use all the wheels. If
you need to update wheels, change the requirements.txt file and run the shell
again to generate the wheels for new versions and then builds will pick them up
and make things quicker.
So the mounting of a volume is only done when you want to generate the wheels.
When wanting to build a final image for production release, you can remove the
wheelhouse directory if want to force everything to be rebuilt from scratch.
Also make sure wheelhouse directory is ignored in git repository so don't
accidentally check it in to repository.
Graham
On 20/06/2015, at 10:43 AM, Graham Dumpleton <[email protected]> wrote:
> The mod_wsgi package should be installed every time you build an image with
> that Dockerfile.
>
> Anyway, I have pushed up new docker images which install mod_wsgi as part of
> the base image now so even that will not happen. So pull down latest images
> and you should see that change.
>
> I am working on the recipe to explain the use of wheels for quicker builds.
> The volume only comes into play when building the wheels, not necessarily
> when building your application image, and definitely now when running it.
>
> Environment variables for pip are giving me a bit of a problem right now, so
> trying to sort that out.
>
> Graham
>
> On 19/06/2015, at 11:03 PM, Eibriel Inv <[email protected]> wrote:
>
>> Thanks!
>> This solution worked just fine for me, I wasn't using hooks.
>>
>> I think Volumes should be for data instead of code (but maybe I'm over
>> reacting).
>>
>> my Dockerfile:
>>
>> FROM grahamdumpleton/mod-wsgi-docker:python-3.4
>>
>> WORKDIR /app
>>
>> RUN pip install Flask
>> RUN pip install eve
>>
>> COPY . /app
>>
>> RUN mod_wsgi-docker-build
>>
>> EXPOSE 80
>> ENTRYPOINT [ "mod_wsgi-docker-start" ]
>>
>> CMD [ "--working-directory", "main_server", \
>> "--url-alias", "/main_server/static", "static", \
>> "main_server.wsgi" ]
>>
>> Now the script skips the requirements because are already installed, is only
>> installing modwsgi, and is fast enough to use it on development :)
>>
>>
>>
>> El jueves, 18 de junio de 2015, 21:42:42 (UTC-3), Graham Dumpleton escribió:
>> Unless I am confused by what you are asking, there isn't a virtual
>> environment.
>>
>> The images have a separate Python installation from any system Python
>> installation so it can ensure is latest version and when you are doing a pip
>> install it is installing into the separate Python installation, not a
>> virtual environment constructed from the Python installation.
>>
>> So the build scripts aren't setting up a virtual environment and that
>> therefore isn't the issue. The issue is more that the build scripts support
>> you being able to specify hooks to perform actions and set extra environment
>> variables dynamically, which need to be done before the pip install can be
>> run.
>>
>> If you aren't using the build hooks, then technically you could do a pip
>> install direct from the Docker file, however not when inheriting from the on
>> build image. Instead you would want to copy the intent of the on build into
>> your Dockerfile, actually dropping the on build and then insert your pip
>> installs.
>>
>> So instead use something like:
>>
>>
>> FROM grahamdumpleton/mod-wsgi-docker:python-2.7
>>
>> WORKDIR /app
>>
>> RUN pip install mypackage
>>
>> COPY . /app
>> RUN mod_wsgi-docker-build
>>
>> EXPOSE 80
>> ENTRYPOINT [ "mod_wsgi-docker-start" ]
>>
>> Any particular reason volumes would be an issue?
>>
>> FWIW, when I had some time I was going to play with wheel deploys was a
>> variant of some ideas described in:
>>
>> https://glyph.twistedmatrix.com/2015/03/docker-deploy-double-dutch.html
>>
>> That is, use one docker container once to build all the wheels in and then
>> the results would be copied into final image to speed up its build.
>>
>> Graham
>>
>> On 19/06/2015, at 10:23 AM, Eibriel Inv <[email protected]> wrote:
>>
>>> Hi Graham, thanks for your response.
>>>
>>> I want to use docker for Development to be sure I'm running under the same
>>> conditions that Production.
>>> I'm using "onbuild" images, you can see the code here:
>>> https://github.com/Eibriel/widu-community-server/blob/master/Dockerfile
>>>
>>> I've no problem rebuilding the container each time (avoiding Volumes if
>>> possible), if the build time is short enough.
>>>
>>> I'm wondering, Why the virtual environment is needed? Without it we will be
>>> able to just pip install on Dockerfile.
>>>
>>> Thanks!!
>>>
>>>
>>> El martes, 16 de junio de 2015, 3:09:32 (UTC-3), Graham Dumpleton escribió:
>>> I am just doing a final build and push of updated images which will permit
>>> the volume mounting trick.
>>>
>>> Besides that, there is one other thing one can do which is not to avoid
>>> running pip, but to use a manual step to first prebuild Python wheels for
>>> all the packages from inside Docker back into a local directory on the
>>> host. You then setup the build hook script to check for a cache directory
>>> of wheels and set an environment variable so pip knows where it is. When
>>> you want to build the image using the cache, you would mount as a volume
>>> the local cache directory with the wheels into the location which pip is
>>> being setup to grab them from.
>>>
>>> By using Python wheels you would speed things up significantly as it avoids
>>> needing to download packages from the network and recompile them if C
>>> extensions.
>>>
>>> I will work on a recipe for that, but if you can let me know the scenario
>>> you are targeting which is the problem do still let me know.
>>>
>>> Graham
>>>
>>>
>>> On 16 June 2015 at 11:07, Graham Dumpleton <[email protected]> wrote:
>>>
>>> On 16/06/2015, at 10:16 AM, Eibriel Inv <[email protected]> wrote:
>>>
>>> > Hi!
>>> > I've noticed that every time I make a change on the code I need to
>>> > rebuild the container and the build script install the pip requirements
>>> > (which is time consuming).
>>> >
>>> > How can I install my own requirements from Dockerfile to prevent that?
>>>
>>> Is the issue with this because you are trying to use the docker image
>>> during development and so are making fast turn around changes?
>>>
>>> The reason you are seeing what you are seeing is because the 'onbuild'
>>> docker image is written as:
>>>
>>> FROM grahamdumpleton/mod-wsgi-docker:python-2.7
>>>
>>> WORKDIR /app
>>>
>>> ONBUILD COPY . /app
>>> ONBUILD RUN mod_wsgi-docker-build
>>>
>>> EXPOSE 80
>>> ENTRYPOINT [ "mod_wsgi-docker-start" ]
>>>
>>> Because the requirements.txt file is a part of the directory that needs to
>>> be copied into the docker image, then any change to the project directory
>>> will result in the latter build phase being triggered. It is the build
>>> phase that pip is run from to install the requirements.
>>>
>>> This sort of arrangement is quite typical for Python docker images.
>>>
>>> If you weren't using this image with the mod_wsgi-docker-build script and
>>> doing things explicitly, you could probably get away with using something
>>> like:
>>>
>>> WORKDIR /app
>>>
>>> COPY requirements.txt /app/requirements.txt
>>>
>>> RUN pip install -r requirements.txt
>>>
>>> COPY . /app
>>>
>>> With this, because the requirements.txt file is copied separately, so long
>>> as that file along isn't changed, the pip wouldn't be run.
>>>
>>> The whole point with mod_wsgi-docker-build though is that it does other
>>> stuff before pip is run to properly set up the environment. This means it
>>> cannot be split out in that way.
>>>
>>> If the problem is because you are trying to use it for quick turnaround
>>> changes in a development environment, there may be a better way, although I
>>> would have to tweak how the images work.
>>>
>>> What one other person did, perhaps due to also wanting to do things in a
>>> development environment, was to copy my docker images and change them to
>>> have the application directory mounted in as a volume.
>>>
>>> They possibly weren't thinking of the pip install problem, but wanted to be
>>> able to modify code on the fly, touch the WSGI script file and have the
>>> mod_wsgi in the docker instance automatically reload it.
>>>
>>> In other words, wouldn't be necessary to stop the container.
>>>
>>> Rather than copy my docker images and change them, notionally you could
>>> achieve the same thing by doing:
>>>
>>> docker build -t myapp .
>>> docker run -it -v `pwd`:/app myapp
>>>
>>> That is, build the docker image once, which will result in all the packages
>>> being installed, and then run it, with it using the live code from your
>>> host. You would only need to do a rebuild if you change the
>>> requirements.txt file.
>>>
>>> Unfortunately, as written this will not work.
>>>
>>> The problem is that the docker images themselves stored various stuff
>>> inside of the /app directory and what is in it isn't just your application
>>> code. Thus the script to start everything up isn't found.
>>>
>>> What I could do though is change the images such that the special stuff
>>> from the base image isn't stored under /app where your application is
>>> placed, and puts it somewhere else.
>>>
>>> If I do that, then the mounting volume trick should work and allow for this
>>> sort of behaviour.
>>>
>>> So can you explain the scenario of where this is a problem so I can
>>> understand better the needs you have and whether this change would work?
>>>
>>> Thanks.
>>>
>>> Graham
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to [email protected].
>>> To post to this group, send email to [email protected].
>>> Visit this group at http://groups.google.com/group/modwsgi.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> Visit this group at http://groups.google.com/group/modwsgi.
>> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
