Hi,
I have some experience with a jupyterhub setup instantiating notebook services
in a swarm of docker-engines.
This operational setup uses:
* docker-engine 1.11.2 (API 1.23)
* swarm/1.2.5 (API 1.22)
As I understand things changed, and in recent docker-engine swarm
functionality is included ab ovo, but the philosophy is somewhat
different. (E.g services/tasks are introduced.)
I'd like to upgrade the software components but cannot figure out how to
properly configure everything.
In a sandbox environment I've been trying to setup a minimal scenario:
* 2 nodes are running ubuntu 16.04 in virtualbox, sharing network 10.0.0.0/24
(10.0.0.1 head, 10.0.0.2 slave1)
* both run docker-engine 1.13.0-rc2 (API 1.25)
* head plays the role of the swarm manager.
* in order to have networking across containers, an attachable overlay network
is created (hubnet 10.42.0.0/24)
* jupyterhub (version 0.7.0.b1) is installed from sources and is running in a
container in the head, connected also to hubnet (10.42.0.2).
* in this container DockerSpawner is downloaded (version 0.5.0)
* self-signed certificates and a single user is also created for testing
purposes
* the dockerspawner's default image jupyterhub/systemuser is pulled in both
nodes
The idea is that jupyterhub (running in a container) spawns notebooks in the
swarm.
The current configuration of jupyterhub contains:
>>>>>>
c.JupyterHub.hub_ip = '10.42.0.2'
c.JupyterHub.port = 443
c.JupyterHub.proxy_api_ip = '10.42.0.2'
c.JupyterHub.spawner_class = 'dockerspawner.SystemUserSpawner'
c.DockerSpawner.container_ip = "0.0.0.0"
network_name = 'hubnet'
c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.network_name = network_name
c.DockerSpawner.extra_host_config = { 'network_mode': network_name }
c.JupyterHub.ssl_cert = '/conf/server.crt'
c.JupyterHub.ssl_key = '/conf/server.key'
>>>>>>
Environment variables:
>>>>>>
export DOCKER_HOST=10.0.0.1:2375
>>>>>>
In this current state things don't work as expected.
Trial 1:
The test user logs in and internal server error is reported in the browser.
Log contains:
Exception: Unknown docker network 'hubnet'. Did you create it with 'docker
network create <name>' and did you pass network_mode=<name> in extra_kwargs?
Which is unexpected, because in head `docker network inspect hubnet` lists the
details of the overlay network perfectly.
Trial 2:
When in the configuration the line below is removed:
#c.DockerSpawner.use_internal_ip = True
The test user logs in and Spawner failed to start is reported in the browser.
The container is created (not a service/task) in head. The container soon dies,
due to unreachability.
Log contains:
TimeoutError: Server at http://127.0.0.1:32776/user/proba didn't respond in 30
seconds
Which may be due to either the created container is not connected to hubnet or
docker port misleads the spawner.
I don't really know how to check, because new container dies very fast to spy
on the network parameters.
Afterwards the dead container reports only about being connected to the bridge.
Trial 3:
In a completely different container docker-py was installed.
I was able to create/start/remove dummy services (sleep 3600). When creating
many of those dummy services their tasks (single containers) spread across
the both nodes as expected. Also the tasks were ping-able across hubnet.
So something is contradictory or I don't understand well enough. In
https://github.com/jupyterhub/dockerspawner it seems SystemuserSpawner is
compatible with the new swarm philosophy, however, in DockerSpawner
implementation cannot see service management implementation and so I don't
see how the user containers will be distributed in the new swarm.
Hints and help are very welcome.
Bests,
József
--
You received this message because you are subscribed to the Google Groups
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jupyter/49ad0fc7-9c59-4f47-b27e-c86e3c3de78e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
