So I'm relatively new to jupyterhub, and I wanted to try out Swarm Spawner across two separate nodes using docker swarm. I've already got a working swarm set up, and I can start services that get distributed across the worker nodes just fine in docker. Jupyterhub even has no problem creating services, too. It's just that for some reason the Hub itself can't seem to see the services that the spawner is creating. I'm posting this here because I think I've seen this error for other spawners, and I wasn't sure if it had a common fix, regardless of spawner.
Here's the error: [I 2017-09-27 00:32:17.595 JupyterHub login:91] User logged in: kel [W 2017-09-27 00:32:47.910 JupyterHub user:278] kel's server never showed up at http://jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1:8888/user/kel after 30 seconds. Giving up [I 2017-09-27 00:32:47.922 JupyterHub swarmspawner:368] Stopping and removing Docker service jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1 (id: aebmhx7) [I 2017-09-27 00:32:47.957 JupyterHub swarmspawner:372] Docker service jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1 (id: aebmhx7) removed [E 2017-09-27 00:32:48.052 JupyterHub gen:914] Exception in Future <tornado.concurrent.Future object at 0x7fb40584beb8> after timeout Traceback (most recent call last): File "/usr/lib64/python3.4/site-packages/tornado/gen.py", line 910, in error_callback future.result() File "/usr/lib/python3.4/site-packages/jupyterhub/user.py", line 294, in spawn raise e File "/usr/lib/python3.4/site-packages/jupyterhub/user.py", line 270, in spawn yield self.server.wait_up(http=True, timeout=spawner.http_timeout) File "/usr/lib/python3.4/site-packages/jupyterhub/orm.py", line 108, in wait_up yield wait_for_http_server(self.url, timeout=timeout) File "/usr/lib/python3.4/site-packages/jupyterhub/utils.py", line 94, in wait_for_http_server **locals() TimeoutError: Server at http://jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1:8888/user/kel didn't respond in 30 seconds Here are the events from docker: 2017-09-27T00:32:07.579789623-04:00 service create aebmhx7uitx4a8btoquyvknz6 (name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1) 2017-09-27T00:32:07.593433646-04:00 service update aebmhx7uitx4a8btoquyvknz6 (name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1) 2017-09-27T00:32:07.930763651-04:00 network create h3j59x5lve5npxv6hc6z2rn1b (name=jupyterhub, type=overlay) 2017-09-27T00:32:08.162563411-04:00 image pull jupyter/base-notebook:latest (name=jupyter/base-notebook) 2017-09-27T00:32:08.281409070-04:00 container create f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b (com.docker.swarm.node.id=t0r8a6997efajop7hmkez2oav, com.docker.swarm.service.id=aebmhx7uitx4a8btoquyvknz6, com.docker.swarm.service.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1, com.docker.swarm.task=, com.docker.swarm.task.id=w9wczjt08gask9klknohj0nrn, com.docker.swarm.task.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn, image=jupyter/base-notebook:latest, name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn) 2017-09-27T00:32:08.636546214-04:00 network connect h3j59x5lve5npxv6hc6z2rn1b (container=f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b, name=jupyterhub, type=overlay) 2017-09-27T00:32:08.662711254-04:00 volume mount jupyterhub-user-d37aebc5ce74fd9e4e744bdb9a9ba06b (container=f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b, destination=/home/joyvan/work, driver=local, propagation=, read/write=true) 2017-09-27T00:32:08.968032911-04:00 container start f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b (com.docker.swarm.node.id=t0r8a6997efajop7hmkez2oav, com.docker.swarm.service.id=aebmhx7uitx4a8btoquyvknz6, com.docker.swarm.service.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1, com.docker.swarm.task=, com.docker.swarm.task.id=w9wczjt08gask9klknohj0nrn, com.docker.swarm.task.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn, image=jupyter/base-notebook:latest, name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn) 2017-09-27T00:32:47.955744716-04:00 service remove aebmhx7uitx4a8btoquyvknz6 (name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1) 2017-09-27T00:32:50.070579901-04:00 container kill f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b (com.docker.swarm.node.id=t0r8a6997efajop7hmkez2oav, com.docker.swarm.service.id=aebmhx7uitx4a8btoquyvknz6, com.docker.swarm.service.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1, com.docker.swarm.task=, com.docker.swarm.task.id=w9wczjt08gask9klknohj0nrn, com.docker.swarm.task.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn, image=jupyter/base-notebook:latest, name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn, signal=15) 2017-09-27T00:32:50.215360352-04:00 container die f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b (com.docker.swarm.node.id=t0r8a6997efajop7hmkez2oav, com.docker.swarm.service.id=aebmhx7uitx4a8btoquyvknz6, com.docker.swarm.service.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1, com.docker.swarm.task=, com.docker.swarm.task.id=w9wczjt08gask9klknohj0nrn, com.docker.swarm.task.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn, exitCode=0, image=jupyter/base-notebook:latest, name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn) 2017-09-27T00:32:50.610724063-04:00 network disconnect h3j59x5lve5npxv6hc6z2rn1b (container=f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b, name=jupyterhub, type=overlay) 2017-09-27T00:32:50.622303293-04:00 volume unmount jupyterhub-user-d37aebc5ce74fd9e4e744bdb9a9ba06b (container=f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b, driver=local) 2017-09-27T00:32:50.668470503-04:00 container stop f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b (com.docker.swarm.node.id=t0r8a6997efajop7hmkez2oav, com.docker.swarm.service.id=aebmhx7uitx4a8btoquyvknz6, com.docker.swarm.service.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1, com.docker.swarm.task=, com.docker.swarm.task.id=w9wczjt08gask9klknohj0nrn, com.docker.swarm.task.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn, image=jupyter/base-notebook:latest, name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn) 2017-09-27T00:32:50.669027724-04:00 network destroy h3j59x5lve5npxv6hc6z2rn1b (name=jupyterhub, type=overlay) 2017-09-27T00:32:50.721176231-04:00 container destroy f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b (com.docker.swarm.node.id=t0r8a6997efajop7hmkez2oav, com.docker.swarm.service.id=aebmhx7uitx4a8btoquyvknz6, com.docker.swarm.service.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1, com.docker.swarm.task=, com.docker.swarm.task.id=w9wczjt08gask9klknohj0nrn, com.docker.swarm.task.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn, image=jupyter/base-notebook:latest, name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0n And lastly, here is my JupyterHub_Configy.py. Some of options are commented out because I've been tweaking this as I test, but this is how I have it set up currently. #Configuration file for JupyterHub import os import subprocess import os import errno import stat c = get_config() #pwd = os.path.dirname(__file__) c.JupyterHub.ssl_cert = '/srv/jupyterhub/jupyter.hub.crt' c.JupyterHub.ssl_key = '/srv/jupyterhub/jupyter.hub.key' # TLS config c.JupyterHub.ip = '0.0.0.0' #c.JupyterHub.port = 443 c.JupyterHub.hub_ip = '0.0.0.0' c.Authenticator.whitelist = {'kel','kellen'} c.JupyterHub.spawner_class = 'cassinyspawner.SwarmSpawner' c.JupyterHub.cleanup_servers = False #c.SwarmSpawner.start_timeout = 300 c.SwarmSpawner.service_image = 'jupyter/base-notebook' c.SwarmSpawner.jupyterhub_service_name = 'jupyterhub_jupyterhub' #c.SwarmSpawner.service_prefix = "jupyterhub" c.SwarmSpawner.networks = ["jupyterhub"] #notebook_dir = os.environ.get('NOTEBOOK_DIR') notebook_dir = '/home/jovyan/work' c.SwarmSpawner.use_user_options = True c.SwarmSpawner.notebook_dir = notebook_dir c.SwarmSpawner.container_spec = { 'command' : ['start-singleuser.sh'], 'Image' :'jupyter/base-notebook:latest', 'mounts' : [{'type' : 'volume', 'source' : 'jupyterhub-user-{username}', 'target' : '/home/joyvan/work'}] } -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/77b3086f-9af7-4b41-9948-9aa436e9e715%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
