Hi Peter, those are the logs for JupyterHub itself, but do not contain logs for the individual container (jupyter-zozo6015) - you will probably have to use `docker logs` to get those. Maybe that will contain some additional information. It looks like JupyterHub is doing its job ok, but the user container is immediately exiting (ExitCode: 0 in your posted logs.)
/Jason On 2/18/20 11:55 AM, Peter Zoltan Keresztes wrote: Already created the overlay network, I get the services up but for a very short time when the services ends with the following error: [W 2020-02-18 09:51:34.551 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-18 09:51:39.566 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-18 09:51:41.553 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-18 09:51:41.555 JupyterHub user:510] zozo6015's server never showed up at http://(:8888/user/zozo6015/ after 30 seconds. Giving up [D 2020-02-18 09:51:41.557 JupyterHub swarmspawner:144] Getting task of service 'jupyter-zozo6015' [D 2020-02-18 09:51:41.558 JupyterHub dockerspawner:777] Getting container 'jupyter-zozo6015' [D 2020-02-18 09:51:41.593 JupyterHub swarmspawner:133] Service 9ni1q9z status: {'ContainerStatus': {'ContainerID': '9480724f2d31d9fe8a784c99a17441f024ceba683951ad9be4e71766cf8499b7', 'ExitCode': 0, 'PID': 18817}, 'Message': 'started', 'PortStatus': {}, 'State': 'running', 'Timestamp': '2020-02-18T14:51:10.859533481Z'} [I 2020-02-18 09:51:41.596 JupyterHub dockerspawner:1107] Stopping service jupyter-zozo6015 (id: 9ni1q9z) [I 2020-02-18 09:51:41.597 JupyterHub swarmspawner:232] Removing service 9ni1q9zsp99khlvpbnl3x4qyx [D 2020-02-18 09:51:41.653 JupyterHub user:578] Deleting oauth client jupyterhub-user-zozo6015 [E 2020-02-18 09:51:41.674 JupyterHub gen:974] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629> exception=TimeoutError("Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 seconds",)> after timeout Traceback (most recent call last): File "/srv/.jhub/lib/python3.5/site-packages/tornado/gen.py", line 970, in error_callback future.result() File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result raise self._exception File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step result = coro.send(None) File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 636, in finish_user_spawn await spawn_future File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 528, in spawn raise e File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 502, in spawn resp = await server.wait_up(http=True, timeout=spawner.http_timeout) File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 197, in wait_for_http_server timeout=timeout File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 155, in exponential_backoff raise TimeoutError(fail_message) TimeoutError: Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 seconds Regards, Peter On Tuesday, 18 February 2020 18:30:38 UTC+2, Jason Anderson wrote: Hi Peter, I don't have much experience with Docker Swarm, but this line jumped out to me: root@jupyterhub:~# docker service ls ID NAME MODE REPLICAS IMAGE PORTS wtq7n8hvymwf jupyter-zozo6015 replicated 0/1 jupyterhub/singleuser:0.9 Replicas still show that 0 are up. I would try to check out the containers that are supposed to be running (`docker container ls -a`) to see what is happening. I did a quick StackOverflow search, you might find some help/ideas here[1]. Long story short, I am not sure it is a problem specifically with the JupyterHub setup. Though, I'm not sure what parts need to be configured for Swarm to work. Another thing the docs[2] mention is that the network needs to exist. Is it created already? docker network create --driver overlay jupyterhub_network Hope that helps, /Jason [1]: https://stackoverflow.com/questions/39284233/docker-service-replicas-remain-0-1 [2]: https://github.com/jupyterhub/dockerspawner/blob/master/README.md#swarmspawner On 2/18/20 4:29 AM, Peter Zoltan Keresztes wrote: Is there anyone in here to help me with this issue I am struggling for more than a week with? On Thursday, 13 February 2020 23:19:52 UTC+2, Peter Zoltan Keresztes wrote: hello, I am trying to convert the dockerspawner setup into swarmspawner. All I am getting is timeout. The configuration I am using looking like this: # Configuration file for JupyterHub.* c.Application.log_level = 30 c.JupyterHub.ssl_key = '/etc/letsencrypt/live/jhub.z0z0.tk/privkey.pem<http://jhub.z0z0.tk/privkey.pem>' c.JupyterHub.ssl_cert = '/etc/letsencrypt/live/jhub.z0z0.tk/fullchain.pem<http://jhub.z0z0.tk/fullchain.pem>' c.JupyterHub.port = 443 #from oauthenticator.github import LocalGitHubOAuthenticator from oauthenticator.github import GitHubOAuthenticator c.JupyterHub.authenticator_class = GitHubOAuthenticator #c.LocalGitHubOAuthenticator.create_system_users = True import os c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner' c.SwarmSpawner.network_name = 'jupyterhub_network' from jupyter_client.localinterfaces import public_ips c.JupyterHub.hub_ip = '0.0.0.0' c.JupyterHub.proxy_api_ip = '0.0.0.0' #c.SwarmSpawner.extra_placement_spec = { 'constraints' : ['node.role==worker'] } while I can see the service starting: root@jupyterhub:~# docker service ls ID NAME MODE REPLICAS IMAGE PORTS wtq7n8hvymwf jupyter-zozo6015 replicated 0/1 jupyterhub/singleuser:0.9 This is what I get in the logs: [W 2020-02-13 16:15:12.977 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:13.035 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:13.214 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:13.461 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:13.781 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:14.293 JupyterHub base:744] User zozo6015 is slow to become responsive (timeout=10) [W 2020-02-13 16:15:15.134 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:18.796 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:21.634 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:26.651 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:31.664 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:36.683 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:41.701 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:44.048 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:45.070 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:45.071 JupyterHub user:510] zozo6015's server never showed up at http://jupyter-zozo6015:8888/user/zozo6015/ after 30 seconds. Giving up [E 2020-02-13 16:15:45.229 JupyterHub gen:974] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629> exception=TimeoutError("Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 seconds",)> after timeout Traceback (most recent call last): File "/srv/.jupyterhub/lib/python3.5/site-packages/tornado/gen.py", line 970, in error_callback future.result() File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result raise self._exception File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step result = coro.send(None) File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 636, in finish_user_spawn await spawn_future File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/user.py", line 528, in spawn raise e File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/user.py", line 502, in spawn resp = await server.wait_up(http=True, timeout=spawner.http_timeout) File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 197, in wait_for_http_server timeout=timeout File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 155, in exponential_backoff raise TimeoutError(fail_message) TimeoutError: Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 seconds 16:17:39.648 [ConfigProxy] info: 200 GET /api/routes Can I get any help to make this up and running? Regards, Peter -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<javascript:>. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/57a3ad96-a24f-4925-964c-3b911ed596a5%40googlegroups.com<https://groups.google.com/d/msgid/jupyter/57a3ad96-a24f-4925-964c-3b911ed596a5%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/3c901a13-6b0e-4d8b-93f7-6d2e45c95a13%40googlegroups.com<https://groups.google.com/d/msgid/jupyter/3c901a13-6b0e-4d8b-93f7-6d2e45c95a13%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/33770353-e0e9-44b8-a28a-0507e985e67d%40uchicago.edu.
