OK, then I would suggest to try raising the timeout and seeing if that helps. Perhaps it is just taking longer than we would expect to spawn the container. I think the settings are Spawner.http_timeout (default=30) and Spawner.start_timeout (default=60). Try setting them to 5 minutes or something quite high and see if that helps. If it does, then maybe the spawn system is just taking a while to get everything going.
/Jason On 2/18/20 1:02 PM, Peter Zoltan Keresztes wrote: The container is shutting down because the jupyterhub it is issuing the shutdown command after it fails connect to the container for 30 seconds.Here are the logs for the container. root@jupyterhub:/srv/jupyterhub# docker logs -f jupyter-zozo6015.1.u5030s66gvd7rkt4af4bn6kux Executing the command: jupyterhub-singleuser --ip="0.0.0.0" --port=8888 [W 2020-02-18 14:27:50.746 SingleUserNotebookApp configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`. Did you mean `browser`? [I 2020-02-18 14:27:52.760 SingleUserNotebookApp extension:158] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab [I 2020-02-18 14:27:52.761 SingleUserNotebookApp extension:159] JupyterLab application directory is /opt/conda/share/jupyter/lab [I 2020-02-18 14:27:53.666 SingleUserNotebookApp singleuser:406] Starting jupyterhub-singleuser server version 0.9.6 [I 2020-02-18 14:27:53.694 SingleUserNotebookApp notebookapp:1924] Serving notebooks from local directory: /home/jovyan [I 2020-02-18 14:27:53.696 SingleUserNotebookApp notebookapp:1924] The Jupyter Notebook is running at: [I 2020-02-18 14:27:53.697 SingleUserNotebookApp notebookapp:1924] http://6ea2c6d31f52:8888/user/zozo6015/ [I 2020-02-18 14:27:53.697 SingleUserNotebookApp notebookapp:1925] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 2020-02-18 14:28:20.462 SingleUserNotebookApp notebookapp:1615] received signal 15, stopping [I 2020-02-18 14:28:20.472 SingleUserNotebookApp notebookapp:1784] Shutting down 0 kernels Regards, Peter On Tuesday, 18 February 2020 20:56:50 UTC+2, Jason Anderson wrote: Hi Peter, those are the logs for JupyterHub itself, but do not contain logs for the individual container (jupyter-zozo6015) - you will probably have to use `docker logs` to get those. Maybe that will contain some additional information. It looks like JupyterHub is doing its job ok, but the user container is immediately exiting (ExitCode: 0 in your posted logs.) /Jason On 2/18/20 11:55 AM, Peter Zoltan Keresztes wrote: Already created the overlay network, I get the services up but for a very short time when the services ends with the following error: [W 2020-02-18 09:51:34.551 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-18 09:51:39.566 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-18 09:51:41.553 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-18 09:51:41.555 JupyterHub user:510] zozo6015's server never showed up at http://(:8888/user/zozo6015/ after 30 seconds. Giving up [D 2020-02-18 09:51:41.557 JupyterHub swarmspawner:144] Getting task of service 'jupyter-zozo6015' [D 2020-02-18 09:51:41.558 JupyterHub dockerspawner:777] Getting container 'jupyter-zozo6015' [D 2020-02-18 09:51:41.593 JupyterHub swarmspawner:133] Service 9ni1q9z status: {'ContainerStatus': {'ContainerID': '9480724f2d31d9fe8a784c99a17441f024ceba683951ad9be4e71766cf8499b7', 'ExitCode': 0, 'PID': 18817}, 'Message': 'started', 'PortStatus': {}, 'State': 'running', 'Timestamp': '2020-02-18T14:51:10.859533481Z'} [I 2020-02-18 09:51:41.596 JupyterHub dockerspawner:1107] Stopping service jupyter-zozo6015 (id: 9ni1q9z) [I 2020-02-18 09:51:41.597 JupyterHub swarmspawner:232] Removing service 9ni1q9zsp99khlvpbnl3x4qyx [D 2020-02-18 09:51:41.653 JupyterHub user:578] Deleting oauth client jupyterhub-user-zozo6015 [E 2020-02-18 09:51:41.674 JupyterHub gen:974] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629> exception=TimeoutError("Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 seconds",)> after timeout Traceback (most recent call last): File "/srv/.jhub/lib/python3.5/site-packages/tornado/gen.py", line 970, in error_callback future.result() File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result raise self._exception File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step result = coro.send(None) File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 636, in finish_user_spawn await spawn_future File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 528, in spawn raise e File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 502, in spawn resp = await server.wait_up(http=True, timeout=spawner.http_timeout) File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 197, in wait_for_http_server timeout=timeout File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 155, in exponential_backoff raise TimeoutError(fail_message) TimeoutError: Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 seconds Regards, Peter On Tuesday, 18 February 2020 18:30:38 UTC+2, Jason Anderson wrote: Hi Peter, I don't have much experience with Docker Swarm, but this line jumped out to me: root@jupyterhub:~# docker service ls ID NAME MODE REPLICAS IMAGE PORTS wtq7n8hvymwf jupyter-zozo6015 replicated 0/1 jupyterhub/singleuser:0.9 Replicas still show that 0 are up. I would try to check out the containers that are supposed to be running (`docker container ls -a`) to see what is happening. I did a quick StackOverflow search, you might find some help/ideas here[1]. Long story short, I am not sure it is a problem specifically with the JupyterHub setup. Though, I'm not sure what parts need to be configured for Swarm to work. Another thing the docs[2] mention is that the network needs to exist. Is it created already? docker network create --driver overlay jupyterhub_network Hope that helps, /Jason [1]: https://stackoverflow.com/questions/39284233/docker-service-replicas-remain-0-1 [2]: https://github.com/jupyterhub/dockerspawner/blob/master/README.md#swarmspawner On 2/18/20 4:29 AM, Peter Zoltan Keresztes wrote: Is there anyone in here to help me with this issue I am struggling for more than a week with? On Thursday, 13 February 2020 23:19:52 UTC+2, Peter Zoltan Keresztes wrote: hello, I am trying to convert the dockerspawner setup into swarmspawner. All I am getting is timeout. The configuration I am using looking like this: # Configuration file for JupyterHub.* c.Application.log_level = 30 c.JupyterHub.ssl_key = '/etc/letsencrypt/live/jhub.z0z0.tk/privkey.pem<http://jhub.z0z0.tk/privkey.pem>' c.JupyterHub.ssl_cert = '/etc/letsencrypt/live/jhub.z0z0.tk/fullchain.pem<http://jhub.z0z0.tk/fullchain.pem>' c.JupyterHub.port = 443 #from oauthenticator.github import LocalGitHubOAuthenticator from oauthenticator.github import GitHubOAuthenticator c.JupyterHub.authenticator_class = GitHubOAuthenticator #c.LocalGitHubOAuthenticator.create_system_users = True import os c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner' c.SwarmSpawner.network_name = 'jupyterhub_network' from jupyter_client.localinterfaces import public_ips c.JupyterHub.hub_ip = '0.0.0.0' c.JupyterHub.proxy_api_ip = '0.0.0.0' #c.SwarmSpawner.extra_placement_spec = { 'constraints' : ['node.role==worker'] } while I can see the service starting: root@jupyterhub:~# docker service ls ID NAME MODE REPLICAS IMAGE PORTS wtq7n8hvymwf jupyter-zozo6015 replicated 0/1 jupyterhub/singleuser:0.9 This is what I get in the logs: [W 2020-02-13 16:15:12.977 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:13.035 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:13.214 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:13.461 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:13.781 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:14.293 JupyterHub base:744] User zozo6015 is slow to become responsive (timeout=10) [W 2020-02-13 16:15:15.134 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:18.796 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:21.634 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:26.651 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:31.664 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:36.683 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:41.701 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:44.048 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:45.070 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-13 16:15:45.071 JupyterHub user:510] zozo6015's server never showed up at http://jupyter-zozo6015:8888/user/zozo6015/ after 30 seconds. Giving up [E 2020-02-13 16:15:45.229 JupyterHub gen:974] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629> exception=TimeoutError("Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 seconds",)> after timeout Traceback (most recent call last): File "/srv/.jupyterhub/lib/python3.5/site-packages/tornado/gen.py", line 970, in error_callback future.result() File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result raise self._exception File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step result = coro.send(None) File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 636, in finish_user_spawn await spawn_future File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/user.py", line 528, in spawn raise e File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/user.py", line 502, in spawn resp = await server.wait_up(http=True, timeout=spawner.http_timeout) File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 197, in wait_for_http_server timeout=timeout File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 155, in exponential_backoff raise TimeoutError(fail_message) TimeoutError: Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 seconds 16:17:39.648 [ConfigProxy] info: 200 GET /api/routes Can I get any help to make this up and running? Regards, Peter -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/57a3ad96-a24f-4925-964c-3b911ed596a5%40googlegroups.com<https://groups.google.com/d/msgid/jupyter/57a3ad96-a24f-4925-964c-3b911ed596a5%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<javascript:>. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/3c901a13-6b0e-4d8b-93f7-6d2e45c95a13%40googlegroups.com<https://groups.google.com/d/msgid/jupyter/3c901a13-6b0e-4d8b-93f7-6d2e45c95a13%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/8d6df650-600a-460d-92bd-7bd17f98f57f%40googlegroups.com<https://groups.google.com/d/msgid/jupyter/8d6df650-600a-460d-92bd-7bd17f98f57f%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/4e28377a-e856-c6e4-234f-b4b3a35801a6%40uchicago.edu.
