There is a mistake in here. The jupyterhub is not running in a container. It is running natively on the master node of the swarm cluster.
Regards, Peter On 18 Feb 2020, 21:58 +0200, Jason Anderson <[email protected]>, wrote: > Having never used Swarm, I'm not sure how to help at this point, though it > feels like something Swarm-related. I would keep the timeout quite high and > do some inspecting of the containers being started by the swarmspawner. Make > sure they are connected to the same network as the JupyterHub container. You > can also try deleting the service and the network, then re-creating the > network. Sometimes I have seen issues that are only resolvable by re-creating > a Docker network, whether due to iptables or Linux bridges getting weird > settings, I don't know. > > If you recreate the network with the --attachable flag you can also attach > other containers to it for debugging. You could use an attached container to > see if you can indeed hit the port on the Jupyter user container. Given that > the error appears to be DNS-related ("Name or service not known"), I'd > suspect that really it's something with the containers not actually being > connected to the same network, or the network is misbehaving. > > Hope that helps. > /Jason > > On 2/18/20 1:45 PM, Peter Zoltan Keresztes wrote: > > Same error: > > > > [W 2020-02-18 14:43:52.180 JupyterHub user:510] zozo6015's server never > > showed up at http://jupyter-zozo6015:8888/user/zozo6015/ after 300 seconds. > > Giving up > > [E 2020-02-18 14:43:52.309 JupyterHub gen:974] Exception in Future <Task > > finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() > > done, defined at > > /srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629> > > exception=TimeoutError("Server at > > http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 300 > > seconds",)> after timeout > > Traceback (most recent call last): > > File "/srv/.jhub/lib/python3.5/site-packages/tornado/gen.py", line 970, in > > error_callback > > future.result() > > File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result > > raise self._exception > > File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step > > result = coro.send(None) > > File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", > > line 636, in finish_user_spawn > > await spawn_future > > File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 528, > > in spawn > > raise e > > File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 502, > > in spawn > > resp = await server.wait_up(http=True, timeout=spawner.http_timeout) > > File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line > > 197, in wait_for_http_server > > timeout=timeout > > File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line > > 155, in exponential_backoff > > raise TimeoutError(fail_message) > > TimeoutError: Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't > > respond in 300 seconds > > On 18 Feb 2020, 21:30 +0200, Jason Anderson <[email protected]>, > > wrote: > > > OK, then I would suggest to try raising the timeout and seeing if that > > > helps. Perhaps it is just taking longer than we would expect to spawn the > > > container. I think the settings are Spawner.http_timeout (default=30) and > > > Spawner.start_timeout (default=60). Try setting them to 5 minutes or > > > something quite high and see if that helps. If it does, then maybe the > > > spawn system is just taking a while to get everything going. > > > > > > /Jason > > > > > > On 2/18/20 1:02 PM, Peter Zoltan Keresztes wrote: > > > > The container is shutting down because the jupyterhub it is issuing the > > > > shutdown command after it fails connect to the container for 30 > > > > seconds.Here are the logs for the container. > > > > > > > > root@jupyterhub:/srv/jupyterhub# docker logs -f > > > > jupyter-zozo6015.1.u5030s66gvd7rkt4af4bn6kux > > > > Executing the command: jupyterhub-singleuser --ip="0.0.0.0" --port=8888 > > > > [W 2020-02-18 14:27:50.746 SingleUserNotebookApp configurable:168] > > > > Config option `open_browser` not recognized by `SingleUserNotebookApp`. > > > > Did you mean `browser`? > > > > [I 2020-02-18 14:27:52.760 SingleUserNotebookApp extension:158] > > > > JupyterLab extension loaded from > > > > /opt/conda/lib/python3.7/site-packages/jupyterlab > > > > [I 2020-02-18 14:27:52.761 SingleUserNotebookApp extension:159] > > > > JupyterLab application directory is /opt/conda/share/jupyter/lab > > > > [I 2020-02-18 14:27:53.666 SingleUserNotebookApp singleuser:406] > > > > Starting jupyterhub-singleuser server version 0.9.6 > > > > [I 2020-02-18 14:27:53.694 SingleUserNotebookApp notebookapp:1924] > > > > Serving notebooks from local directory: /home/jovyan > > > > [I 2020-02-18 14:27:53.696 SingleUserNotebookApp notebookapp:1924] The > > > > Jupyter Notebook is running at: > > > > [I 2020-02-18 14:27:53.697 SingleUserNotebookApp notebookapp:1924] > > > > http://6ea2c6d31f52:8888/user/zozo6015/ > > > > [I 2020-02-18 14:27:53.697 SingleUserNotebookApp notebookapp:1925] Use > > > > Control-C to stop this server and shut down all kernels (twice to skip > > > > confirmation). > > > > [C 2020-02-18 14:28:20.462 SingleUserNotebookApp notebookapp:1615] > > > > received signal 15, stopping > > > > [I 2020-02-18 14:28:20.472 SingleUserNotebookApp notebookapp:1784] > > > > Shutting down 0 kernels > > > > > > > > Regards, > > > > Peter > > > > > > > > On Tuesday, 18 February 2020 20:56:50 UTC+2, Jason Anderson wrote: > > > > > Hi Peter, those are the logs for JupyterHub itself, but do not > > > > > contain logs for the individual container (jupyter-zozo6015) - you > > > > > will probably have to use `docker logs` to get those. Maybe that will > > > > > contain some additional information. It looks like JupyterHub is > > > > > doing its job ok, but the user container is immediately exiting > > > > > (ExitCode: 0 in your posted logs.) > > > > > > > > > > /Jason > > > > > > > > > > On 2/18/20 11:55 AM, Peter Zoltan Keresztes wrote: > > > > > > Already created the overlay network, I get the services up but for > > > > > > a very short time when the services ends with the following error: > > > > > > > > > > > > > > > > > > [W 2020-02-18 09:51:34.551 JupyterHub utils:192] Failed to connect > > > > > > to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or > > > > > > service not known) > > > > > > [W 2020-02-18 09:51:39.566 JupyterHub utils:192] Failed to connect > > > > > > to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or > > > > > > service not known) > > > > > > [W 2020-02-18 09:51:41.553 JupyterHub utils:192] Failed to connect > > > > > > to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or > > > > > > service not known) > > > > > > [W 2020-02-18 09:51:41.555 JupyterHub user:510] zozo6015's server > > > > > > never showed up at http://(:8888/user/zozo6015/ after 30 seconds. > > > > > > Giving up > > > > > > [D 2020-02-18 09:51:41.557 JupyterHub swarmspawner:144] Getting > > > > > > task of service 'jupyter-zozo6015' > > > > > > [D 2020-02-18 09:51:41.558 JupyterHub dockerspawner:777] Getting > > > > > > container 'jupyter-zozo6015' > > > > > > [D 2020-02-18 09:51:41.593 JupyterHub swarmspawner:133] Service > > > > > > 9ni1q9z status: {'ContainerStatus': {'ContainerID': > > > > > > '9480724f2d31d9fe8a784c99a17441f024ceba683951ad9be4e71766cf8499b7', > > > > > > 'ExitCode': 0, > > > > > > 'PID': 18817}, > > > > > > 'Message': 'started', > > > > > > 'PortStatus': {}, > > > > > > 'State': 'running', > > > > > > 'Timestamp': '2020-02-18T14:51:10.859533481Z'} > > > > > > [I 2020-02-18 09:51:41.596 JupyterHub dockerspawner:1107] Stopping > > > > > > service jupyter-zozo6015 (id: 9ni1q9z) > > > > > > [I 2020-02-18 09:51:41.597 JupyterHub swarmspawner:232] Removing > > > > > > service 9ni1q9zsp99khlvpbnl3x4qyx > > > > > > [D 2020-02-18 09:51:41.653 JupyterHub user:578] Deleting oauth > > > > > > client jupyterhub-user-zozo6015 > > > > > > [E 2020-02-18 09:51:41.674 JupyterHub gen:974] Exception in Future > > > > > > <Task finished > > > > > > coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() > > > > > > done, defined at > > > > > > /srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629> > > > > > > exception=TimeoutError("Server at > > > > > > http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 > > > > > > seconds",)> after timeout > > > > > > Traceback (most recent call last): > > > > > > File "/srv/.jhub/lib/python3.5/site-packages/tornado/gen.py", > > > > > > line 970, in error_callback > > > > > > future.result() > > > > > > File "/usr/lib/python3.5/asyncio/futures.py", line 293, in > > > > > > result > > > > > > raise self._exception > > > > > > File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step > > > > > > result = coro.send(None) > > > > > > File > > > > > > "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", > > > > > > line 636, in finish_user_spawn > > > > > > await spawn_future > > > > > > File > > > > > > "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line > > > > > > 528, in spawn > > > > > > raise e > > > > > > File > > > > > > "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line > > > > > > 502, in spawn > > > > > > resp = await server.wait_up(http=True, > > > > > > timeout=spawner.http_timeout) > > > > > > File > > > > > > "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line > > > > > > 197, in wait_for_http_server > > > > > > timeout=timeout > > > > > > File > > > > > > "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line > > > > > > 155, in exponential_backoff > > > > > > raise TimeoutError(fail_message) > > > > > > TimeoutError: Server at > > > > > > http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 > > > > > > seconds > > > > > > > > > > > > Regards, > > > > > > Peter > > > > > > > > > > > > On Tuesday, 18 February 2020 18:30:38 UTC+2, Jason Anderson wrote: > > > > > > > Hi Peter, > > > > > > > > > > > > > > I don't have much experience with Docker Swarm, but this line > > > > > > > jumped out to me: > > > > > > > > > > > > > > root@jupyterhub:~# docker service ls > > > > > > > ID NAME MODE > > > > > > > REPLICAS IMAGE PORTS > > > > > > > wtq7n8hvymwf jupyter-zozo6015 replicated 0/1 > > > > > > > jupyterhub/singleuser:0.9 > > > > > > > > > > > > > > Replicas still show that 0 are up. I would try to check out the > > > > > > > containers that are supposed to be running (`docker container ls > > > > > > > -a`) to see what is happening. I did a quick StackOverflow > > > > > > > search, you might find some help/ideas here[1]. > > > > > > > > > > > > > > Long story short, I am not sure it is a problem specifically with > > > > > > > the JupyterHub setup. Though, I'm not sure what parts need to be > > > > > > > configured for Swarm to work. Another thing the docs[2] mention > > > > > > > is that the network needs to exist. Is it created already? > > > > > > > > > > > > > > docker network create --driver overlay jupyterhub_network > > > > > > > > > > > > > > Hope that helps, > > > > > > > /Jason > > > > > > > > > > > > > > [1]: > > > > > > > https://stackoverflow.com/questions/39284233/docker-service-replicas-remain-0-1 > > > > > > > [2]: > > > > > > > https://github.com/jupyterhub/dockerspawner/blob/master/README.md#swarmspawner > > > > > > > > > > > > > > On 2/18/20 4:29 AM, Peter Zoltan Keresztes wrote: > > > > > > > > Is there anyone in here to help me with this issue I am > > > > > > > > struggling for more than a week with? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thursday, 13 February 2020 23:19:52 UTC+2, Peter Zoltan > > > > > > > > Keresztes wrote: > > > > > > > > > hello, > > > > > > > > > > > > > > > > > > I am trying to convert the dockerspawner setup into > > > > > > > > > swarmspawner. All I am getting is timeout. > > > > > > > > > > > > > > > > > > The configuration I am using looking like this: > > > > > > > > > > > > > > > > > > # Configuration file for JupyterHub.* > > > > > > > > > c.Application.log_level = 30 > > > > > > > > > c.JupyterHub.ssl_key = > > > > > > > > > '/etc/letsencrypt/live/jhub.z0z0.tk/privkey.pem' > > > > > > > > > c.JupyterHub.ssl_cert = > > > > > > > > > '/etc/letsencrypt/live/jhub.z0z0.tk/fullchain.pem' > > > > > > > > > c.JupyterHub.port = 443 > > > > > > > > > > > > > > > > > > #from oauthenticator.github import LocalGitHubOAuthenticator > > > > > > > > > from oauthenticator.github import GitHubOAuthenticator > > > > > > > > > c.JupyterHub.authenticator_class = GitHubOAuthenticator > > > > > > > > > #c.LocalGitHubOAuthenticator.create_system_users = True > > > > > > > > > > > > > > > > > > import os > > > > > > > > > c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner' > > > > > > > > > c.SwarmSpawner.network_name = 'jupyterhub_network' > > > > > > > > > from jupyter_client.localinterfaces import public_ips > > > > > > > > > c.JupyterHub.hub_ip = '0.0.0.0' > > > > > > > > > c.JupyterHub.proxy_api_ip = '0.0.0.0' > > > > > > > > > #c.SwarmSpawner.extra_placement_spec = { 'constraints' : > > > > > > > > > ['node.role==worker'] } > > > > > > > > > > > > > > > > > > > > > > > > > > > while I can see the service starting: > > > > > > > > > > > > > > > > > > root@jupyterhub:~# docker service ls > > > > > > > > > ID NAME MODE > > > > > > > > > REPLICAS IMAGE PORTS > > > > > > > > > wtq7n8hvymwf jupyter-zozo6015 replicated > > > > > > > > > 0/1 jupyterhub/singleuser:0.9 > > > > > > > > > > > > > > > > > > This is what I get in the logs: > > > > > > > > > > > > > > > > > > [W 2020-02-13 16:15:12.977 JupyterHub utils:192] Failed to > > > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > > > ([Errno -2] Name or service not known) > > > > > > > > > [W 2020-02-13 16:15:13.035 JupyterHub utils:192] Failed to > > > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > > > ([Errno -2] Name or service not known) > > > > > > > > > [W 2020-02-13 16:15:13.214 JupyterHub utils:192] Failed to > > > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > > > ([Errno -2] Name or service not known) > > > > > > > > > [W 2020-02-13 16:15:13.461 JupyterHub utils:192] Failed to > > > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > > > ([Errno -2] Name or service not known) > > > > > > > > > [W 2020-02-13 16:15:13.781 JupyterHub utils:192] Failed to > > > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > > > ([Errno -2] Name or service not known) > > > > > > > > > [W 2020-02-13 16:15:14.293 JupyterHub base:744] User zozo6015 > > > > > > > > > is slow to become responsive (timeout=10) > > > > > > > > > [W 2020-02-13 16:15:15.134 JupyterHub utils:192] Failed to > > > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > > > ([Errno -2] Name or service not known) > > > > > > > > > [W 2020-02-13 16:15:18.796 JupyterHub utils:192] Failed to > > > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > > > ([Errno -2] Name or service not known) > > > > > > > > > [W 2020-02-13 16:15:21.634 JupyterHub utils:192] Failed to > > > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > > > ([Errno -2] Name or service not known) > > > > > > > > > [W 2020-02-13 16:15:26.651 JupyterHub utils:192] Failed to > > > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > > > ([Errno -2] Name or service not known) > > > > > > > > > [W 2020-02-13 16:15:31.664 JupyterHub utils:192] Failed to > > > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > > > ([Errno -2] Name or service not known) > > > > > > > > > [W 2020-02-13 16:15:36.683 JupyterHub utils:192] Failed to > > > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > > > ([Errno -2] Name or service not known) > > > > > > > > > [W 2020-02-13 16:15:41.701 JupyterHub utils:192] Failed to > > > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > > > ([Errno -2] Name or service not known) > > > > > > > > > [W 2020-02-13 16:15:44.048 JupyterHub utils:192] Failed to > > > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > > > ([Errno -2] Name or service not known) > > > > > > > > > [W 2020-02-13 16:15:45.070 JupyterHub utils:192] Failed to > > > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > > > ([Errno -2] Name or service not known) > > > > > > > > > [W 2020-02-13 16:15:45.071 JupyterHub user:510] zozo6015's > > > > > > > > > server never showed up at > > > > > > > > > http://jupyter-zozo6015:8888/user/zozo6015/ after 30 seconds. > > > > > > > > > Giving up > > > > > > > > > [E 2020-02-13 16:15:45.229 JupyterHub gen:974] Exception in > > > > > > > > > Future <Task finished > > > > > > > > > coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() > > > > > > > > > done, defined at > > > > > > > > > /srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629> > > > > > > > > > exception=TimeoutError("Server at > > > > > > > > > http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in > > > > > > > > > 30 seconds",)> after timeout > > > > > > > > > Traceback (most recent call last): > > > > > > > > > File > > > > > > > > > "/srv/.jupyterhub/lib/python3.5/site-packages/tornado/gen.py", > > > > > > > > > line 970, in error_callback > > > > > > > > > future.result() > > > > > > > > > File "/usr/lib/python3.5/asyncio/futures.py", line 293, > > > > > > > > > in result > > > > > > > > > raise self._exception > > > > > > > > > File "/usr/lib/python3.5/asyncio/tasks.py", line 239, > > > > > > > > > in _step > > > > > > > > > result = coro.send(None) > > > > > > > > > File > > > > > > > > > "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", > > > > > > > > > line 636, in finish_user_spawn > > > > > > > > > await spawn_future > > > > > > > > > File > > > > > > > > > "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/user.py", > > > > > > > > > line 528, in spawn > > > > > > > > > raise e > > > > > > > > > File > > > > > > > > > "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/user.py", > > > > > > > > > line 502, in spawn > > > > > > > > > resp = await server.wait_up(http=True, > > > > > > > > > timeout=spawner.http_timeout) > > > > > > > > > File > > > > > > > > > "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/utils.py", > > > > > > > > > line 197, in wait_for_http_server > > > > > > > > > timeout=timeout > > > > > > > > > File > > > > > > > > > "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/utils.py", > > > > > > > > > line 155, in exponential_backoff > > > > > > > > > raise TimeoutError(fail_message) > > > > > > > > > TimeoutError: Server at > > > > > > > > > http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in > > > > > > > > > 30 seconds > > > > > > > > > > > > > > > > > > 16:17:39.648 [ConfigProxy] info: 200 GET /api/routes > > > > > > > > > > > > > > > > > > Can I get any help to make this up and running? > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Peter > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the > > > > > > > > Google Groups "Project Jupyter" group. > > > > > > > > To unsubscribe from this group and stop receiving emails from > > > > > > > > it, send an email to [email protected]. > > > > > > > > To view this discussion on the web visit > > > > > > > > https://groups.google.com/d/msgid/jupyter/57a3ad96-a24f-4925-964c-3b911ed596a5%40googlegroups.com. > > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "Project Jupyter" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [email protected]. > > > > > > To view this discussion on the web visit > > > > > > https://groups.google.com/d/msgid/jupyter/3c901a13-6b0e-4d8b-93f7-6d2e45c95a13%40googlegroups.com. > > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "Project Jupyter" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [email protected]. > > > > To view this discussion on the web visit > > > > https://groups.google.com/d/msgid/jupyter/8d6df650-600a-460d-92bd-7bd17f98f57f%40googlegroups.com. > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Project Jupyter" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [email protected]. > > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/jupyter/4e28377a-e856-c6e4-234f-b4b3a35801a6%40uchicago.edu. > > -- > > You received this message because you are subscribed to the Google Groups > > "Project Jupyter" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/jupyter/da65a48b-6e17-4408-8010-110ba169c3cf%40Spark. > > -- > You received this message because you are subscribed to the Google Groups > "Project Jupyter" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jupyter/ec9c318b-29dd-3df5-22d4-1c871794d300%40uchicago.edu. -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/11779b81-0e24-4d7f-b1ef-3cdcf5cddf6c%40Spark.
