Same error: [W 2020-02-18 14:43:52.180 JupyterHub user:510] zozo6015's server never showed up at http://jupyter-zozo6015:8888/user/zozo6015/ after 300 seconds. Giving up [E 2020-02-18 14:43:52.309 JupyterHub gen:974] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629> exception=TimeoutError("Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 300 seconds",)> after timeout Traceback (most recent call last): File "/srv/.jhub/lib/python3.5/site-packages/tornado/gen.py", line 970, in error_callback future.result() File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result raise self._exception File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step result = coro.send(None) File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 636, in finish_user_spawn await spawn_future File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 528, in spawn raise e File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 502, in spawn resp = await server.wait_up(http=True, timeout=spawner.http_timeout) File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 197, in wait_for_http_server timeout=timeout File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 155, in exponential_backoff raise TimeoutError(fail_message) TimeoutError: Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 300 seconds On 18 Feb 2020, 21:30 +0200, Jason Anderson <[email protected]>, wrote: > OK, then I would suggest to try raising the timeout and seeing if that helps. > Perhaps it is just taking longer than we would expect to spawn the container. > I think the settings are Spawner.http_timeout (default=30) and > Spawner.start_timeout (default=60). Try setting them to 5 minutes or > something quite high and see if that helps. If it does, then maybe the spawn > system is just taking a while to get everything going. > > /Jason > > On 2/18/20 1:02 PM, Peter Zoltan Keresztes wrote: > > The container is shutting down because the jupyterhub it is issuing the > > shutdown command after it fails connect to the container for 30 > > seconds.Here are the logs for the container. > > > > root@jupyterhub:/srv/jupyterhub# docker logs -f > > jupyter-zozo6015.1.u5030s66gvd7rkt4af4bn6kux > > Executing the command: jupyterhub-singleuser --ip="0.0.0.0" --port=8888 > > [W 2020-02-18 14:27:50.746 SingleUserNotebookApp configurable:168] Config > > option `open_browser` not recognized by `SingleUserNotebookApp`. Did you > > mean `browser`? > > [I 2020-02-18 14:27:52.760 SingleUserNotebookApp extension:158] JupyterLab > > extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab > > [I 2020-02-18 14:27:52.761 SingleUserNotebookApp extension:159] JupyterLab > > application directory is /opt/conda/share/jupyter/lab > > [I 2020-02-18 14:27:53.666 SingleUserNotebookApp singleuser:406] Starting > > jupyterhub-singleuser server version 0.9.6 > > [I 2020-02-18 14:27:53.694 SingleUserNotebookApp notebookapp:1924] Serving > > notebooks from local directory: /home/jovyan > > [I 2020-02-18 14:27:53.696 SingleUserNotebookApp notebookapp:1924] The > > Jupyter Notebook is running at: > > [I 2020-02-18 14:27:53.697 SingleUserNotebookApp notebookapp:1924] > > http://6ea2c6d31f52:8888/user/zozo6015/ > > [I 2020-02-18 14:27:53.697 SingleUserNotebookApp notebookapp:1925] Use > > Control-C to stop this server and shut down all kernels (twice to skip > > confirmation). > > [C 2020-02-18 14:28:20.462 SingleUserNotebookApp notebookapp:1615] received > > signal 15, stopping > > [I 2020-02-18 14:28:20.472 SingleUserNotebookApp notebookapp:1784] Shutting > > down 0 kernels > > > > Regards, > > Peter > > > > On Tuesday, 18 February 2020 20:56:50 UTC+2, Jason Anderson wrote: > > > Hi Peter, those are the logs for JupyterHub itself, but do not contain > > > logs for the individual container (jupyter-zozo6015) - you will probably > > > have to use `docker logs` to get those. Maybe that will contain some > > > additional information. It looks like JupyterHub is doing its job ok, but > > > the user container is immediately exiting (ExitCode: 0 in your posted > > > logs.) > > > > > > /Jason > > > > > > On 2/18/20 11:55 AM, Peter Zoltan Keresztes wrote: > > > > Already created the overlay network, I get the services up but for a > > > > very short time when the services ends with the following error: > > > > > > > > > > > > [W 2020-02-18 09:51:34.551 JupyterHub utils:192] Failed to connect to > > > > http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service > > > > not known) > > > > [W 2020-02-18 09:51:39.566 JupyterHub utils:192] Failed to connect to > > > > http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service > > > > not known) > > > > [W 2020-02-18 09:51:41.553 JupyterHub utils:192] Failed to connect to > > > > http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service > > > > not known) > > > > [W 2020-02-18 09:51:41.555 JupyterHub user:510] zozo6015's server never > > > > showed up at http://(:8888/user/zozo6015/ after 30 seconds. Giving up > > > > [D 2020-02-18 09:51:41.557 JupyterHub swarmspawner:144] Getting task of > > > > service 'jupyter-zozo6015' > > > > [D 2020-02-18 09:51:41.558 JupyterHub dockerspawner:777] Getting > > > > container 'jupyter-zozo6015' > > > > [D 2020-02-18 09:51:41.593 JupyterHub swarmspawner:133] Service 9ni1q9z > > > > status: {'ContainerStatus': {'ContainerID': > > > > '9480724f2d31d9fe8a784c99a17441f024ceba683951ad9be4e71766cf8499b7', > > > > 'ExitCode': 0, > > > > 'PID': 18817}, > > > > 'Message': 'started', > > > > 'PortStatus': {}, > > > > 'State': 'running', > > > > 'Timestamp': '2020-02-18T14:51:10.859533481Z'} > > > > [I 2020-02-18 09:51:41.596 JupyterHub dockerspawner:1107] Stopping > > > > service jupyter-zozo6015 (id: 9ni1q9z) > > > > [I 2020-02-18 09:51:41.597 JupyterHub swarmspawner:232] Removing > > > > service 9ni1q9zsp99khlvpbnl3x4qyx > > > > [D 2020-02-18 09:51:41.653 JupyterHub user:578] Deleting oauth client > > > > jupyterhub-user-zozo6015 > > > > [E 2020-02-18 09:51:41.674 JupyterHub gen:974] Exception in Future > > > > <Task finished > > > > coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, > > > > defined at > > > > /srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629> > > > > exception=TimeoutError("Server at > > > > http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 > > > > seconds",)> after timeout > > > > Traceback (most recent call last): > > > > File "/srv/.jhub/lib/python3.5/site-packages/tornado/gen.py", > > > > line 970, in error_callback > > > > future.result() > > > > File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result > > > > raise self._exception > > > > File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step > > > > result = coro.send(None) > > > > File > > > > "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", > > > > line 636, in finish_user_spawn > > > > await spawn_future > > > > File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", > > > > line 528, in spawn > > > > raise e > > > > File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", > > > > line 502, in spawn > > > > resp = await server.wait_up(http=True, > > > > timeout=spawner.http_timeout) > > > > File > > > > "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 197, > > > > in wait_for_http_server > > > > timeout=timeout > > > > File > > > > "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 155, > > > > in exponential_backoff > > > > raise TimeoutError(fail_message) > > > > TimeoutError: Server at http://jupyter-zozo6015:8888/user/zozo6015/ > > > > didn't respond in 30 seconds > > > > > > > > Regards, > > > > Peter > > > > > > > > On Tuesday, 18 February 2020 18:30:38 UTC+2, Jason Anderson wrote: > > > > > Hi Peter, > > > > > > > > > > I don't have much experience with Docker Swarm, but this line jumped > > > > > out to me: > > > > > > > > > > root@jupyterhub:~# docker service ls > > > > > ID NAME MODE REPLICAS > > > > > IMAGE PORTS > > > > > wtq7n8hvymwf jupyter-zozo6015 replicated 0/1 > > > > > jupyterhub/singleuser:0.9 > > > > > > > > > > Replicas still show that 0 are up. I would try to check out the > > > > > containers that are supposed to be running (`docker container ls -a`) > > > > > to see what is happening. I did a quick StackOverflow search, you > > > > > might find some help/ideas here[1]. > > > > > > > > > > Long story short, I am not sure it is a problem specifically with the > > > > > JupyterHub setup. Though, I'm not sure what parts need to be > > > > > configured for Swarm to work. Another thing the docs[2] mention is > > > > > that the network needs to exist. Is it created already? > > > > > > > > > > docker network create --driver overlay jupyterhub_network > > > > > > > > > > Hope that helps, > > > > > /Jason > > > > > > > > > > [1]: > > > > > https://stackoverflow.com/questions/39284233/docker-service-replicas-remain-0-1 > > > > > [2]: > > > > > https://github.com/jupyterhub/dockerspawner/blob/master/README.md#swarmspawner > > > > > > > > > > On 2/18/20 4:29 AM, Peter Zoltan Keresztes wrote: > > > > > > Is there anyone in here to help me with this issue I am struggling > > > > > > for more than a week with? > > > > > > > > > > > > > > > > > > > > > > > > On Thursday, 13 February 2020 23:19:52 UTC+2, Peter Zoltan > > > > > > Keresztes wrote: > > > > > > > hello, > > > > > > > > > > > > > > I am trying to convert the dockerspawner setup into swarmspawner. > > > > > > > All I am getting is timeout. > > > > > > > > > > > > > > The configuration I am using looking like this: > > > > > > > > > > > > > > # Configuration file for JupyterHub.* > > > > > > > c.Application.log_level = 30 > > > > > > > c.JupyterHub.ssl_key = > > > > > > > '/etc/letsencrypt/live/jhub.z0z0.tk/privkey.pem' > > > > > > > c.JupyterHub.ssl_cert = > > > > > > > '/etc/letsencrypt/live/jhub.z0z0.tk/fullchain.pem' > > > > > > > c.JupyterHub.port = 443 > > > > > > > > > > > > > > #from oauthenticator.github import LocalGitHubOAuthenticator > > > > > > > from oauthenticator.github import GitHubOAuthenticator > > > > > > > c.JupyterHub.authenticator_class = GitHubOAuthenticator > > > > > > > #c.LocalGitHubOAuthenticator.create_system_users = True > > > > > > > > > > > > > > import os > > > > > > > c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner' > > > > > > > c.SwarmSpawner.network_name = 'jupyterhub_network' > > > > > > > from jupyter_client.localinterfaces import public_ips > > > > > > > c.JupyterHub.hub_ip = '0.0.0.0' > > > > > > > c.JupyterHub.proxy_api_ip = '0.0.0.0' > > > > > > > #c.SwarmSpawner.extra_placement_spec = { 'constraints' : > > > > > > > ['node.role==worker'] } > > > > > > > > > > > > > > > > > > > > > while I can see the service starting: > > > > > > > > > > > > > > root@jupyterhub:~# docker service ls > > > > > > > ID NAME MODE > > > > > > > REPLICAS IMAGE PORTS > > > > > > > wtq7n8hvymwf jupyter-zozo6015 replicated 0/1 > > > > > > > jupyterhub/singleuser:0.9 > > > > > > > > > > > > > > This is what I get in the logs: > > > > > > > > > > > > > > [W 2020-02-13 16:15:12.977 JupyterHub utils:192] Failed to > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno > > > > > > > -2] Name or service not known) > > > > > > > [W 2020-02-13 16:15:13.035 JupyterHub utils:192] Failed to > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno > > > > > > > -2] Name or service not known) > > > > > > > [W 2020-02-13 16:15:13.214 JupyterHub utils:192] Failed to > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno > > > > > > > -2] Name or service not known) > > > > > > > [W 2020-02-13 16:15:13.461 JupyterHub utils:192] Failed to > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno > > > > > > > -2] Name or service not known) > > > > > > > [W 2020-02-13 16:15:13.781 JupyterHub utils:192] Failed to > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno > > > > > > > -2] Name or service not known) > > > > > > > [W 2020-02-13 16:15:14.293 JupyterHub base:744] User zozo6015 is > > > > > > > slow to become responsive (timeout=10) > > > > > > > [W 2020-02-13 16:15:15.134 JupyterHub utils:192] Failed to > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno > > > > > > > -2] Name or service not known) > > > > > > > [W 2020-02-13 16:15:18.796 JupyterHub utils:192] Failed to > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno > > > > > > > -2] Name or service not known) > > > > > > > [W 2020-02-13 16:15:21.634 JupyterHub utils:192] Failed to > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno > > > > > > > -2] Name or service not known) > > > > > > > [W 2020-02-13 16:15:26.651 JupyterHub utils:192] Failed to > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno > > > > > > > -2] Name or service not known) > > > > > > > [W 2020-02-13 16:15:31.664 JupyterHub utils:192] Failed to > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno > > > > > > > -2] Name or service not known) > > > > > > > [W 2020-02-13 16:15:36.683 JupyterHub utils:192] Failed to > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno > > > > > > > -2] Name or service not known) > > > > > > > [W 2020-02-13 16:15:41.701 JupyterHub utils:192] Failed to > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno > > > > > > > -2] Name or service not known) > > > > > > > [W 2020-02-13 16:15:44.048 JupyterHub utils:192] Failed to > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno > > > > > > > -2] Name or service not known) > > > > > > > [W 2020-02-13 16:15:45.070 JupyterHub utils:192] Failed to > > > > > > > connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno > > > > > > > -2] Name or service not known) > > > > > > > [W 2020-02-13 16:15:45.071 JupyterHub user:510] zozo6015's server > > > > > > > never showed up at http://jupyter-zozo6015:8888/user/zozo6015/ > > > > > > > after 30 seconds. Giving up > > > > > > > [E 2020-02-13 16:15:45.229 JupyterHub gen:974] Exception in > > > > > > > Future <Task finished > > > > > > > coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() > > > > > > > done, defined at > > > > > > > /srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629> > > > > > > > exception=TimeoutError("Server at > > > > > > > http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 > > > > > > > seconds",)> after timeout > > > > > > > Traceback (most recent call last): > > > > > > > File > > > > > > > "/srv/.jupyterhub/lib/python3.5/site-packages/tornado/gen.py", > > > > > > > line 970, in error_callback > > > > > > > future.result() > > > > > > > File "/usr/lib/python3.5/asyncio/futures.py", line 293, in > > > > > > > result > > > > > > > raise self._exception > > > > > > > File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in > > > > > > > _step > > > > > > > result = coro.send(None) > > > > > > > File > > > > > > > "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", > > > > > > > line 636, in finish_user_spawn > > > > > > > await spawn_future > > > > > > > File > > > > > > > "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/user.py", > > > > > > > line 528, in spawn > > > > > > > raise e > > > > > > > File > > > > > > > "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/user.py", > > > > > > > line 502, in spawn > > > > > > > resp = await server.wait_up(http=True, > > > > > > > timeout=spawner.http_timeout) > > > > > > > File > > > > > > > "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/utils.py", > > > > > > > line 197, in wait_for_http_server > > > > > > > timeout=timeout > > > > > > > File > > > > > > > "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/utils.py", > > > > > > > line 155, in exponential_backoff > > > > > > > raise TimeoutError(fail_message) > > > > > > > TimeoutError: Server at > > > > > > > http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 > > > > > > > seconds > > > > > > > > > > > > > > 16:17:39.648 [ConfigProxy] info: 200 GET /api/routes > > > > > > > > > > > > > > Can I get any help to make this up and running? > > > > > > > > > > > > > > Regards, > > > > > > > Peter > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "Project Jupyter" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [email protected]. > > > > > > To view this discussion on the web visit > > > > > > https://groups.google.com/d/msgid/jupyter/57a3ad96-a24f-4925-964c-3b911ed596a5%40googlegroups.com. > > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "Project Jupyter" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [email protected]. > > > > To view this discussion on the web visit > > > > https://groups.google.com/d/msgid/jupyter/3c901a13-6b0e-4d8b-93f7-6d2e45c95a13%40googlegroups.com. > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Project Jupyter" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/jupyter/8d6df650-600a-460d-92bd-7bd17f98f57f%40googlegroups.com. > > -- > You received this message because you are subscribed to the Google Groups > "Project Jupyter" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jupyter/4e28377a-e856-c6e4-234f-b4b3a35801a6%40uchicago.edu.
-- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/da65a48b-6e17-4408-8010-110ba169c3cf%40Spark.
