Having never used Swarm, I'm not sure how to help at this point, though it
feels like something Swarm-related. I would keep the timeout quite high and do
some inspecting of the containers being started by the swarmspawner. Make sure
they are connected to the same network as the JupyterHub container. You can
also try deleting the service and the network, then re-creating the network.
Sometimes I have seen issues that are only resolvable by re-creating a Docker
network, whether due to iptables or Linux bridges getting weird settings, I
don't know.
If you recreate the network with the --attachable flag you can also attach
other containers to it for debugging. You could use an attached container to
see if you can indeed hit the port on the Jupyter user container. Given that
the error appears to be DNS-related ("Name or service not known"), I'd suspect
that really it's something with the containers not actually being connected to
the same network, or the network is misbehaving.
Hope that helps.
/Jason
On 2/18/20 1:45 PM, Peter Zoltan Keresztes wrote:
Same error:
[W 2020-02-18 14:43:52.180 JupyterHub user:510] zozo6015's server never showed
up at http://jupyter-zozo6015:8888/user/zozo6015/ after 300 seconds. Giving up
[E 2020-02-18 14:43:52.309 JupyterHub gen:974] Exception in Future <Task
finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done,
defined at
/srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629>
exception=TimeoutError("Server at http://jupyter-zozo6015:8888/user/zozo6015/
didn't respond in 300 seconds",)> after timeout
Traceback (most recent call last):
File "/srv/.jhub/lib/python3.5/site-packages/tornado/gen.py", line 970, in
error_callback
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line
636, in finish_user_spawn
await spawn_future
File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 528, in
spawn
raise e
File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 502, in
spawn
resp = await server.wait_up(http=True, timeout=spawner.http_timeout)
File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 197, in
wait_for_http_server
timeout=timeout
File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 155, in
exponential_backoff
raise TimeoutError(fail_message)
TimeoutError: Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't
respond in 300 seconds
On 18 Feb 2020, 21:30 +0200, Jason Anderson
<[email protected]><mailto:[email protected]>, wrote:
OK, then I would suggest to try raising the timeout and seeing if that helps.
Perhaps it is just taking longer than we would expect to spawn the container. I
think the settings are Spawner.http_timeout (default=30) and
Spawner.start_timeout (default=60). Try setting them to 5 minutes or something
quite high and see if that helps. If it does, then maybe the spawn system is
just taking a while to get everything going.
/Jason
On 2/18/20 1:02 PM, Peter Zoltan Keresztes wrote:
The container is shutting down because the jupyterhub it is issuing the
shutdown command after it fails connect to the container for 30 seconds.Here
are the logs for the container.
root@jupyterhub:/srv/jupyterhub# docker logs -f
jupyter-zozo6015.1.u5030s66gvd7rkt4af4bn6kux
Executing the command: jupyterhub-singleuser --ip="0.0.0.0" --port=8888
[W 2020-02-18 14:27:50.746 SingleUserNotebookApp configurable:168] Config
option `open_browser` not recognized by `SingleUserNotebookApp`. Did you mean
`browser`?
[I 2020-02-18 14:27:52.760 SingleUserNotebookApp extension:158] JupyterLab
extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
[I 2020-02-18 14:27:52.761 SingleUserNotebookApp extension:159] JupyterLab
application directory is /opt/conda/share/jupyter/lab
[I 2020-02-18 14:27:53.666 SingleUserNotebookApp singleuser:406] Starting
jupyterhub-singleuser server version 0.9.6
[I 2020-02-18 14:27:53.694 SingleUserNotebookApp notebookapp:1924] Serving
notebooks from local directory: /home/jovyan
[I 2020-02-18 14:27:53.696 SingleUserNotebookApp notebookapp:1924] The Jupyter
Notebook is running at:
[I 2020-02-18 14:27:53.697 SingleUserNotebookApp notebookapp:1924]
http://6ea2c6d31f52:8888/user/zozo6015/
[I 2020-02-18 14:27:53.697 SingleUserNotebookApp notebookapp:1925] Use
Control-C to stop this server and shut down all kernels (twice to skip
confirmation).
[C 2020-02-18 14:28:20.462 SingleUserNotebookApp notebookapp:1615] received
signal 15, stopping
[I 2020-02-18 14:28:20.472 SingleUserNotebookApp notebookapp:1784] Shutting
down 0 kernels
Regards,
Peter
On Tuesday, 18 February 2020 20:56:50 UTC+2, Jason Anderson wrote:
Hi Peter, those are the logs for JupyterHub itself, but do not contain logs for
the individual container (jupyter-zozo6015) - you will probably have to use
`docker logs` to get those. Maybe that will contain some additional
information. It looks like JupyterHub is doing its job ok, but the user
container is immediately exiting (ExitCode: 0 in your posted logs.)
/Jason
On 2/18/20 11:55 AM, Peter Zoltan Keresztes wrote:
Already created the overlay network, I get the services up but for a very short
time when the services ends with the following error:
[W 2020-02-18 09:51:34.551 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-18 09:51:39.566 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-18 09:51:41.553 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-18 09:51:41.555 JupyterHub user:510] zozo6015's server never showed
up at http://(:8888/user/zozo6015/ after 30 seconds. Giving up
[D 2020-02-18 09:51:41.557 JupyterHub swarmspawner:144] Getting task of service
'jupyter-zozo6015'
[D 2020-02-18 09:51:41.558 JupyterHub dockerspawner:777] Getting container
'jupyter-zozo6015'
[D 2020-02-18 09:51:41.593 JupyterHub swarmspawner:133] Service 9ni1q9z status:
{'ContainerStatus': {'ContainerID':
'9480724f2d31d9fe8a784c99a17441f024ceba683951ad9be4e71766cf8499b7',
'ExitCode': 0,
'PID': 18817},
'Message': 'started',
'PortStatus': {},
'State': 'running',
'Timestamp': '2020-02-18T14:51:10.859533481Z'}
[I 2020-02-18 09:51:41.596 JupyterHub dockerspawner:1107] Stopping service
jupyter-zozo6015 (id: 9ni1q9z)
[I 2020-02-18 09:51:41.597 JupyterHub swarmspawner:232] Removing service
9ni1q9zsp99khlvpbnl3x4qyx
[D 2020-02-18 09:51:41.653 JupyterHub user:578] Deleting oauth client
jupyterhub-user-zozo6015
[E 2020-02-18 09:51:41.674 JupyterHub gen:974] Exception in Future <Task
finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done,
defined at
/srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629>
exception=TimeoutError("Server at http://jupyter-zozo6015:8888/user/zozo6015/
didn't respond in 30 seconds",)> after timeout
Traceback (most recent call last):
File "/srv/.jhub/lib/python3.5/site-packages/tornado/gen.py", line 970,
in error_callback
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File
"/srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 636,
in finish_user_spawn
await spawn_future
File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line
528, in spawn
raise e
File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line
502, in spawn
resp = await server.wait_up(http=True, timeout=spawner.http_timeout)
File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line
197, in wait_for_http_server
timeout=timeout
File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line
155, in exponential_backoff
raise TimeoutError(fail_message)
TimeoutError: Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't
respond in 30 seconds
Regards,
Peter
On Tuesday, 18 February 2020 18:30:38 UTC+2, Jason Anderson wrote:
Hi Peter,
I don't have much experience with Docker Swarm, but this line jumped out to me:
root@jupyterhub:~# docker service ls
ID NAME MODE REPLICAS
IMAGE PORTS
wtq7n8hvymwf jupyter-zozo6015 replicated 0/1
jupyterhub/singleuser:0.9
Replicas still show that 0 are up. I would try to check out the containers that
are supposed to be running (`docker container ls -a`) to see what is happening.
I did a quick StackOverflow search, you might find some help/ideas here[1].
Long story short, I am not sure it is a problem specifically with the
JupyterHub setup. Though, I'm not sure what parts need to be configured for
Swarm to work. Another thing the docs[2] mention is that the network needs to
exist. Is it created already?
docker network create --driver overlay jupyterhub_network
Hope that helps,
/Jason
[1]:
https://stackoverflow.com/questions/39284233/docker-service-replicas-remain-0-1
[2]:
https://github.com/jupyterhub/dockerspawner/blob/master/README.md#swarmspawner
On 2/18/20 4:29 AM, Peter Zoltan Keresztes wrote:
Is there anyone in here to help me with this issue I am struggling for more
than a week with?
On Thursday, 13 February 2020 23:19:52 UTC+2, Peter Zoltan Keresztes wrote:
hello,
I am trying to convert the dockerspawner setup into swarmspawner. All I am
getting is timeout.
The configuration I am using looking like this:
# Configuration file for JupyterHub.*
c.Application.log_level = 30
c.JupyterHub.ssl_key =
'/etc/letsencrypt/live/jhub.z0z0.tk/privkey.pem<http://jhub.z0z0.tk/privkey.pem>'
c.JupyterHub.ssl_cert =
'/etc/letsencrypt/live/jhub.z0z0.tk/fullchain.pem<http://jhub.z0z0.tk/fullchain.pem>'
c.JupyterHub.port = 443
#from oauthenticator.github import LocalGitHubOAuthenticator
from oauthenticator.github import GitHubOAuthenticator
c.JupyterHub.authenticator_class = GitHubOAuthenticator
#c.LocalGitHubOAuthenticator.create_system_users = True
import os
c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'
c.SwarmSpawner.network_name = 'jupyterhub_network'
from jupyter_client.localinterfaces import public_ips
c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.proxy_api_ip = '0.0.0.0'
#c.SwarmSpawner.extra_placement_spec = { 'constraints' : ['node.role==worker'] }
while I can see the service starting:
root@jupyterhub:~# docker service ls
ID NAME MODE REPLICAS
IMAGE PORTS
wtq7n8hvymwf jupyter-zozo6015 replicated 0/1
jupyterhub/singleuser:0.9
This is what I get in the logs:
[W 2020-02-13 16:15:12.977 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-13 16:15:13.035 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-13 16:15:13.214 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-13 16:15:13.461 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-13 16:15:13.781 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-13 16:15:14.293 JupyterHub base:744] User zozo6015 is slow to become
responsive (timeout=10)
[W 2020-02-13 16:15:15.134 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-13 16:15:18.796 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-13 16:15:21.634 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-13 16:15:26.651 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-13 16:15:31.664 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-13 16:15:36.683 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-13 16:15:41.701 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-13 16:15:44.048 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-13 16:15:45.070 JupyterHub utils:192] Failed to connect to
http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not
known)
[W 2020-02-13 16:15:45.071 JupyterHub user:510] zozo6015's server never showed
up at http://jupyter-zozo6015:8888/user/zozo6015/ after 30 seconds. Giving up
[E 2020-02-13 16:15:45.229 JupyterHub gen:974] Exception in Future <Task
finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done,
defined at
/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629>
exception=TimeoutError("Server at http://jupyter-zozo6015:8888/user/zozo6015/
didn't respond in 30 seconds",)> after timeout
Traceback (most recent call last):
File "/srv/.jupyterhub/lib/python3.5/site-packages/tornado/gen.py", line
970, in error_callback
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File
"/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py",
line 636, in finish_user_spawn
await spawn_future
File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/user.py",
line 528, in spawn
raise e
File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/user.py",
line 502, in spawn
resp = await server.wait_up(http=True, timeout=spawner.http_timeout)
File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/utils.py",
line 197, in wait_for_http_server
timeout=timeout
File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/utils.py",
line 155, in exponential_backoff
raise TimeoutError(fail_message)
TimeoutError: Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't
respond in 30 seconds
16:17:39.648 [ConfigProxy] info: 200 GET /api/routes
Can I get any help to make this up and running?
Regards,
Peter
--
You received this message because you are subscribed to the Google Groups
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jupyter/57a3ad96-a24f-4925-964c-3b911ed596a5%40googlegroups.com<https://groups.google.com/d/msgid/jupyter/57a3ad96-a24f-4925-964c-3b911ed596a5%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected]<javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jupyter/3c901a13-6b0e-4d8b-93f7-6d2e45c95a13%40googlegroups.com<https://groups.google.com/d/msgid/jupyter/3c901a13-6b0e-4d8b-93f7-6d2e45c95a13%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected]<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jupyter/8d6df650-600a-460d-92bd-7bd17f98f57f%40googlegroups.com<https://groups.google.com/d/msgid/jupyter/8d6df650-600a-460d-92bd-7bd17f98f57f%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected]<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jupyter/4e28377a-e856-c6e4-234f-b4b3a35801a6%40uchicago.edu<https://groups.google.com/d/msgid/jupyter/4e28377a-e856-c6e4-234f-b4b3a35801a6%40uchicago.edu?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected]<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jupyter/da65a48b-6e17-4408-8010-110ba169c3cf%40Spark<https://groups.google.com/d/msgid/jupyter/da65a48b-6e17-4408-8010-110ba169c3cf%40Spark?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jupyter/ec9c318b-29dd-3df5-22d4-1c871794d300%40uchicago.edu.
