Dear group, I am writing to seek help on the behaviour a recent installation of Jupiterhub is showing. I hope someone can help me to single out where the problem is. I have followed the how-to (https://tljh.jupyter.org/en/latest/ ) for the server installation which is hosted on a virtual server with a public ip. The installation worked fine and I have done just minor modifications as suggested in the document. The hub seems to work fine, I've enabled ssl and open the firewall ports. I can connect and start the lab (the app I've set as default) However from time to time when I try to run any notebook/console/terminal everything freeze and the kernel keeps on trying to connect for a while, switching to disconnected after some time. I have tried to dig into the log (journalctl -u jupyterhub, journalctl -u traefik, journalctl -u jupyter-rago) but none of the above showed any clear error. My tljh config is as follow:
root@AItesting:/home# tljh-config show users: admin: - rago https: enabled: true letsencrypt: email: [email protected] domains: - xxxxxxx(fully resolved website) user_environment: default_app: jupyterlab Can anyone suggest a way to diagnose the problem? Thank you in advance. Antonio Rago Below are the final part of my logs : root@AItesting:/home# journalctl -u jupyterhub Feb 15 22:28:23 AItesting python3[2015]: [I 2021-02-15 22:28:23.251 JupyterHub log:181] 200 GET /hub/api/users ([email protected]) 8.34ms Feb 15 22:29:23 AItesting python3[2015]: [I 2021-02-15 22:29:23.255 JupyterHub log:181] 200 GET /hub/api/users ([email protected]) 11.77ms Feb 15 22:29:26 AItesting python3[2015]: /opt/tljh/hub/lib/python3.8/site-packages/jupyterhub_traefik_proxy/toml.py:104: RuntimeWarning: Escape character '_' cannot be a safe character. Set allow_co> Feb 15 22:29:26 AItesting python3[2015]: escaped_routespec = escapism.escape(traefik_routespec, safe=safe) Feb 15 22:29:26 AItesting python3[2015]: [I 2021-02-15 22:29:26.843 JupyterHub proxy:320] Checking routes Feb 15 22:29:35 AItesting python3[2015]: [I 2021-02-15 22:29:35.046 JupyterHub pages:148] User is running: rago Feb 15 22:29:35 AItesting python3[2015]: [I 2021-02-15 22:29:35.047 JupyterHub log:181] 302 GET /hub/spawn?next=/hub/home -> /hub/home ([email protected]) 9.69ms Feb 15 22:29:35 AItesting python3[2015]: [I 2021-02-15 22:29:35.085 JupyterHub log:181] 200 GET /hub/home ([email protected]) 15.55ms Feb 15 22:29:36 AItesting python3[2015]: [I 2021-02-15 22:29:36.562 JupyterHub log:181] 200 GET /hub/api/authorizations/token/[secret] ([email protected]) 15.78ms Feb 15 22:30:23 AItesting python3[2015]: [I 2021-02-15 22:30:23.255 JupyterHub log:181] 200 GET /hub/api/users ([email protected]) 11.19ms Feb 15 22:31:23 AItesting python3[2015]: [I 2021-02-15 22:31:23.254 JupyterHub log:181] 200 GET /hub/api/users ([email protected]) 10.63ms Feb 15 22:32:23 AItesting python3[2015]: [I 2021-02-15 22:32:23.258 JupyterHub log:181] 200 GET /hub/api/users ([email protected]) 14.82ms Feb 15 22:33:06 AItesting python3[2015]: [I 2021-02-15 22:33:06.488 JupyterHub log:181] 200 POST /hub/api/users/rago/activity ([email protected]) 15.77ms Feb 15 22:33:23 AItesting python3[2015]: [I 2021-02-15 22:33:23.251 JupyterHub log:181] 200 GET /hub/api/users ([email protected]) 8.08ms Feb 15 22:34:23 AItesting python3[2015]: [I 2021-02-15 22:34:23.254 JupyterHub log:181] 200 GET /hub/api/users ([email protected]) 10.77ms Feb 15 22:34:26 AItesting python3[2015]: /opt/tljh/hub/lib/python3.8/site-packages/jupyterhub_traefik_proxy/toml.py:104: RuntimeWarning: Escape character '_' cannot be a safe character. Set allow_co> Feb 15 22:34:26 AItesting python3[2015]: escaped_routespec = escapism.escape(traefik_routespec, safe=safe) Feb 15 22:34:26 AItesting python3[2015]: [I 2021-02-15 22:34:26.843 JupyterHub proxy:320] Checking routes Feb 15 22:35:23 AItesting python3[2015]: [I 2021-02-15 22:35:23.255 JupyterHub log:181] 200 GET /hub/api/users ([email protected]) 11.72ms root@AItesting:/home# journalctl -u traefik Feb 15 21:25:30 AItesting traefik[2012]: time="2021-02-15T21:25:30+01:00" level=info msg="Skipping same configuration for provider file" Feb 15 21:25:31 AItesting traefik[2012]: time="2021-02-15T21:25:31+01:00" level=info msg="Server configuration reloaded on :80" Feb 15 21:25:31 AItesting traefik[2012]: time="2021-02-15T21:25:31+01:00" level=info msg="Server configuration reloaded on :443" Feb 15 21:25:31 AItesting traefik[2012]: time="2021-02-15T21:25:31+01:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099" Feb 15 21:25:33 AItesting traefik[2012]: time="2021-02-15T21:25:33+01:00" level=info msg="Server configuration reloaded on :80" Feb 15 21:25:33 AItesting traefik[2012]: time="2021-02-15T21:25:33+01:00" level=info msg="Server configuration reloaded on :443" Feb 15 21:25:33 AItesting traefik[2012]: time="2021-02-15T21:25:33+01:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099" Feb 15 21:25:40 AItesting traefik[2012]: {"BackendAddr":"127.0.0.1:33115","BackendName":"backend__2Fuser_2Fblucini","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"127.0.0.1:33115","Pa> Feb 15 21:51:23 AItesting traefik[2012]: time="2021-02-15T21:51:23+01:00" level=info msg="Skipping same configuration for provider file" Feb 15 21:51:23 AItesting traefik[2012]: time="2021-02-15T21:51:23+01:00" level=info msg="Server configuration reloaded on :80" Feb 15 21:51:23 AItesting traefik[2012]: time="2021-02-15T21:51:23+01:00" level=info msg="Server configuration reloaded on :443" Feb 15 21:51:23 AItesting traefik[2012]: time="2021-02-15T21:51:23+01:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099" Feb 15 21:51:25 AItesting traefik[2012]: time="2021-02-15T21:51:25+01:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099" Feb 15 21:51:25 AItesting traefik[2012]: time="2021-02-15T21:51:25+01:00" level=info msg="Server configuration reloaded on :80" Feb 15 21:51:25 AItesting traefik[2012]: time="2021-02-15T21:51:25+01:00" level=info msg="Server configuration reloaded on :443" Feb 15 22:23:02 AItesting traefik[2012]: time="2021-02-15T22:23:02+01:00" level=info msg="Skipping same configuration for provider file" Feb 15 22:23:02 AItesting traefik[2012]: time="2021-02-15T22:23:02+01:00" level=info msg="Server configuration reloaded on :80" Feb 15 22:23:02 AItesting traefik[2012]: time="2021-02-15T22:23:02+01:00" level=info msg="Server configuration reloaded on :443" Feb 15 22:23:02 AItesting traefik[2012]: time="2021-02-15T22:23:02+01:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099" Feb 15 22:23:05 AItesting traefik[2012]: time="2021-02-15T22:23:05+01:00" level=info msg="Server configuration reloaded on :80" Feb 15 22:23:05 AItesting traefik[2012]: time="2021-02-15T22:23:05+01:00" level=info msg="Server configuration reloaded on :443" Feb 15 22:23:05 AItesting traefik[2012]: time="2021-02-15T22:23:05+01:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099" Feb 15 22:23:09 AItesting traefik[2012]: {"BackendAddr":"127.0.0.1:43581","BackendName":"backend__2Fuser_2Frago","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"127.0.0.1:43581","Path"> Feb 15 22:29:39 AItesting traefik[2012]: {"BackendAddr":"127.0.0.1:43581","BackendName":"backend__2Fuser_2Frago","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"127.0.0.1:43581","Path"> root@AItesting:/home# journalctl -u jupyter-rago.service Feb 15 22:41:06 AItesting bash[23379]: [I 2021-02-15 22:41:06.199 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425266090 ([email protected]) 1.86ms Feb 15 22:41:06 AItesting bash[23379]: [I 2021-02-15 22:41:06.290 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernelspecs?1613425266182 ([email protected]) 2.32ms Feb 15 22:41:07 AItesting bash[23379]: [I 2021-02-15 22:41:07.570 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425267462 ([email protected]) 1.62ms Feb 15 22:41:07 AItesting bash[23379]: [I 2021-02-15 22:41:07.753 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425267642 ([email protected]) 6.97ms Feb 15 22:41:12 AItesting bash[23379]: [I 2021-02-15 22:41:12.788 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425272674 ([email protected]) 7.92ms Feb 15 22:41:15 AItesting bash[23379]: [I 2021-02-15 22:41:15.104 SingleUserNotebookApp log:181] 200 GET /user/rago/api/contents/aitrading-repo?content=1&1613425274994 ([email protected]) 3.37ms Feb 15 22:41:16 AItesting bash[23379]: [I 2021-02-15 22:41:16.201 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernels?1613425276091 ([email protected]) 1.85ms Feb 15 22:41:16 AItesting bash[23379]: [I 2021-02-15 22:41:16.290 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425276184 ([email protected]) 2.09ms Feb 15 22:41:17 AItesting bash[23379]: [I 2021-02-15 22:41:17.596 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425277490 ([email protected]) 1.50ms Feb 15 22:41:17 AItesting bash[23379]: [I 2021-02-15 22:41:17.821 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425277708 ([email protected]) 8.63ms Feb 15 22:41:22 AItesting bash[23379]: [I 2021-02-15 22:41:22.854 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425282740 ([email protected]) 7.80ms Feb 15 22:41:25 AItesting bash[23379]: [I 2021-02-15 22:41:25.134 SingleUserNotebookApp log:181] 200 GET /user/rago/api/contents/aitrading-repo?content=1&1613425285024 ([email protected]) 3.26ms Feb 15 22:41:26 AItesting bash[23379]: [I 2021-02-15 22:41:26.227 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernels?1613425286121 ([email protected]) 1.47ms Feb 15 22:41:26 AItesting bash[23379]: [I 2021-02-15 22:41:26.314 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425286208 ([email protected]) 1.50ms Feb 15 22:41:27 AItesting bash[23379]: [I 2021-02-15 22:41:27.622 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425287515 ([email protected]) 1.42ms Feb 15 22:41:27 AItesting bash[23379]: [I 2021-02-15 22:41:27.884 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425287773 ([email protected]) 6.97ms Feb 15 22:41:28 AItesting bash[23379]: [I 2021-02-15 22:41:28.489 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernelspecs?1613425288381 ([email protected]) 3.30ms Feb 15 22:41:32 AItesting bash[23379]: [I 2021-02-15 22:41:32.916 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425292804 ([email protected]) 7.15ms Feb 15 22:41:35 AItesting bash[23379]: [I 2021-02-15 22:41:35.162 SingleUserNotebookApp log:181] 200 GET /user/rago/api/contents/aitrading-repo?content=1&1613425295054 ([email protected]) 2.69ms Feb 15 22:41:36 AItesting bash[23379]: [I 2021-02-15 22:41:36.252 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernels?1613425296146 ([email protected]) 1.45ms Feb 15 22:41:36 AItesting bash[23379]: [I 2021-02-15 22:41:36.340 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425296235 ([email protected]) 1.49ms Feb 15 22:41:37 AItesting bash[23379]: [I 2021-02-15 22:41:37.649 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425297543 ([email protected]) 1.59ms Feb 15 22:41:37 AItesting bash[23379]: [I 2021-02-15 22:41:37.949 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425297836 ([email protected]) 7.20ms Feb 15 22:41:42 AItesting bash[23379]: [I 2021-02-15 22:41:42.982 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425302868 ([email protected]) 6.77ms Feb 15 22:41:45 AItesting bash[23379]: [I 2021-02-15 22:41:45.194 SingleUserNotebookApp log:181] 200 GET /user/rago/api/contents/aitrading-repo?content=1&1613425305084 ([email protected]) 2.79ms Feb 15 22:41:46 AItesting bash[23379]: [I 2021-02-15 22:41:46.284 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernels?1613425306173 ([email protected]) 1.75ms Feb 15 22:41:46 AItesting bash[23379]: [I 2021-02-15 22:41:46.366 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425306261 ([email protected]) 1.69ms Feb 15 22:41:47 AItesting bash[23379]: [I 2021-02-15 22:41:47.674 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425307569 ([email protected]) 1.15ms Feb 15 22:41:48 AItesting bash[23379]: [I 2021-02-15 22:41:48.019 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425307903 ([email protected]) 11.07ms -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/10bc1af5-ec60-46fb-9b58-02fd35e8bcb5n%40googlegroups.com.
