On Thu, Jan 12, 2017 at 5:18 AM, Christopher Painter-Wakefield < [email protected]> wrote:
Hi all, > > I'm having an issue that I can't seem to find any reference to anywhere > else, but I'm hoping someone has seen it and can help me. > > I've set up jupyterhub with the sudospawner according to the instructions > on the wiki, on an ubuntu server. The user logins for this server > (including myself) are actually provided via some directory service voodoo > (active directory maybe?) that our institution sysadmins manage and that I > know nothing about. > > With the sudospawner, jupyterhub works fine for me logging in as myself. > My users can also login just fine (that is, they authenticate without > problem), but then they get a 500 Server Error message and it appears their > single user server is not starting up. There's nothing in the jupyterhub > log that seems relevant, but I'll post it below. > > In desperation I tried a number of things, and one thing seemed to work: > having a user ssh to the host and launch 'jupyter notebook' once. After > this (after shutting down the notebook server thus created) they were able > to successfully go through jupyterhub. I figured this had something to do > with the files created in .local/share/juptyerhub, so I tried destroying > those files for my own account in hopes of reproducing the problem, but I > can't reproduce - my account stubbornly refuses to not work :) I've not > yet reproduced this solution with another user (hope to try tomorrow, but I > need to resolve this ASAP). So I'm not certain this is the issue (maybe > just a coincidence). Perhaps the issue has something to do with the way > users are managed. If it isn't a coincidence, then I'm hoping someone can > tell me a workaround so all my users (~50 of them) don't have to ssh and > launch jupyter notebook, etc. Some of them will not be comfortable > performing this task! (I cannot do it for them with sudo without violating > the rules of my institution.) > How are you capturing the JupyterHub logs? Are you redirecting output, or configuring a log file? Redirecting output is a better practice, as it captures all output from the process tree, rather than only the explicit log statements from the Hub. This might be why you are not seeing info from single-user servers. To run JupyterHub with a log file: jupyterhub 2>&1 &>> /var/log/jupyterhub.log If this works for some users and not others, it could be a permissions issue. Check: stat -F $(which -a jupyterhub-singleuser) stat -F $(which -a sudospawner) stat -F $(python3 -c 'import jupyterhub; print(jupyterhub.__file__)') You need to make sure that all users have access to these files. If you’ve done some installation as you, it’s possible that one or more of these is only installed with permissions for you and/or admin. Since it seems to run successfully after a single notebook startup, it may have something to do with first-run output. Was anyone able to test having just ssh’d in, but not calling jupyter notebook? It’s possible that it’s a first-run issue caused by the first login, and not related to Jupyter. This PR <https://github.com/jupyterhub/sudospawner/pull/30> will log the output that’s failing to parse, which might have us a hint. You can install that branch directly with: pip install https://github.com/minrk/sudospawner/archive/log-failed-output.tar.gz > Here's the output of 'jupyter troubleshoot': > $PATH: > /usr/local/bin > /u/af/ci/<me>/bin > /u/af/ci/<me>/.local/bin > /usr/local/sbin > /usr/local/bin > /usr/sbin > /usr/bin > /sbin > /bin > /usr/games > /usr/local/games > /snap/bin > > sys.path: > /usr/local/bin > /usr/lib/python35.zip > /usr/lib/python3.5 > /usr/lib/python3.5/plat-x86_64-linux-gnu > /usr/lib/python3.5/lib-dynload > /usr/local/lib/python3.5/dist-packages > /usr/lib/python3/dist-packages > > sys.executable: > /usr/bin/python3 > > sys.version: > 3.5.2 (default, Nov 17 2016, 17:05:23) > [GCC 5.4.0 20160609] > > platform.platform(): > Linux-4.4.0-59-generic-x86_64-with-Ubuntu-16.04-xenial > > which -a jupyter: > /usr/local/bin/jupyter > /usr/local/bin/jupyter > > pip list: > alembic (0.8.9) > chardet (2.3.0) > command-not-found (0.3) > cycler (0.10.0) > decorator (4.0.10) > entrypoints (0.2.2) > ipykernel (4.5.2) > ipython (5.1.0) > ipython-genutils (0.1.0) > ipywidgets (5.2.2) > Jinja2 (2.8) > jsonschema (2.5.1) > jupyter (1.0.0) > jupyter-client (4.4.0) > jupyter-console (5.0.0) > jupyter-core (4.2.1) > jupyterhub (0.7.0) > language-selector (0.1) > Mako (1.0.6) > MarkupSafe (0.23) > matplotlib (1.5.3) > mistune (0.7.3) > nbconvert (4.2.0) > nbformat (4.2.0) > notebook (4.3.0) > numpy (1.11.2) > pamela (0.3.0) > pandas (0.19.2) > pexpect (4.2.1) > pickleshare (0.7.4) > pip (9.0.1) > prompt-toolkit (1.0.9) > ptyprocess (0.5.1) > pycurl (7.43.0) > Pygments (2.1.3) > pygobject (3.20.0) > pyparsing (2.1.10) > python-apt (1.1.0b1) > python-dateutil (2.6.0) > python-debian (0.1.27) > python-editor (1.0.3) > python-systemd (231) > pytz (2016.10) > pyzmq (16.0.2) > qtconsole (4.2.1) > requests (2.9.1) > rise (4.0.0b1) > scikit-learn (0.18.1) > scipy (0.18.1) > seaborn (0.7.1) > setuptools (20.7.0) > simplegeneric (0.8.1) > six (1.10.0) > SQLAlchemy (1.1.4) > ssh-import-id (5.5) > sudospawner (0.3.0) > terminado (0.6) > tornado (4.4.2) > traitlets (4.3.1) > ufw (0.35) > unattended-upgrades (0.1) > urllib3 (1.13.1) > wcwidth (0.1.7) > wheel (0.29.0) > widgetsnbextension (1.2.6) > > > and here's the relevant portion of the log: > > Traceback (most recent call last): > File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line > 1469, i > n _execute > result = yield result > File "/usr/local/lib/python3.5/dist-packages/jupyterhub/ > handlers/login.py" > , line 84, in post > yield self.spawn_single_user(user) > File > "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", > line 328, in spawn_single_user > yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), > f) > File "/usr/local/lib/python3.5/dist-packages/jupyterhub/user.py", > line 255, in spawn > raise e > File "/usr/local/lib/python3.5/dist-packages/jupyterhub/user.py", > line 229, in spawn > ip_port = yield gen.with_timeout(timedelta( > seconds=spawner.start_timeout), f) > File "/usr/local/lib/python3.5/dist-packages/sudospawner/spawner.py", > line 83, in start > reply = yield self.do(action='spawn', args=self.get_args(), > env=self.get_env()) > File "/usr/local/lib/python3.5/dist-packages/sudospawner/spawner.py", > line 69, in do > data_str = data_str[data_str.index('{'):data_str.rindex('}')+1] > ValueError: substring not found > > > Thanks for any help!!! > > -Christopher > > -- > You received this message because you are subscribed to the Google Groups > "Project Jupyter" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/jupyter/99ad72b6-3cd3-48d9-9cf3-e00830c2b4d7%40googlegroups.com > <https://groups.google.com/d/msgid/jupyter/99ad72b6-3cd3-48d9-9cf3-e00830c2b4d7%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAHNn8BXv9t_9Q%3DsZ1M0vhriwqnjNvd3FR3kHscAXJoWS3T8cFQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
