Well to try to figure this out I set up another server and have tracked
down the offending issue, though I am not at all clear why this is causing
the problem. Basically I had been following the instructions here,
https://github.com/jupyterhub/jupyterhub/wiki/Deploying-JupyterHub-on-AWS
And after some experimentation it seems that the problem is the iptables
rerouting, specifially this instruction
- Use iptable to NAT 8443 to https
$ sudo iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT
--to 8443
If I skip this step (or delete the forwarding) then HTTPS/SSL handshakes
still work fine from my docker containers. So what I am doing for a
workaround now is to not run jupyterhub with HTTPS at all, instead its just
running --no-ssl and then I have an AWS load balancer set up to let me
accessit over https (and the non-ssl port is only open to other ips in my
VPC so that all external traffic must come through the https load balancer)
If you have any idea why this iptable routing is messing up the docker ssl
connections, I'd love to hear it
Thanks
On Tuesday, October 4, 2016 at 9:43:13 AM UTC-7, Ted Liefeld wrote:
>
> On the hub AMI I see the correct behavior for this. However if I shell
> into one of the jupyterhub/singleuser docker containers its getting the
> certificate for the hub. From this post it looks like there might be
> something odd in the iptables
>
>
> http://stackoverflow.com/questions/37970990/ssl-certificate-verification-fails-inside-docker-container-on-specific-server
>
> however when I look at the container it has no iptables and on the hub I
> can't quite see anything that explains it (included below the curl output
> below)
>
> I am going to try to setup another hub to see if this repeats since I am
> running out of other ideas at this point. I think I'll try Amazon linux
> instead of ubuntu for this second try
>
> jovyan@0955ba141195:~/work$ curl -v https://github.com >
> /dev/null
>
> * Rebuilt URL to: https://github.com/
> * Hostname was NOT found in DNS cache
> % Total % Received % Xferd Average Speed Time Time Time
> Current
> Dload Upload Total Spent Left
> Speed
> 0 0 0 0 0 0 0 0 --:--:-- --:--:--
> --:--:-- 0* Trying 192.30.253.113...
> * Connected to github.com (192.30.253.113) port 443 (#0)
> * successfully set certificate verify locations:
> * CAfile: /etc/ssl/certs/ca-certificates.crt
> CApath: /etc/ssl/certs
> * SSLv3, TLS handshake, Client hello (1):
> } [data not shown]
> * SSLv3, TLS handshake, Server hello (2):
> { [data not shown]
> * SSLv3, TLS handshake, CERT (11):
> { [data not shown]
> * SSLv3, TLS handshake, Server finished (14):
> { [data not shown]
> * SSLv3, TLS handshake, Client key exchange (16):
> } [data not shown]
> * SSLv3, TLS change cipher, Client hello (1):
> } [data not shown]
> * SSLv3, TLS handshake, Finished (20):
> } [data not shown]
> * SSLv3, TLS change cipher, Client hello (1):
> { [data not shown]
> * SSLv3, TLS handshake, Finished (20):
> { [data not shown]
> * SSL connection using TLSv1.2 / AES256-GCM-SHA384
> * Server certificate:
> * subject: CN=jupyterhubtest.genomespace.org
> * start date: 2016-08-01 19:53:00 GMT
> * expire date: 2016-10-30 19:53:00 GMT
> * subjectAltName does not match github.com
> * SSL: no alternative certificate subject name matches target host name '
> github.com'
> * Closing connection 0
> * SSLv3, TLS alert, Client hello (1):
> } [data not shown]
> curl: (51) SSL: no alternative certificate subject name matches target
> host name 'github.com'
>
> ========= iptable on the hub ============
>
> sudo iptables --list
> Chain INPUT (policy ACCEPT)
> target prot opt source destination
>
> Chain FORWARD (policy ACCEPT)
> target prot opt source destination
> DOCKER-ISOLATION all -- anywhere anywhere
> DOCKER all -- anywhere anywhere
> ACCEPT all -- anywhere anywhere ctstate
> RELATED,ESTABLISHED
> ACCEPT all -- anywhere anywhere
> ACCEPT all -- anywhere anywhere
>
> Chain OUTPUT (policy ACCEPT)
> target prot opt source destination
>
> Chain DOCKER (1 references)
> target prot opt source destination
> ACCEPT tcp -- anywhere ip-172-17-0-2.ec2.internal tcp
> dpt:8888
> ACCEPT tcp -- anywhere ip-172-17-0-3.ec2.internal tcp
> dpt:8888
>
> Chain DOCKER-ISOLATION (1 references)
> target prot opt source destination
> RETURN all -- anywhere anywhere
>
>
>
>
>
>
>
> On Tuesday, October 4, 2016 at 4:18:59 AM UTC-7, Min RK wrote:
>>
>> What do you see in
>>
>> !curl -v https://github.com > /dev/null
>>
>> ?
>>
>> It’s certainly odd that requests is connecting to something funny.
>>
>>
>> On Mon, Oct 3, 2016 at 10:37 PM, Ted Liefeld <lie...@broadinstitute.org>
>> wrote:
>>
>>> Min
>>>
>>> here is what I see
>>>
>>> environ({
>>> 'HOSTNAME': '0955ba141195',
>>> 'GS_CODE_ROOT_DIR': '/combined',
>>> 'LANGUAGE': 'en_US.UTF-8',
>>> 'PATH':
>>> '/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
>>>
>>> 'LANG': 'en_US.UTF-8',
>>> 'CLICOLOR': '1',
>>> 'JPY_HUB_API_URL': 'http://10.0.2.188:8081/hub/api',
>>> 'JPY_BASE_URL': '/user/ted',
>>> 'HOME': '/home/jovyan',
>>> 'JPY_COOKIE_NAME': 'jupyter-hub-token-ted',
>>> 'PWD': '/home/jovyan/work',
>>> 'JPY_HUB_PREFIX': '/hub/',
>>> 'NB_USER': 'jovyan',
>>> 'CONDA_DIR': '/opt/conda',
>>> 'JPY_USER': 'ted',
>>> 'GIT_PAGER': 'cat',
>>> 'PYTHONPATH': '/combined/jupyter-notebook/profile/extensions',
>>> 'DEBIAN_FRONTEND': 'noninteractive',
>>> 'JPY_PARENT_PID': '5',
>>> 'TERM': 'xterm-color',
>>> 'GS_NOTEBOOK_EXTENSION_DIR':
>>> '/combined/jupyter-notebook/profile/extensions',
>>> 'PAGER': 'cat',
>>> 'NB_UID': '1000',
>>> 'LC_ALL': 'en_US.UTF-8',
>>> 'SHELL': '/bin/bash'})
>>>
>>> and the nslookup is the same in both the docker container and the host AMI
>>>
>>> $ nslookup github.com
>>> Server: 10.0.0.2
>>> Address: 10.0.0.2#53
>>>
>>> Non-authoritative answer:
>>> Name: github.com
>>> Address: 192.30.253.112
>>>
>>>
>>> FWIW this is running on the ubuntu AMI on AWS taht was published with
>>> Jupyterhub in it (
>>> ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20160627
>>> (ami-2d39803a)
>>> )
>>>
>>>
>>> On Monday, October 3, 2016 at 2:13:30 AM UTC-7, Min RK wrote:
>>>>
>>>> That's exceedingly strange, and I have not seen it before. The proxy is
>>>> involved when your browser connects to the notebook server, but it
>>>> shouldn't be involved at all when the kernel connects to the outside
>>>> world.
>>>> Can you check `os.environ` to see if there are any relevant environment
>>>> variables that might be sneaking in here? You might also check things like
>>>> `nslookup github.com` to make sure DNS isn't confused.
>>>>
>>>> -Min
>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Project Jupyter" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to jupyter+u...@googlegroups.com.
>>> To post to this group, send email to jup...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/jupyter/a25894e4-82af-4a9e-a993-0426ad6a2437%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/jupyter/a25894e4-82af-4a9e-a993-0426ad6a2437%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jupyter+unsubscr...@googlegroups.com.
To post to this group, send email to jupyter@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jupyter/25370d6e-8771-4cc6-8c20-0dd398f4228f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
