Re: urllib.request.urlopen fails with https - SOLVED

[Irv Kalb]

Thank you, thank you, thank you.

That fixed it (at least on my computer, I'll see if I can do that at my school).

Irv


> On Mar 15, 2018, at 7:39 PM, Ned Deily  wrote:
> 
> On 2018-03-14 18:04, Irv Kalb wrote:
>> File 
>> "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py",
>>  line 1320, in do_open
>>   raise URLError(err)
>> urllib.error.URLError: > certificate verify failed (_ssl.c:749)>
> 
> If you are using Python 3.6 for macOS from a python.org installer, did
> you follow the instructions displayed in the installer ReadMe and also
> saved at:
> 
> /Applications/Python 3.6/ReadMe.rtf
> 
> to run the "Install Certificates.command" ?
> 
> Either double-click on it in the Finder or, from a shell command line, type:
> 
>   open "/Applications/Python 3.6/Install Certificates.command"
> 
> 
> Certificate verification and OpenSSL
> 
> **NEW** This variant of Python 3.6 now includes its own private copy of
> OpenSSL 1.0.2.  Unlike previous releases, the deprecated Apple-supplied
> OpenSSL libraries are no longer used.  This also means that the trust
> certificates in system and user keychains managed by the Keychain Access
> application and the security command line utility are no longer used as
> defaults by the Python ssl module.  For 3.6.0, a sample command script
> is included in /Applications/Python 3.6 to install a curated bundle of
> default root certificates from the third-party certifi package
> (https://pypi.python.org/pypi/certifi).  If you choose to use certifi,
> you should consider subscribing to the project's email update service to
> be notified when the certificate bundle is updated.
> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: urllib.request.urlopen fails with https

[Ned Deily]

On 2018-03-14 18:04, Irv Kalb wrote:
>   File 
> "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py",
>  line 1320, in do_open
> raise URLError(err)
> urllib.error.URLError:  certificate verify failed (_ssl.c:749)>

If you are using Python 3.6 for macOS from a python.org installer, did
you follow the instructions displayed in the installer ReadMe and also
saved at:

/Applications/Python 3.6/ReadMe.rtf

to run the "Install Certificates.command" ?

Either double-click on it in the Finder or, from a shell command line, type:

open "/Applications/Python 3.6/Install Certificates.command"


Certificate verification and OpenSSL

**NEW** This variant of Python 3.6 now includes its own private copy of
OpenSSL 1.0.2.  Unlike previous releases, the deprecated Apple-supplied
OpenSSL libraries are no longer used.  This also means that the trust
certificates in system and user keychains managed by the Keychain Access
application and the security command line utility are no longer used as
defaults by the Python ssl module.  For 3.6.0, a sample command script
is included in /Applications/Python 3.6 to install a curated bundle of
default root certificates from the third-party certifi package
(https://pypi.python.org/pypi/certifi).  If you choose to use certifi,
you should consider subscribing to the project's email update service to
be notified when the certificate bundle is updated.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: urllib.request.urlopen fails with https

[Irv Kalb]

On Mar 14, 2018, at 9:54 PM, Gregory Ewing  wrote:
> 
> Chris Angelico wrote:
>> That means going back to the original problem: "how do we get a usable
>> stock price API?".
> 
> Does it have to be stock prices in particular?
> Or just some simple piece of data that demonstrates
> the principles of fetching a url and parsing the
> result?
> 
> -- 
> Greg
> -- 

I am the OP.  No, it does not have to be stock prices.  I used stock prices 
because it made for a very clear example in my curriculum.  In my class, I demo 
how we can write a small Python program to "pretend to be a browser" and get a 
full page of html, then I talk about how you can use an API with a small Python 
program to get just the data you really want.  I found a Yahoo API that did 
exactly what I wanted, and that has worked for years - but now Yahoo has 
cancelled that service.  I thought it would be simple to find another simple 
API to get a stock price but I haven't found one.  All the ones that do work 
seem to require https, which doesn't work easily with url lib.request.urlopen.

I do use other API's like openweathermap.org to get weather data and 
api.fixer.io  to get currency exchange rates.   I get the 
data back and work through parsing the results.

I am still very interested in getting a stock quote example but If you have 
other examples that have a URL that is http based, I am interested. 

Irv
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: urllib.request.urlopen fails with https

[Chris Angelico]

On Thu, Mar 15, 2018 at 3:54 PM, Gregory Ewing
 wrote:
> Chris Angelico wrote:
>>
>> That means going back to the original problem: "how do we get a usable
>> stock price API?".
>
>
> Does it have to be stock prices in particular?
> Or just some simple piece of data that demonstrates
> the principles of fetching a url and parsing the
> result?
>

You'd have to ask the OP, but if you make too big a change to the API
used, the rest of the course might have to change too. If the next
thing done with the downloaded data is to graph it, for instance, then
you'll need some other API that gives graphable data. Etcetera,
etcetera, etcetera.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: urllib.request.urlopen fails with https

[Gregory Ewing]

Chris Angelico wrote:

That means going back to the original problem: "how do we get a usable
stock price API?".


Does it have to be stock prices in particular?
Or just some simple piece of data that demonstrates
the principles of fetching a url and parsing the
result?

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: urllib.request.urlopen fails with https

[Chris Angelico]

On Thu, Mar 15, 2018 at 10:27 AM, Gregory Ewing
 wrote:
> Chris Angelico wrote:
>>
>> (Basically,
>> what you're doing is downgrading the protection of HTTPS to something
>> nearer plain HTTP. That's fine for what you're doing, but any code you
>> give to students is likely to be copied and pasted into their
>> production code.)
>>
>> See if you can tie in with your OS's cert store first. If you can't,
>> look at "import ssl" and creating a custom SSL context that doesn't
>> verify.
>
>
> That's likely to distract and confuse students as well.
>
> I would suggest looking for a different example that
> doesn't require interacting with an https site.

That means going back to the original problem: "how do we get a usable
stock price API?".

My preferred solution is to just install 'requests', of course.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: urllib.request.urlopen fails with https

[Gregory Ewing]

Chris Angelico wrote:

(Basically,
what you're doing is downgrading the protection of HTTPS to something
nearer plain HTTP. That's fine for what you're doing, but any code you
give to students is likely to be copied and pasted into their
production code.)

See if you can tie in with your OS's cert store first. If you can't,
look at "import ssl" and creating a custom SSL context that doesn't
verify.


That's likely to distract and confuse students as well.

I would suggest looking for a different example that
doesn't require interacting with an https site.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: urllib.request.urlopen fails with https (was: Re: Stock quote APi)

[Chris Angelico]

On Thu, Mar 15, 2018 at 9:04 AM, Irv Kalb  wrote:
> ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed 
> (_ssl.c:749)
>
> Am I doing something wrong?  Is there another way (besides using the requests 
> module which DOES work for me) to get data from an https URL?

So my reading of this is that you have a major issue with root
certificates. Bleh. The normal response is "use pip to install
something that has your certs", but if you really absolutely cannot do
that, you may have to just disable certificate verification. This is
NOT a good thing to do, and you should have a big noisy comment
explaining why you have to make your code insecure. Put that RIGHT
next to the code that disables verification, so that if anyone copies
and pastes it, they'll have a good chance of seeing it. (Basically,
what you're doing is downgrading the protection of HTTPS to something
nearer plain HTTP. That's fine for what you're doing, but any code you
give to students is likely to be copied and pasted into their
production code.)

See if you can tie in with your OS's cert store first. If you can't,
look at "import ssl" and creating a custom SSL context that doesn't
verify.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list