Re: urllib.request.urlopen fails with https (was: Re: Stock quote APi)

2018-03-14 Thread Chris Angelico
On Thu, Mar 15, 2018 at 9:04 AM, Irv Kalb  wrote:
> ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed 
> (_ssl.c:749)
>
> Am I doing something wrong?  Is there another way (besides using the requests 
> module which DOES work for me) to get data from an https URL?

So my reading of this is that you have a major issue with root
certificates. Bleh. The normal response is "use pip to install
something that has your certs", but if you really absolutely cannot do
that, you may have to just disable certificate verification. This is
NOT a good thing to do, and you should have a big noisy comment
explaining why you have to make your code insecure. Put that RIGHT
next to the code that disables verification, so that if anyone copies
and pastes it, they'll have a good chance of seeing it. (Basically,
what you're doing is downgrading the protection of HTTPS to something
nearer plain HTTP. That's fine for what you're doing, but any code you
give to students is likely to be copied and pasted into their
production code.)

See if you can tie in with your OS's cert store first. If you can't,
look at "import ssl" and creating a custom SSL context that doesn't
verify.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


urllib.request.urlopen fails with https (was: Re: Stock quote APi)

2018-03-14 Thread Irv Kalb
Thanks to Chris A and Roger C for their information.  Very helpful and I am 
working on both of their suggestions.  But now I've run into a new related 
problem.

I still am trying to get stock information using Standard Library calls for now 
(although I understand that the requests module makes this kind of thing very 
easy).

My general approach (Python 3) is to build up a URL by concatenating an 
parameters, and make a call to url.request.urlopen.  I then do a 'read' on what 
was returned. For example, in my class, I use the following code to get weather 
data from weatherman.org:

URL = 'http://api.openweathermap.org/data/2.5/weather?q=' + city + 
'=xml'+ '=' + API_KEY
response = urllib.request.urlopen(URL)
responseString = str(response.read())

I then have some code to parse out the pieces of the returned information.  
This works great and an excellent example of the power of using API's with 
Python.


In looking for a stock data API, Chris suggested  alpha vantage.co (and Roger 
made a nice suggestion as to how to parse it).  But when I use their API, I get 
an error:  

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed 
(_ssl.c:749)

I have tried a number of different API's and found that if the URL starts with 
'https', it always fails.  Even a simple example that I found here:  
https://pythonspot.com/urllib-tutorial-python-3/ 
  fails the same way.

Here's what I see when I try that example:

>>> import urllib.request
>>> html = urllib.request.urlopen('https://arstechnica.com').read()
Traceback (most recent call last):
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py",
 line 1318, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py",
 line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py",
 line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py",
 line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py",
 line 1026, in _send_output
self.send(msg)
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py",
 line 964, in send
self.connect()
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py",
 line 1400, in connect
server_hostname=server_hostname)
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 
401, in wrap_socket
_context=self, _session=session)
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 
808, in __init__
self.do_handshake()
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 
1061, in do_handshake
self._sslobj.do_handshake()
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 
683, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed 
(_ssl.c:749)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "", line 1, in 
html = urllib.request.urlopen('https://arstechnica.com').read()
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py",
 line 223, in urlopen
return opener.open(url, data, timeout)
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py",
 line 526, in open
response = self._open(req, data)
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py",
 line 544, in _open
'_open', req)
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py",
 line 504, in _call_chain
result = func(*args)
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py",
 line 1361, in https_open
context=self._context, check_hostname=self._check_hostname)
  File 
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py",
 line 1320, in do_open
raise URLError(err)
urllib.error.URLError: 
>>> 

All my tests return the same tracebacks.

Am I doing something wrong?  Is there another way (besides using the requests 
module which DOES work for me) to get data from an https URL?

Any help would be appreciated.

Thanks,

Irv
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Stock quote API ?

2018-03-11 Thread Chris Angelico
On Mon, Mar 12, 2018 at 3:19 PM, Irv Kalb  wrote:
>
>> On Mar 10, 2018, at 9:26 PM, Chris Angelico  wrote:
>>
>> On Sun, Mar 11, 2018 at 4:18 PM, Irv Kalb  wrote:
>>> Hi,
>>>
>>> I teach courses on beginning Python (Python3).  In one of my topics, I 
>>> explain how we can write simple programs that reach out to the internet and 
>>> download data (request/response).
>>>
>>> I show a number of examples using:   urllib.request.urlopen( 
>>>  )  to get things like weather data, currency exchange 
>>> rates, etc.
>>>
>>> I just tried my examples again, and they are all working fine, except for 
>>> one.  I had an example where I used the call above to get simple (American) 
>>> stock quotes from Yahoo.  However, with this example, now I get a bunch 
>>> errors.  In tracking it down, I found that Yahoo has shut down this public 
>>> API, discontinued this service.
>>>
>>> So ... I am looking for a replacement.  I have done quite a bit of 
>>> searching, but I have not been able to find a simple way to get a stock 
>>> quote (no need for historical data - most recent price is fine).  I have 
>>> found many examples where people have built custom packages for doing this 
>>> type of thing.  However, I am in a college environment, and I cannot 
>>> install any new packages on the computers there.  I've also seen examples 
>>> of people building SQL-style queries to get this type of information, but 
>>> that's beyond what I am trying to teach.
>>>
>>> Wondering if anyone has any example of an API where I could just make a 
>>> call using Python Standard Library interfaces to get stock quotes?
>>>
>>
>> Check out https://www.alphavantage.co/ for something you can query for
>> free. Extensive and amazingly useful. One of my students did some
>> second-tier analysis on the data they provide as a capstone project on
>> stock trading analysis.
>>
>> You may want to consider, though, modifying the "no new packages"
>> rule. The 'requests' library is WAY better for teaching Python and web
>> APIs than the raw urllib. Get just a small handful of pip-installable
>> packages whitelisted and your life will be better.
>>
>> ChrisA
>>
>
> Hi Chris,
>
> Thank you very much for this.  It is very close to what I am looking for.  I 
> had seen this early in my searches but I didn't go into it in detail because 
> it looked like it was designed to give way more information than I was 
> looking for - for example, the first example is about time series data.
>
> I did look into it today, and I got a free API key to check it out.  It does 
> have the ability to give just a stock quote for a symbol, but it looks like 
> the minimum I can get back is a csv:
>
> symbol,price,volume,timestamp
> MSFT,96.1800,--,2018-03-09 16:01:30
>
> which is easy enough for me to break apart.  I just wish there was a way to 
> eliminate the header line so I wouldn't have to go through an explanation 
> about that.
>
> Thanks very much.  If I can't find another one that just give back a price, 
> I'll probably use this one.

It's usually easier to find a decent API that gives too much info than
to find something that gives exactly what you want - because "what you
want" usually isn't the same as "what someone else wants", and the
easiest way to give both is to give lots of info.

> PS:  The "no new packages" rule is not mine.  It's the rule imposed by the 
> college.  They are the administrators of the computers and I don't have an 
> admin password.
>

Oh, I'm sure. But talk to the college about getting a small number of
packages whitelisted. You don't have to ask for arbitrary package
installation, just for a tiny handful of really REALLY useful
packages. If you were teaching data science with Python, would you
restrict yourself to the standard library, or would you use numpy,
pandas, etc? Web development is the same - while you CAN do everything
with just the stdlib, it's way better to pick up a few well-known
packages that tie in well with that.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Stock quote API ?

2018-03-11 Thread Irv Kalb

> On Mar 10, 2018, at 9:26 PM, Chris Angelico  wrote:
> 
> On Sun, Mar 11, 2018 at 4:18 PM, Irv Kalb  wrote:
>> Hi,
>> 
>> I teach courses on beginning Python (Python3).  In one of my topics, I 
>> explain how we can write simple programs that reach out to the internet and 
>> download data (request/response).
>> 
>> I show a number of examples using:   urllib.request.urlopen( 
>>  )  to get things like weather data, currency exchange 
>> rates, etc.
>> 
>> I just tried my examples again, and they are all working fine, except for 
>> one.  I had an example where I used the call above to get simple (American) 
>> stock quotes from Yahoo.  However, with this example, now I get a bunch 
>> errors.  In tracking it down, I found that Yahoo has shut down this public 
>> API, discontinued this service.
>> 
>> So ... I am looking for a replacement.  I have done quite a bit of 
>> searching, but I have not been able to find a simple way to get a stock 
>> quote (no need for historical data - most recent price is fine).  I have 
>> found many examples where people have built custom packages for doing this 
>> type of thing.  However, I am in a college environment, and I cannot install 
>> any new packages on the computers there.  I've also seen examples of people 
>> building SQL-style queries to get this type of information, but that's 
>> beyond what I am trying to teach.
>> 
>> Wondering if anyone has any example of an API where I could just make a call 
>> using Python Standard Library interfaces to get stock quotes?
>> 
> 
> Check out https://www.alphavantage.co/ for something you can query for
> free. Extensive and amazingly useful. One of my students did some
> second-tier analysis on the data they provide as a capstone project on
> stock trading analysis.
> 
> You may want to consider, though, modifying the "no new packages"
> rule. The 'requests' library is WAY better for teaching Python and web
> APIs than the raw urllib. Get just a small handful of pip-installable
> packages whitelisted and your life will be better.
> 
> ChrisA
> 

Hi Chris,

Thank you very much for this.  It is very close to what I am looking for.  I 
had seen this early in my searches but I didn't go into it in detail because it 
looked like it was designed to give way more information than I was looking for 
- for example, the first example is about time series data.  

I did look into it today, and I got a free API key to check it out.  It does 
have the ability to give just a stock quote for a symbol, but it looks like the 
minimum I can get back is a csv:

symbol,price,volume,timestamp
MSFT,96.1800,--,2018-03-09 16:01:30

which is easy enough for me to break apart.  I just wish there was a way to 
eliminate the header line so I wouldn't have to go through an explanation about 
that.  

Thanks very much.  If I can't find another one that just give back a price, 
I'll probably use this one.

Irv

PS:  The "no new packages" rule is not mine.  It's the rule imposed by the 
college.  They are the administrators of the computers and I don't have an 
admin password. 

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Stock quote API ?

2018-03-10 Thread Chris Angelico
On Sun, Mar 11, 2018 at 4:18 PM, Irv Kalb  wrote:
> Hi,
>
> I teach courses on beginning Python (Python3).  In one of my topics, I 
> explain how we can write simple programs that reach out to the internet and 
> download data (request/response).
>
> I show a number of examples using:   urllib.request.urlopen( 
>  )  to get things like weather data, currency exchange 
> rates, etc.
>
> I just tried my examples again, and they are all working fine, except for 
> one.  I had an example where I used the call above to get simple (American) 
> stock quotes from Yahoo.  However, with this example, now I get a bunch 
> errors.  In tracking it down, I found that Yahoo has shut down this public 
> API, discontinued this service.
>
> So ... I am looking for a replacement.  I have done quite a bit of searching, 
> but I have not been able to find a simple way to get a stock quote (no need 
> for historical data - most recent price is fine).  I have found many examples 
> where people have built custom packages for doing this type of thing.  
> However, I am in a college environment, and I cannot install any new packages 
> on the computers there.  I've also seen examples of people building SQL-style 
> queries to get this type of information, but that's beyond what I am trying 
> to teach.
>
> Wondering if anyone has any example of an API where I could just make a call 
> using Python Standard Library interfaces to get stock quotes?
>

Check out https://www.alphavantage.co/ for something you can query for
free. Extensive and amazingly useful. One of my students did some
second-tier analysis on the data they provide as a capstone project on
stock trading analysis.

You may want to consider, though, modifying the "no new packages"
rule. The 'requests' library is WAY better for teaching Python and web
APIs than the raw urllib. Get just a small handful of pip-installable
packages whitelisted and your life will be better.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list