Do you have an example of a script where you can see correct data going
in and a bad result coming out?
Henry Rich
On 11/24/2019 9:30 PM, Thomas McGuire wrote:
The following J script, patterned after Brad Lucas’ Bash script
(http://blog.bradlucas.com/posts/2017-06-02-new-yahoo-finance-quote-download-url/
<http://blog.bradlucas.com/posts/2017-06-02-new-yahoo-finance-quote-download-url/>),
doesn’t seem to work. I won’t go into the history of the yahoo and the change in
their website except to say that by making a double access with curl and pulling out
elements of the first response to use in the second access you should be able to
programmatically get historical price quotes.
I believe I have set up the URLs properly. I am using cookies with
web/gethttp function. I see the cookie file get created and I receive a first
response from the yahoo website. From there you need to find the ‘crumb’ in the
response and use that to build a URL to request the historical stock prices.
The crumb can sometimes contain \u00xx unicode escape characters. These need to
be translated into a character. Bash has a built in echo command (different
than the linux echo command) that will translate this properly. In J I use
reapply to translate these sequences. The Bash version redirects the output of
the second curl call to a file. I am just trying to get the data to print in
Jqt where I run the script.
I will show the output and the J script will follow. I tried this with 2!:0
foreign conjunction to run curl directly and have the same problem. The first
call to curl gets an appropriate response and the second call gets an error
stating invalid cookie. This error is not displayed in J I modified the code to
log the errors to a file and found it there.
OUTPUT:
gethistorical 'AAPL';'11/23/2018';'11/23/2019'
start URL = https://finance.yahoo.com/quote/AAPL/history?p=AAPL
raw crumb string = "}}}}},"CrumbStore":{"crumb":"34jVbQ1iOCI"},"StreamStore":{"
final crumb string = 34jVbQ1iOCI
https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1543010400&period2=1574546400&interval=1d&events=history&crumb=34jVbQ1iOCI
OUTPUT (with unicode escape sequence):
gethistorical 'AAPL';'11/23/2018';'11/23/2019'
start URL = https://finance.yahoo.com/quote/AAPL/?p=AAPL
raw crumb string = "}}}}},"CrumbStore":{"crumb":"yw3klV8Pa\u002Fc"},"StreamStor
final crumb string = yw3klV8Pa/c
https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1543010400&period2=1574546400&interval=1d&events=history&crumb=yw3klV8Pa/c
JCODE:
NB. Navigating yahoo.com to programmatically get historical stock prices
NB.
require 'web/gethttp'
require 'regex'
NB.require 'types/datetime'
NB. time stamp functions
NB. jts =: 6!:0
NB. epoch =: todayno 1970 01 01
NB. linuxts =: 13 : '(((toDayNo y)-epoch)*24*60*60)'
NB. use the linux date command to create a linux time stamp
epochtime =: 3 : 0
2!:0 'date -jf ''%m/%d/%Y %H:%M:%S %p'' ''',y,' 05:00:00 PM'' ''+%s'''
)
NB. precision functions
ppq =: 9 !: 10 NB. print-precision query
pps =: 9 !: 11 NB. print-precision set
NB. I set the precision to 16 to ensure full printing of the linux timestamps
HEX=:16#.'0123456789abcdef'i.]
xutf =: 3 : 0
u: HEX tolower 2 }. y
)
crumbstr =: '"CrumbStore":{"crumb":"'
NB. the crumb is on the page with the link to downloading the historical
NB. data. If you call the correct first page you only need to search
NB. for the above crumbstr there will be only one.
getcrumb =: 3 : 0
sidx =. (#crumbstr)+({: I. crumbstr E. y)
sstr =. (sidx + i. 30){y
smoutput 'raw crumb string = ',(sidx + _30 + i.60){y
eidx =. {. I. '"' E. sstr
NB. using rxapply convert all \u00xx unicode escape sequences
crumb =. '(\\u[0-9a-fA-F][0-9a-fA-F][0-9a-fA-F][0-9a-fA-F])' xutf rxapply
(i.eidx){sstr
)
financeURL =: 'https://finance.yahoo.com/quote/' NB. AAPL/history?p=AAPL'
histURL =: 'https://query1.finance.yahoo.com/v7/finance/download/'
NB. the histURL needs to have a ticker symbol followed by:
NB. ?period1=<unixts for p1>&period2=<unixts for
p2&interval=1d&events=history&crumb=<crumbval>
NB.
https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1543024670&period2=1574560670&interval=1d&events=history&crumb=jZO816Y7CSK
gethistorical=: 3 : 0
'symbol d1 d2' =. y
NB. Create start URL at the start page with the crumb to get historical
downloadNN
NB. a BASH implementation uses the commented out format
NB. sURL =. financeURL,symbol,'/?p=',symbol
NB. But the link to the download of historical prices is really here:
sURL =. financeURL,symbol,'/history?p=',symbol
smoutput 'start URL = ',sURL
NB. Get the response using gethttp. --cookie-jar cookie.txt will open a cookie
file
res =: '-s -c cookie.txt' gethttp sURL
NB. tried with curl through the 2!:0 interface with the same results
NB. res =: 2!:0 'curl -s -c cookie.txt ',sURL
crumb =. getcrumb res
smoutput 'final crumb string = ',crumb
qstr =. '?period1=',(}:epochtime d1),'&period2=',(}:epochtime
d2),'&interval=1d&events=history&crumb=',crumb
smoutput URL=. histURL,symbol,qstr
res2 =: '-s -b cookie.txt ' gethttp URL
NB. res2 =: 2!:0 'curl -s -b cookie.txt ',URL
res2
)
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
