This issue was now solved in TTR::stockSymbols() by package author https://github.com/joshuaulrich/TTR/commit/98dec2b5aa68c3cee750397c7d11b164895e0140
Thanks for all the help and ideas. Best, Sam On Fri, Jul 17, 2020, 13:54 Sam H <sam.h...@gmail.com> wrote: > Hello, > > Thank you very much to you all to look into this. > > I came across this problem when I was using TTR::stockSymbols() ( > https://github.com/joshuaulrich/TTR/blob/e6609b9f7621f3a4b1a204c159af61aebc89997e/R/WebData.R) > . > > As a workaround I added this function to my private R package and instead > of read.csv I am now using data.table::fread() which properly (without > failing) downloads the file and reads it. > > Best, > Sam > > > On Fri, Jul 17, 2020 at 4:30 AM ruipbarradas <ruipbarra...@sapo.pt> wrote: > >> Hello, >> >> Yes, I thought it's a site policy issue too. But the file can be accessed >> and read/downloaded from RStudio and Firefox so apparently there's no >> reason why R console shouldn't . >> >> Anyway, I believe it's time for the OP to say someyhing, maybe he has >> solved it and there's no point in continuing. >> >> Rui Barradas >> >> >> >> >> >> Enviado a partir do meu smartphone Samsung Galaxy. >> -------- Mensagem original -------- >> De: luke-tier...@uiowa.edu >> Data: 17/07/2020 02:59 (GMT+00:00) >> Para: Ista Zahn <istaz...@gmail.com> >> Cc: Rui Barradas <ruipbarra...@sapo.pt>, r-help@r-project.org, Sam H < >> sam.h...@gmail.com> >> Assunto: Re: [External] Re: [R] read.csv fails in R console in Ubuntu >> terminal but works in RStudio after R 3.6.3 upgrade to R 4.0.2? >> >> On my Ubuntu system the download with read.csv succeeds in an R >> console if I set the HTTPUserAgent and download.file.method options to >> match the ones used by RStudio. >> >> Given how picky the server is being I would worry about whether this >> use is in line with the site's terms of service. >> >> Best, >> >> luke >> >> On Thu, 16 Jul 2020, Ista Zahn wrote: >> >> > On Thu, Jul 16, 2020 at 5:15 PM Ista Zahn <istaz...@gmail.com> wrote: >> >> >> >> On Thu, Jul 16, 2020 at 8:18 AM Rui Barradas <ruipbarra...@sapo.pt> >> wrote: >> >>> >> >>> Hello, >> >>> >> >>> Thanks, but no, download.file still gives 403 Forbidden with both >> method >> >>> = "libcurl" and method = "wget". >> >> >> >> I think that makes it "not an R question". Ask on >> >> https://unix.stackexchange.com/ maybe? >> > >> > Oh, sorry I misread your message. Nevertheless: >> > >> > $ curl " >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> " >> > <HTML><HEAD> >> > <TITLE>Access Denied</TITLE> >> > </HEAD><BODY> >> > <H1>Access Denied</H1> >> > >> > You don't have permission to access >> > >> "http://old.nasdaq.com/screening/companies-by-name.aspx?" >> > on this server.<P> >> > Reference #18.5506d217.1594934303.938edcb >> > </BODY> >> > </HTML> >> > >> > $ wget " >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> " >> > --2020-07-16 17:19:12-- >> > >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> > Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt' >> > Resolving old.nasdaq.com (old.nasdaq.com)... 2600:1400:9000:28f::1b46, >> > 2600:1400:9000:29b::1b46, 23.78.161.120 >> > Connecting to old.nasdaq.com >> > (old.nasdaq.com)|2600:1400:9000:28f::1b46|:443... connected. >> > HTTP request sent, awaiting response... 403 Forbidden >> > 2020-07-16 17:19:12 ERROR 403: Forbidden. >> > >> > I don't think this is an R problem. >> > >> > Best, >> > Ista >> > >> >> >> >> Best, >> >> Ista >> >> >> >>> >> >>> Rui Barradas >> >>> >> >>> Às 05:31 de 16/07/20, Jeff Newmiller escreveu: >> >>>> Perhaps read FAQ 7.43? [1] >> >>>> >> >>>> [1] >> https://cran.r-project.org/doc/FAQ/R-FAQ.html#How-can-I-enable-secure-https-downloads-in-R_003f >> >>>> >> >>>> On July 15, 2020 4:02:27 PM PDT, Rui Barradas <ruipbarra...@sapo.pt> >> wrote: >> >>>>> Hello, >> >>>>> >> >>>>> R 4.0.2 on Ubuntu 20.04 LTS, sessionInfo below. >> >>>>> >> >>>>> I'm also unable to read the file with Rscript from the Ubuntu >> terminal >> >>>>> but the error is not the same as the OP's. >> >>>>> >> >>>>> >> >>>>> The first try was a file test1.R with the following commands. >> >>>>> >> >>>>> x<-" >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> " >> >>>>> read.csv(x, as.is=TRUE, na="n/a") >> >>>>> >> >>>>> >> >>>>> And run with Rscript >> >>>>> >> >>>>> rui@rui:~$ Rscript --vanilla test1.R >> >>>>> Error in file(file, "rt") : >> >>>>> cannot open the connection to >> >>>>> ' >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> ' >> >>>>> Calls: read.csv -> read.table -> file >> >>>>> In addition: Warning message: >> >>>>> In file(file, "rt") : >> >>>>> cannot open URL >> >>>>> ' >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> ': >> >>>>> >> >>>>> HTTP status was '403 Forbidden' >> >>>>> Execution halted >> >>>>> >> >>>>> >> >>>>> >> >>>>> The second try was download.file() and then read it. >> >>>>> File test2.R is: >> >>>>> >> >>>>> x<-" >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> " >> >>>>> download.file(x, "companylist.csv") >> >>>>> read.csv("companylist.csv", as.is=TRUE, na="n/a") >> >>>>> >> >>>>> >> >>>>> But this too failed with error 403 Forbiden. >> >>>>> >> >>>>> rui@rui:~$ Rscript --vanilla test2.R >> >>>>> trying URL >> >>>>> ' >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> ' >> >>>>> Error in download.file(x, "companylist.csv") : >> >>>>> cannot open URL >> >>>>> ' >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> ' >> >>>>> In addition: Warning message: >> >>>>> In download.file(x, "companylist.csv") : >> >>>>> cannot open URL >> >>>>> ' >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> ': >> >>>>> >> >>>>> HTTP status was '403 Forbidden' >> >>>>> Execution halted >> >>>>> >> >>>>> >> >>>>> This is my session info. >> >>>>> >> >>>>> rui@rui:~$ Rscript --vanilla -e 'sessionInfo()' >> >>>>> R version 4.0.2 (2020-06-22) >> >>>>> Platform: x86_64-pc-linux-gnu (64-bit) >> >>>>> Running under: Ubuntu 20.04 LTS >> >>>>> >> >>>>> Matrix products: default >> >>>>> BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0 >> >>>>> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0 >> >>>>> >> >>>>> locale: >> >>>>> [1] LC_CTYPE=pt_PT.UTF-8 LC_NUMERIC=C >> >>>>> [3] LC_TIME=pt_PT.UTF-8 LC_COLLATE=pt_PT.UTF-8 >> >>>>> [5] LC_MONETARY=pt_PT.UTF-8 LC_MESSAGES=pt_PT.UTF-8 >> >>>>> [7] LC_PAPER=pt_PT.UTF-8 LC_NAME=C >> >>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C >> >>>>> [11] LC_MEASUREMENT=pt_PT.UTF-8 LC_IDENTIFICATION=C >> >>>>> >> >>>>> attached base packages: >> >>>>> [1] stats graphics grDevices utils datasets methods base >> >>>>> >> >>>>> loaded via a namespace (and not attached): >> >>>>> [1] compiler_4.0.2 >> >>>>> >> >>>>> >> >>>>> >> >>>>> Às 08:45 de 15/07/20, Sam H escreveu: >> >>>>>> Hi, >> >>>>>> >> >>>>>> I am trying to download some data using read.csv and it works >> >>>>> perfectly in >> >>>>>> RStudio and fails in the R console in the terminal in Ubuntu 18.04 >> >>>>> after >> >>>>>> upgrading from R 3.6.3 to 4.0.2. Before upgrading this worked in >> the >> >>>>> R >> >>>>>> console in the terminal also without any issues. >> >>>>>> >> >>>>>> Why would that be? How to fix this? >> >>>>>> >> >>>>>> Below please find R code output and sessionInfo(). >> >>>>>> >> >>>>>> *Works in RStudio* >> >>>>>> >> >>>>>>> >> >>>>> read.csv(" >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> ", >> >>>>> header=TRUE, as.is=TRUE, na="n/a") >> >>>>>> Symbol Name >> >>>>>> LastSale MarketCap IPOyear1 TXG >> >>>>>> 10x Genomics, Inc. 87.4400 $8.6B 20192 YI >> >>>>>> 111, Inc. 6.4800 $533.69M 20183 >> >>>>>> PIH 1347 Property Insurance Holdings, Inc. 4.5350 >> >>>>>> $27.52M 2014 >> >>>>>> sessionInfo() >> >>>>>> R version 4.0.2 (2020-06-22) >> >>>>>> Platform: x86_64-pc-linux-gnu (64-bit) >> >>>>>> Running under: Ubuntu 18.04.4 LTS >> >>>>>> >> >>>>>> Matrix products: default >> >>>>>> BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3 >> >>>>>> LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so >> >>>>>> >> >>>>>> locale: >> >>>>>> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C >> >>>>>> LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 >> >>>>>> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 >> >>>>>> LC_PAPER=en_US.UTF-8 LC_NAME=C >> >>>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C >> >>>>>> LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C >> >>>>>> >> >>>>>> attached base packages:[1] stats graphics grDevices utils >> >>>>>> datasets methods base >> >>>>>> >> >>>>>> loaded via a namespace (and not attached):[1] compiler_4.0.2 >> >>>>> tools_4.0.2 >> >>>>>> >> >>>>>> *Fails in R console in terminal* >> >>>>>> >> >>>>>> > >> >>>>> read.csv(" >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> ", >> >>>>>> header=TRUE, as.is=TRUE, na="n/a") >> >>>>>> Error in file(file, "rt") : >> >>>>>> cannot open the connection to >> >>>>>> >> >>>>> ' >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> ' >> >>>>>> In addition: Warning message: >> >>>>>> In file(file, "rt") : >> >>>>>> URL >> >>>>> ' >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> ': >> >>>>>> status was 'Failure when receiving data from the peer'> >> traceback()3: >> >>>>>> file(file, "rt")2: read.table(file = file, header = header, sep = >> >>>>> sep, >> >>>>>> quote = quote, >> >>>>>> dec = dec, fill = fill, comment.char = comment.char, >> ...)1: >> >>>>>> >> >>>>> read.csv(" >> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download >> ", >> >>>>>> header = TRUE, as.is = TRUE, na = "n/a")> sessionInfo() >> >>>>>> R version 4.0.2 (2020-06-22) >> >>>>>> Platform: x86_64-pc-linux-gnu (64-bit) >> >>>>>> Running under: Ubuntu 18.04.4 LTS >> >>>>>> >> >>>>>> Matrix products: default >> >>>>>> BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3 >> >>>>>> LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so >> >>>>>> >> >>>>>> locale: >> >>>>>> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C >> >>>>>> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 >> >>>>>> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 >> >>>>>> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C >> >>>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C [11] >> >>>>>> LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C >> >>>>>> >> >>>>>> attached base packages:[1] stats graphics grDevices utils >> >>>>>> datasets methods base >> >>>>>> >> >>>>>> loaded via a namespace (and not attached):[1] compiler_4.0.2> >> >>>>>> >> >>>>>> I also asked this question here >> >>>>>> >> >>>>> >> https://stackoverflow.com/questions/62898008/why-read-csv-fails-in-r-console-in-ubuntu-terminal-but-works-in-rstudio-after-r >> >>>>>> . Since there was no answer on stackoverflow I sent this question >> >>>>> also to >> >>>>>> this list. >> >>>>>> >> >>>>>> Best regards, >> >>>>>> Sam >> >>>>>> >> >>>>>> [[alternative HTML version deleted]] >> >>>>>> >> >>>>>> ______________________________________________ >> >>>>>> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see >> >>>>>> https://stat.ethz.ch/mailman/listinfo/r-help >> >>>>>> PLEASE do read the posting guide >> >>>>> http://www.R-project.org/posting-guide.html >> >>>>>> and provide commented, minimal, self-contained, reproducible code. >> >>>>>> >> >>>>> >> >>>>> ______________________________________________ >> >>>>> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see >> >>>>> https://stat.ethz.ch/mailman/listinfo/r-help >> >>>>> PLEASE do read the posting guide >> >>>>> http://www.R-project.org/posting-guide.html >> >>>>> and provide commented, minimal, self-contained, reproducible code. >> >>>> >> >>> >> >>> ______________________________________________ >> >>> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see >> >>> https://stat.ethz.ch/mailman/listinfo/r-help >> >>> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> >>> and provide commented, minimal, self-contained, reproducible code. >> > >> > ______________________________________________ >> > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see >> > https://stat.ethz.ch/mailman/listinfo/r-help >> > PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> > and provide commented, minimal, self-contained, reproducible code. >> > >> >> -- >> Luke Tierney >> Ralph E. Wareham Professor of Mathematical Sciences >> University of Iowa Phone: 319-335-3386 >> Department of Statistics and Fax: 319-335-3017 >> Actuarial Science >> 241 Schaeffer Hall email: luke-tier...@uiowa.edu >> Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu >> > > > -- > Best regards, > Sam > [[alternative HTML version deleted]] ______________________________________________ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.