[Rd] Retrieving data from aspx pages
Hi. I'm trying to write an application to retrieve financial data (specially bonds data) from FINRA. The web page is served dynamically from an asp.net application: http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx I'd like to know if it's possible to fill dynamically the web page form from R and, after filling it (with the issuer name), retrieve the web page, parse the data, and covert it to appropiate R objects. For example, suppose I want to search data for ATT bonds. I'd like to know if it's possible, within R, to fill the page served from: http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx select the corporate option and fill with ATT the field for Issuer name, ask the page to display the results, and retrieve the results for each of the bonds issued by ATT (for example: http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/BondDetail.aspx?ID=MDAxOTU3Qko3) and parsing the data from the web page. Thanks in advance. __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] contr.sum() and contrast names
Le samedi 27 octobre 2012 à 10:44 -0400, John Fox a écrit : Hi Milan, Take a look at the contr.Sum() and contr.Treatment() functions in the car package. Yeah, this is the kind of function I had in mind. Just that I think we should have an equivalent in the base packages. (I recall, BTW, the sometimes acrimonious previous discussion of this issue.) I could not find this discussion, do you have any pointer? Anyways, going deeper and deeper into the archives, I've found you made the same proposal almost exactly 10 years ago[1]! Oddly enough, it only prompted one reply at the time - has the rest of the discussion been removed from the archives because it was too rude? ;-) Should we reopen the debate now that some time has been spent without actions being taken on that front? Regards 1: http://tolstoy.newcastle.edu.au/R/devel/02b/0878.html __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Retrieving data from aspx pages
Hi Jose, I think this is more of an r-help question so I'm going to forward your question there: R-devel is really for the development of R itself, not development with R. Cheers, Michael On Mon, Oct 29, 2012 at 8:11 PM, jose ramon mazaira jram...@gmail.com wrote: Hi. I'm trying to write an application to retrieve financial data (specially bonds data) from FINRA. The web page is served dynamically from an asp.net application: http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx I'd like to know if it's possible to fill dynamically the web page form from R and, after filling it (with the issuer name), retrieve the web page, parse the data, and covert it to appropiate R objects. For example, suppose I want to search data for ATT bonds. I'd like to know if it's possible, within R, to fill the page served from: http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx select the corporate option and fill with ATT the field for Issuer name, ask the page to display the results, and retrieve the results for each of the bonds issued by ATT (for example: http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/BondDetail.aspx?ID=MDAxOTU3Qko3) and parsing the data from the web page. Thanks in advance. __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Retrieving data from aspx pages
On Oct 30, 2012, at 10:44 , R. Michael Weylandt wrote: Hi Jose, I think this is more of an r-help question so I'm going to forward your question there: R-devel is really for the development of R itself, not development with R. Er, no! R help is (mainly) for helping beginners. We usually put the dividing line where the answers would become incomprehensible to many of the list readers. So compiler issues and internal C code is off-limits there, but I'd say that so is HTML and ASP. So unless there happens to be a precooked package that fits the query exactly, I'd say that it belongs here. To wit: This list is intended for questions and discussion about code development in R. Questions likely to prompt discussion unintelligible to non-programmers or topics that are too technical for R-help's audience should go to R-devel, see the posting guide section. Cheers, Michael On Mon, Oct 29, 2012 at 8:11 PM, jose ramon mazaira jram...@gmail.com wrote: Hi. I'm trying to write an application to retrieve financial data (specially bonds data) from FINRA. The web page is served dynamically from an asp.net application: http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx I'd like to know if it's possible to fill dynamically the web page form from R and, after filling it (with the issuer name), retrieve the web page, parse the data, and covert it to appropiate R objects. For example, suppose I want to search data for ATT bonds. I'd like to know if it's possible, within R, to fill the page served from: http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx select the corporate option and fill with ATT the field for Issuer name, ask the page to display the results, and retrieve the results for each of the bonds issued by ATT (for example: http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/BondDetail.aspx?ID=MDAxOTU3Qko3) and parsing the data from the web page. Thanks in advance. __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel -- Peter Dalgaard, Professor Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Email: pd@cbs.dk Priv: pda...@gmail.com __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] There is pmin and pmax each taking na.rm, how about psum?
Hi, Please consider the following : x = c(1,3,NA,5) y = c(2,NA,4,1) min(x,y,na.rm=TRUE)# ok [1] 1 max(x,y,na.rm=TRUE)# ok [1] 5 sum(x,y,na.rm=TRUE)# ok [1] 16 pmin(x,y,na.rm=TRUE) # ok [1] 1 3 4 1 pmax(x,y,na.rm=TRUE) # ok [1] 2 3 4 5 psum(x,y,na.rm=TRUE) [1] 3 3 4 6 # expected result Error: could not find function psum # actual result I realise that + is already like psum, but what about NA? x+y [1] 3 NA NA 6# can't supply `na.rm=TRUE` to `+` Is there a case to add psum? Or have I missed something. This question survived when I asked on Stack Overflow : http://stackoverflow.com/questions/13123638/there-is-pmin-and-pmax-each-taking-na-rm-why-no-psum And a search of the archives found that has Gabor has suggested it too as an aside : http://r.789695.n4.nabble.com/How-to-do-it-without-for-loops-tp794745p794750.html If someone from R core is willing to sponsor the idea, I am willing to write, test and submit the code for psum. Implemented in a very similar fashion to pmin and pmax. Or perhaps it exists already in a package somewhere (I searched but didn't find it). Matthew __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] contr.sum() and contrast names
Hi Milan, On Tue, 30 Oct 2012 10:25:56 +0100 Milan Bouchet-Valat nalimi...@club.fr wrote: Le samedi 27 octobre 2012 à 10:44 -0400, John Fox a écrit : Hi Milan, Take a look at the contr.Sum() and contr.Treatment() functions in the car package. Yeah, this is the kind of function I had in mind. Just that I think we should have an equivalent in the base packages. Given the history of the issue, I think that it's unlikely that this will happen. Your message hasn't generated a cascade of discussion. And, of course, even if you had provoked a discussion, R Core would have to agree that this kind of change is desirable. (I recall, BTW, the sometimes acrimonious previous discussion of this issue.) I could not find this discussion, do you have any pointer? Sorry. I don't recall when the discussion took place and failed to find it just now when I tried to locate it in the R email list archives. Anyways, going deeper and deeper into the archives, I've found you made the same proposal almost exactly 10 years ago[1]! Oddly enough, it only prompted one reply at the time - has the rest of the discussion been removed from the archives because it was too rude? ;-) I think that you know that the answer to that is no. And to be clear, the nasty comments that I recall weren't directed at me, and the rudeness was bidirectional (at least insofar as I remember the exchange accurately, which I may not). Should we reopen the debate now that some time has been spent without actions being taken on that front? You *have* reopened the issue but haven't elicited a response. A nice characteristic of R is that if you don't like how something is implemented, you can offer your own version, which is what I did with contr.Treatment(), contr.Sum(), and contr.Helmert() in the car package, and similarly with Anova() in the car package. Sometimes these kinds of changes get into the standard R distribution, but more commonly, people who agree with you are free to use the alternatives that you provide. Best, John Regards 1: http://tolstoy.newcastle.edu.au/R/devel/02b/0878.html __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] There is pmin and pmax each taking na.rm, how about psum?
Why don't you make a matrix and use colSums or rowSums? x = c(1,3,NA,5) y = c(2,NA,4,1) colSums(rbind(x, y), na.rm = TRUE) ir. Thierry Onkelinx Instituut voor natuur- en bosonderzoek / Research Institute for Nature and Forest team Biometrie Kwaliteitszorg / team Biometrics Quality Assurance Kliniekstraat 25 1070 Anderlecht Belgium + 32 2 525 02 51 + 32 54 43 61 85 thierry.onkel...@inbo.be www.inbo.be To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey -Oorspronkelijk bericht- Van: r-devel-boun...@r-project.org [mailto:r-devel-boun...@r-project.org] Namens Matthew Dowle Verzonden: dinsdag 30 oktober 2012 12:03 Aan: r-devel@r-project.org Onderwerp: [Rd] There is pmin and pmax each taking na.rm, how about psum? Hi, Please consider the following : x = c(1,3,NA,5) y = c(2,NA,4,1) min(x,y,na.rm=TRUE)# ok [1] 1 max(x,y,na.rm=TRUE)# ok [1] 5 sum(x,y,na.rm=TRUE)# ok [1] 16 pmin(x,y,na.rm=TRUE) # ok [1] 1 3 4 1 pmax(x,y,na.rm=TRUE) # ok [1] 2 3 4 5 psum(x,y,na.rm=TRUE) [1] 3 3 4 6 # expected result Error: could not find function psum # actual result I realise that + is already like psum, but what about NA? x+y [1] 3 NA NA 6# can't supply `na.rm=TRUE` to `+` Is there a case to add psum? Or have I missed something. This question survived when I asked on Stack Overflow : http://stackoverflow.com/questions/13123638/there-is-pmin-and-pmax-each-taking-na-rm-why-no-psum And a search of the archives found that has Gabor has suggested it too as an aside : http://r.789695.n4.nabble.com/How-to-do-it-without-for-loops-tp794745p794750.html If someone from R core is willing to sponsor the idea, I am willing to write, test and submit the code for psum. Implemented in a very similar fashion to pmin and pmax. Or perhaps it exists already in a package somewhere (I searched but didn't find it). Matthew __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel * * * * * * * * * * * * * D I S C L A I M E R * * * * * * * * * * * * * Dit bericht en eventuele bijlagen geven enkel de visie van de schrijver weer en binden het INBO onder geen enkel beding, zolang dit bericht niet bevestigd is door een geldig ondertekend document. The views expressed in this message and any annex are purely those of the writer and may not be regarded as stating an official position of INBO, as long as the message is not confirmed by a duly signed document. __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] There is pmin and pmax each taking na.rm, how about psum?
Because that's inconsistent with pmin and pmax when two NAs are summed. x = c(1,3,NA,NA,5) y = c(2,NA,4,NA,1) colSums(rbind(x, y), na.rm = TRUE) [1] 3 3 4 0 6# actual [1] 3 3 4 NA 6 # desired and it would be less convenient/natural (and slower) than a psum which would call .Internal(psum(na.rm,...)) in the same way as pmin and pmax. Why don't you make a matrix and use colSums or rowSums? x = c(1,3,NA,5) y = c(2,NA,4,1) colSums(rbind(x, y), na.rm = TRUE) ir. Thierry Onkelinx Instituut voor natuur- en bosonderzoek / Research Institute for Nature and Forest team Biometrie Kwaliteitszorg / team Biometrics Quality Assurance Kliniekstraat 25 1070 Anderlecht Belgium + 32 2 525 02 51 + 32 54 43 61 85 thierry.onkel...@inbo.be www.inbo.be To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey -Oorspronkelijk bericht- Van: r-devel-boun...@r-project.org [mailto:r-devel-boun...@r-project.org] Namens Matthew Dowle Verzonden: dinsdag 30 oktober 2012 12:03 Aan: r-devel@r-project.org Onderwerp: [Rd] There is pmin and pmax each taking na.rm, how about psum? Hi, Please consider the following : x = c(1,3,NA,5) y = c(2,NA,4,1) min(x,y,na.rm=TRUE)# ok [1] 1 max(x,y,na.rm=TRUE)# ok [1] 5 sum(x,y,na.rm=TRUE)# ok [1] 16 pmin(x,y,na.rm=TRUE) # ok [1] 1 3 4 1 pmax(x,y,na.rm=TRUE) # ok [1] 2 3 4 5 psum(x,y,na.rm=TRUE) [1] 3 3 4 6 # expected result Error: could not find function psum # actual result I realise that + is already like psum, but what about NA? x+y [1] 3 NA NA 6# can't supply `na.rm=TRUE` to `+` Is there a case to add psum? Or have I missed something. This question survived when I asked on Stack Overflow : http://stackoverflow.com/questions/13123638/there-is-pmin-and-pmax-each-taking-na-rm-why-no-psum And a search of the archives found that has Gabor has suggested it too as an aside : http://r.789695.n4.nabble.com/How-to-do-it-without-for-loops-tp794745p794750.html If someone from R core is willing to sponsor the idea, I am willing to write, test and submit the code for psum. Implemented in a very similar fashion to pmin and pmax. Or perhaps it exists already in a package somewhere (I searched but didn't find it). Matthew __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel * * * * * * * * * * * * * D I S C L A I M E R * * * * * * * * * * * * * Dit bericht en eventuele bijlagen geven enkel de visie van de schrijver weer en binden het INBO onder geen enkel beding, zolang dit bericht niet bevestigd is door een geldig ondertekend document. The views expressed in this message and any annex are purely those of the writer and may not be regarded as stating an official position of INBO, as long as the message is not confirmed by a duly signed document. __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Retrieving data from aspx pages
I don't know of an easy way to do this in R. I've been doing something similar with python scripts called from R. If anyone knows how to do this with just R, I would appreciate hearing too. Paul On 12-10-29 04:11 PM, jose ramon mazaira wrote: Hi. I'm trying to write an application to retrieve financial data (specially bonds data) from FINRA. The web page is served dynamically from an asp.net application: http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx I'd like to know if it's possible to fill dynamically the web page form from R and, after filling it (with the issuer name), retrieve the web page, parse the data, and covert it to appropiate R objects. For example, suppose I want to search data for ATT bonds. I'd like to know if it's possible, within R, to fill the page served from: http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx select the corporate option and fill with ATT the field for Issuer name, ask the page to display the results, and retrieve the results for each of the bonds issued by ATT (for example: http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/BondDetail.aspx?ID=MDAxOTU3Qko3) and parsing the data from the web page. Thanks in advance. __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Retrieving data from aspx pages
I think RHTMLForms works if you have a single form, but I have not been able to see how to use it when you need to go through a sequence of dynamically generated forms (like you can do with Python mechanize). Paul On 12-10-30 09:08 AM, Gabriel Becker wrote: I haven't used it extensively myself, and can't speak to it's current state but on quick inspection RHTMLForms seems worth a look for what you want. http://www.omegahat.org/RHTMLForms/ ~G On Tue, Oct 30, 2012 t 5:38 AM, Paul Gilbert pgilbert...@gmail.com mailto:pgilbert...@gmail.com wrote: I don't know of an easy way to do this in R. I've been doing something similar with python scripts called from R. If anyone knows how to do this with just R, I would appreciate hearing too. Paul On 12-10-29 04:11 PM, jose ramon mazaira wrote: Hi. I'm trying to write an application to retrieve financial data (specially bonds data) from FINRA. The web page is served dynamically from an asp.net http://asp.net application: http://cxa.gtm.__idmanagedsolutions.com/finra/__BondCenter/AdvancedScreener.__aspx http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx I'd like to know if it's possible to fill dynamically the web page form from R and, after filling it (with the issuer name), retrieve the web page, parse the data, and covert it to appropiate R objects. For example, suppose I want to search data for ATT bonds. I'd like to know if it's possible, within R, to fill the page served from: http://cxa.gtm.__idmanagedsolutions.com/finra/__BondCenter/AdvancedScreener.__aspx http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx select the corporate option and fill with ATT the field for Issuer name, ask the page to display the results, and retrieve the results for each of the bonds issued by ATT (for example: http://cxa.gtm.__idmanagedsolutions.com/finra/__BondCenter/BondDetail.aspx?ID=__MDAxOTU3Qko3 http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/BondDetail.aspx?ID=MDAxOTU3Qko3) and parsing the data from the web page. Thanks in advance. R-devel@r-project.org mailto:R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/__listinfo/r-devel https://stat.ethz.ch/mailman/listinfo/r-devel R-devel@r-project.org mailto:R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/__listinfo/r-devel https://stat.ethz.ch/mailman/listinfo/r-devel -- Gabriel Becker Graduate Student Statistics Department University of California, Davis __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] There is pmin and pmax each taking na.rm, how about psum?
Is there a case to add psum? Or have I missed something. If psum, then why not pdiff (-), pprod (*) and precip (/) ? And similarly, what about equivalent functions for ^, %%, %/%, , and | ? Hadley -- RStudio / Rice University http://had.co.nz/ __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] install.packages() fails if .libPaths() set
On 29.10.2012 23:35, Hadley Wickham wrote: Reproducible example: dir.create(test) .libPaths(test) install.packages(relations, type = source) Fails with: ... * installing *source* package ‘relations’ ... ... ** testing if installed package can be loaded *** arch - i386 Error : package ‘sets’ required by ‘relations’ could not be found Error: loading failed Execution halted *** arch - x86_64 Error : package ‘sets’ required by ‘relations’ could not be found Error: loading failed Execution halted ERROR: loading failed for ‘i386’, ‘x86_64’ If I don't set .libPaths() it installs without error. Yes, since your package set is installed in a library that is not test. And you have omitted that library from the search path by changing .libPaths(). Uwe Ligges This is on a fairly recent version of R-devel (Oct 3) and on R 2.15.1 Hadley __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] install.packages() fails if .libPaths() set
* installing *source* package ‘relations’ ... ... ** testing if installed package can be loaded *** arch - i386 Error : package ‘sets’ required by ‘relations’ could not be found Error: loading failed Execution halted *** arch - x86_64 Error : package ‘sets’ required by ‘relations’ could not be found Error: loading failed Execution halted ERROR: loading failed for ‘i386’, ‘x86_64’ If I don't set .libPaths() it installs without error. Yes, since your package set is installed in a library that is not test. And you have omitted that library from the search path by changing .libPaths(). I'm not sure I follow. I thought install.packages() installs into the first element of .libPaths(), and indeed I get this message: Installing package into ‘/Users/hadley/Desktop/test’ (as ‘lib’ is unspecified) Hadley -- RStudio / Rice University http://had.co.nz/ __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] install.packages() fails if .libPaths() set
On 30.10.2012 15:01, Hadley Wickham wrote: * installing *source* package ‘relations’ ... ... ** testing if installed package can be loaded *** arch - i386 Error : package ‘sets’ required by ‘relations’ could not be found Error: loading failed Execution halted *** arch - x86_64 Error : package ‘sets’ required by ‘relations’ could not be found Error: loading failed Execution halted ERROR: loading failed for ‘i386’, ‘x86_64’ If I don't set .libPaths() it installs without error. Yes, since your package set is installed in a library that is not test. And you have omitted that library from the search path by changing .libPaths(). I'm not sure I follow. I thought install.packages() installs into the first element of .libPaths(), and indeed I get this message: Installing package into ‘/Users/hadley/Desktop/test’ (as ‘lib’ is unspecified) Yes, but sets is not there. Uwe Hadley __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Retrieving data from aspx pages
Jose As far as getting to the data, I think the best way to do this sort of thing would be if the site supports a SOAP or REST interface. When they don't (yet) then one is faced with clicking through some pages. Python or Java is one way to automate the process of clicking through the pages. I don't know how to do that in R, but would like to know if it is possible. But, I guess I was confused about the part you want to improve. What I have works fairly smoothly parsing and passing back JSON data, converted from a csv file, into R. The downside is that this approach requires more than R to be installed on the client machine. But if the object you get back is ASPX, then you either need to parse it directly, or convert it to JSON, or something else you can deal with. I suspect that will be fairly specific to a particular web site, but I don't really know enough about ASPX to be sure. Paul On 12-10-30 01:12 PM, jose ramon mazaira wrote: Thanks for your interest, Paul. I've checked the source code of TSjson and I've seen that what it does is to call a Python script to retrieve the data. In fact, I've already done this with Java using the URLConnection class and sending the requested values to fill the form. However, I think it would be more useful to open a connection with R and to send the requested values within R, and not through an external program. The application I've designed, like yours, is also page-specific (i.e., designed for http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx), but I think that our applications would be more powerful if they were able to parse the name-value pairs generated from ASPX (or of any other dynamically generated web page) and ask the user to select the appropiate values. 2012/10/30, Paul Gilbert pgilbert...@gmail.com: I think RHTMLForms works if you have a single form, but I have not been able to see how to use it when you need to go through a sequence of dynamically generated forms (like you can do with Python mechanize). Paul On 12-10-30 09:08 AM, Gabriel Becker wrote: I haven't used it extensively myself, and can't speak to it's current state but on quick inspection RHTMLForms seems worth a look for what you want. http://www.omegahat.org/RHTMLForms/ ~G On Tue, Oct 30, 2012 t 5:38 AM, Paul Gilbert pgilbert...@gmail.com mailto:pgilbert...@gmail.com wrote: I don't know of an easy way to do this in R. I've been doing something similar with python scripts called from R. If anyone knows how to do this with just R, I would appreciate hearing too. Paul On 12-10-29 04:11 PM, jose ramon mazaira wrote: Hi. I'm trying to write an application to retrieve financial data (specially bonds data) from FINRA. The web page is served dynamically from an asp.net http://asp.net application: http://cxa.gtm.__idmanagedsolutions.com/finra/__BondCenter/AdvancedScreener.__aspx http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx I'd like to know if it's possible to fill dynamically the web page form from R and, after filling it (with the issuer name), retrieve the web page, parse the data, and covert it to appropiate R objects. For example, suppose I want to search data for ATT bonds. I'd like to know if it's possible, within R, to fill the page served from: http://cxa.gtm.__idmanagedsolutions.com/finra/__BondCenter/AdvancedScreener.__aspx http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx select the corporate option and fill with ATT the field for Issuer name, ask the page to display the results, and retrieve the results for each of the bonds issued by ATT (for example: http://cxa.gtm.__idmanagedsolutions.com/finra/__BondCenter/BondDetail.aspx?ID=__MDAxOTU3Qko3 http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/BondDetail.aspx?ID=MDAxOTU3Qko3) and parsing the data from the web page. Thanks in advance. R-devel@r-project.org mailto:R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/__listinfo/r-devel https://stat.ethz.ch/mailman/listinfo/r-devel R-devel@r-project.org mailto:R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/__listinfo/r-devel https://stat.ethz.ch/mailman/listinfo/r-devel -- Gabriel Becker Graduate Student Statistics Department University of California, Davis __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Class generator functions for reference classes
A heads up for those using Rcpp and perhaps other packages related to reference classes. The changes in 61035 cause a problem for classes defined with a slot using the return value of setRefClass(), notably for Rcpp. The name of that class changed. Rather than introduce back-incompatibility, I will rename a couple of classes so the class name of the return value remains the same. (It's not as natural a naming scheme as what is in place, but not the first time back-compatibility has won out over preferred design.) It will be a day or two before I can get to this, so don't panic if your package does not install from the current r-devel until then. John On 10/27/12 2:07 PM, John Chambers wrote: As of rev. 61035 in r-devel, setRefClass() now returns a generator function, as setClass() has done since 2.15.0. The convenient style is now: mEdit - setRefClass(mEdit,..) xx - mEdit(data = xMat) instead of xx - mEdit$new(data = xMat) The returned object still has fields and methods accessible as before. See the Value and Reference Class Generators sections of ?ReferenceClasses for details. Thanks to Romain François for suggesting this. John __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Class generator functions for reference classes
As of rev. 61035 in r-devel, setRefClass() now returns a generator function, as setClass() has done since 2.15.0. The convenient style is now: mEdit - setRefClass(mEdit,..) xx - mEdit(data = xMat) instead of xx - mEdit$new(data = xMat) The returned object still has fields and methods accessible as before. Similarly, would it be possible for setGeneric and setMethod to return the equivalent generic and method objects? Hadley -- RStudio / Rice University http://had.co.nz/ __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] install.packages() fails if .libPaths() set
It appears that this problem only happens when R_LIBS is specified in ~/.Renviron. When it compiles 'relations', it's somehow not passing along the correct lib paths. This is the test code: dir.create(test) .libPaths(test) install.packages(relations, type = source) The 'relations' package depends on 'sets' and 'slam'. When there is no ~/.Renviron file it works: all three packages get installed into test/. The story is different when there is an file ~/.Renviron with the contents: R_LIBS=~/R (And the ~/R directory exists, of course.) It manages to install 'sets' and 'slam' into test/, but it fails to install 'relations'. I've added the output to the bottom of this email. I test this with R 2.15.1 and R 2.15.2 on Windows XP, and R 2.15.1 on Mac OS X 10.7.5. In all cases, I get the same result. Here's the output when there _is_ an .Renviron file: install.packages(relations, type = source) Installing package(s) into ‘C:/Documents and Settings/user/My Documents/test’ (as ‘lib’ is unspecified) also installing the dependencies ‘sets’, ‘slam’ trying URL 'http://cran.rstudio.com/src/contrib/sets_1.0-11.tar.gz' Content type 'application/x-gzip' length 283388 bytes (276 Kb) opened URL downloaded 276 Kb trying URL 'http://cran.rstudio.com/src/contrib/slam_0.1-26.tar.gz' Content type 'application/x-gzip' length 39506 bytes (38 Kb) opened URL downloaded 38 Kb trying URL 'http://cran.rstudio.com/src/contrib/relations_0.6.tar.gz' Content type 'application/x-gzip' length 493292 bytes (481 Kb) opened URL downloaded 481 Kb * installing *source* package 'sets' ... ** package 'sets' successfully unpacked and MD5 sums checked ** libs cygwin warning: MS-DOS style path detected: C:/R/R-215~1.1/etc/i386/Makeconf Preferred POSIX equivalent is: /cygdrive/c/R/R-215~1.1/etc/i386/Makeconf CYGWIN environment variable option nodosfilewarning turns off this warning. Consult the user's guide for more details about POSIX paths: http://cygwin.com/cygwin-ug-net/using.html#using-pathnames gcc -IC:/R/R-215~1.1/include -DNDEBUG -O3 -Wall -std=gnu99 -mtune=core2 -c closure.c -o closure.o gcc -shared -s -static-libgcc -o sets.dll tmp.def closure.o -LC:/R/R-215~1.1/bin/i386 -lR installing to C:/Documents and Settings/user/My Documents/test/sets/libs/i386 ** R ** data ** inst ** preparing package for lazy loading ** help *** installing help indices ** building package indices ** installing vignettes 'sets.Rnw' ** testing if installed package can be loaded * DONE (sets) * installing *source* package 'slam' ... ** package 'slam' successfully unpacked and MD5 sums checked ** libs cygwin warning: MS-DOS style path detected: C:/R/R-215~1.1/etc/i386/Makeconf Preferred POSIX equivalent is: /cygdrive/c/R/R-215~1.1/etc/i386/Makeconf CYGWIN environment variable option nodosfilewarning turns off this warning. Consult the user's guide for more details about POSIX paths: http://cygwin.com/cygwin-ug-net/using.html#using-pathnames gcc -IC:/R/R-215~1.1/include -DNDEBUG -O3 -Wall -std=gnu99 -mtune=core2 -c grouped.c -o grouped.o gcc -IC:/R/R-215~1.1/include -DNDEBUG -O3 -Wall -std=gnu99 -mtune=core2 -c sparse.c -o sparse.o gcc -IC:/R/R-215~1.1/include -DNDEBUG -O3 -Wall -std=gnu99 -mtune=core2 -c util.c -o util.o gcc -shared -s -static-libgcc -o slam.dll tmp.def grouped.o sparse.o util.o -LC:/R/R-215~1.1/bin/i386 -lRblas -lgfortran -LC:/R/R-215~1.1/bin/i386 -lR installing to C:/Documents and Settings/user/My Documents/test/slam/libs/i386 ** R ** preparing package for lazy loading ** help *** installing help indices ** building package indices ** testing if installed package can be loaded * DONE (slam) * installing *source* package 'relations' ... ** package 'relations' successfully unpacked and MD5 sums checked ** R ** data ** moving datasets to lazyload DB ** inst ** preparing package for lazy loading ** help *** installing help indices ** building package indices ** installing vignettes 'relations.Rnw' ** testing if installed package can be loaded Error : package 'sets' required by 'relations' could not be found Error: loading failed Execution halted ERROR: loading failed * removing 'C:/Documents and Settings/user/My Documents/test/relations' Warning in install.packages : running command 'C:/R/R-215~1.1/bin/i386/R CMD INSTALL -l C:/Documents and Settings/user/My Documents/test C:\DOCUME~1\user\LOCALS~1\Temp\RtmpGSxf4J/downloaded_packages/relations_0.6.tar.gz' had status 1 Warning in install.packages : installation of package ‘relations’ had non-zero exit status The downloaded source packages are in ‘C:\Documents and Settings\user\Local Settings\Temp\RtmpGSxf4J\downloaded_packages’ __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
