Re: [Rd] using Paraview "in-situ" with R?
I'm posting here instead of R-package since your response was the most relevant :) I started using the "in-situ" workflow and its very nice. Thew viewer stays up between runs and across data sources and its the same UI regardless of source :) fwiw. I started digging through some noisy data, dog diet data in which the supplements were pulsed ( in essence PWM lol). This made it hard to find the right average doses to correlate with outcomes. I had to play with trailing average filtering but ultimately some trends emerged that made sense with subjective observations ( maybe they are not real but that is not the point of this post ). I did not use R or Visit but I used my own code for filtering and MJMDatascope for viewing and expect to do more sophisticated stuff in R soon. I can tell you though that eliminating the annoyance and confusion of temp files was very helpful. The workflow, and mind flow, for the "in situ" mode makes you wonder how you ever did without it. I will probably need to update my R interface package and MJMDatascope is hardly ideal but improving both seems worth the effort. Thanks. Mike Marchywka 44 Crosscreek Trail Jasper GA 30143 was 306 Charles Cox Drive Canton, GA 30115 470-758-0799 404-788-1216 From: George Ostrouchov Sent: Wednesday, January 10, 2024 3:06 PM To: r-devel@r-project.org Cc: Mike Marchywka Subject: Re: [Rd] using Paraview "in-situ" with R? At ORNL, we worked with VisIt (a sibling of Paraview, both funded largely by DOE) around 2016 and made an in situ demo with R. We used packages pbdMPI (on CRAN) and pbdDMAT (on GitHub/RbigData), which were in part built for this purpose. Later also the package hola (on GitHub/RbigData) was built to connect with adios2, which can do buffered in situ connections with various codes. But the VisIt developers were not interested in R (preferring to roll their own), so that direction fizzled. Paraview is a competetive sibling of VisIt, so I don’t know if they would be interested. The packages we developed are viable for that purpose. There is a lot in R that could benefit Paraview (or VisIt). George > > Message: 1 > Date: Tue, 9 Jan 2024 14:20:17 + > From: Mike Marchywka > To: R-devel > Subject: [Rd] using Paraview "in-situ" with R? > Message-ID: > > > > Content-Type: text/plain; charset="iso-8859-1" > > I had previously asked about R interfaces to various "other" visualization > tools specifically lightweights for monitoring progress of > various codes. I was working on this, > > https://github.com/mmarchywka/mjmdatascope > > but in the meantime found out that Paraview has an "in-situ" > capability for similar objectives. > > https://discourse.paraview.org/t/does-or-can-paraview-support-streaming-input/13637/9 > > While R does have a lot of plotting features, > it seems like an excellent tool to interface to R allowing visualization > without > a bunch of temp files or > > Is anyone aware of anyone doing this interface or reasons its a boondoggle? > > Thanks. > > > > Mike Marchywka > 44 Crosscreek Trail > Jasper GA 30143 > was 306 Charles Cox Drive Canton, GA 30115 > 470-758-0799 > 404-788-1216 > __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] using Paraview "in-situ" with R?
Thanks. I take it though you see "R" in this role as adding to the capabilities of the viewers, maybe adding some quick model fits over FEM results or something? Right now I was imagining working with freefem and rolling my own c++ code with supporting use of R code. Ideally I could easily overlay stuff without messing around with temp files. There are a lot of R things, probably optimizations etc, that may be nice to view as they progress with more than just a figure of merit. Right now I'm just trying to use Runge-Kutta on a simple orbit and the mjmdatascope output is much more useful on-the-fly than text or after the fact. Mike Marchywka 44 Crosscreek Trail Jasper GA 30143 was 306 Charles Cox Drive Canton, GA 30115 470-758-0799 404-788-1216 From: George Ostrouchov Sent: Wednesday, January 10, 2024 3:06 PM To: r-devel@r-project.org Cc: Mike Marchywka Subject: Re: [Rd] using Paraview "in-situ" with R? At ORNL, we worked with VisIt (a sibling of Paraview, both funded largely by DOE) around 2016 and made an in situ demo with R. We used packages pbdMPI (on CRAN) and pbdDMAT (on GitHub/RbigData), which were in part built for this purpose. Later also the package hola (on GitHub/RbigData) was built to connect with adios2, which can do buffered in situ connections with various codes. But the VisIt developers were not interested in R (preferring to roll their own), so that direction fizzled. Paraview is a competetive sibling of VisIt, so I don’t know if they would be interested. The packages we developed are viable for that purpose. There is a lot in R that could benefit Paraview (or VisIt). George > > Message: 1 > Date: Tue, 9 Jan 2024 14:20:17 + > From: Mike Marchywka > To: R-devel > Subject: [Rd] using Paraview "in-situ" with R? > Message-ID: > > > > Content-Type: text/plain; charset="iso-8859-1" > > I had previously asked about R interfaces to various "other" visualization > tools specifically lightweights for monitoring progress of > various codes. I was working on this, > > https://github.com/mmarchywka/mjmdatascope > > but in the meantime found out that Paraview has an "in-situ" > capability for similar objectives. > > https://discourse.paraview.org/t/does-or-can-paraview-support-streaming-input/13637/9 > > While R does have a lot of plotting features, > it seems like an excellent tool to interface to R allowing visualization > without > a bunch of temp files or > > Is anyone aware of anyone doing this interface or reasons its a boondoggle? > > Thanks. > > > > Mike Marchywka > 44 Crosscreek Trail > Jasper GA 30143 > was 306 Charles Cox Drive Canton, GA 30115 > 470-758-0799 > 404-788-1216 > __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[R-pkg-devel] Fw: [Rd] using Paraview "in-situ" with R?
The r-devel list suggested I try this here. Thanks. From: R-devel on behalf of Mike Marchywka Sent: Tuesday, January 9, 2024 9:20 AM To: R-devel Subject: [Rd] using Paraview "in-situ" with R? I had previously asked about R interfaces to various "other" visualization tools specifically lightweights for monitoring progress of various codes. I was working on this, https://github.com/mmarchywka/mjmdatascope but in the meantime found out that Paraview has an "in-situ" capability for similar objectives. https://discourse.paraview.org/t/does-or-can-paraview-support-streaming-input/13637/9 While R does have a lot of plotting features, it seems like an excellent tool to interface to R allowing visualization without a bunch of temp files or Is anyone aware of anyone doing this interface or reasons its a boondoggle? Thanks. Mike Marchywka 44 Crosscreek Trail Jasper GA 30143 was 306 Charles Cox Drive Canton, GA 30115 470-758-0799 404-788-1216 __ r-de...@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
[Rd] using Paraview "in-situ" with R?
I had previously asked about R interfaces to various "other" visualization tools specifically lightweights for monitoring progress of various codes. I was working on this, https://github.com/mmarchywka/mjmdatascope but in the meantime found out that Paraview has an "in-situ" capability for similar objectives. https://discourse.paraview.org/t/does-or-can-paraview-support-streaming-input/13637/9 While R does have a lot of plotting features, it seems like an excellent tool to interface to R allowing visualization without a bunch of temp files or Is anyone aware of anyone doing this interface or reasons its a boondoggle? Thanks. Mike Marchywka 44 Crosscreek Trail Jasper GA 30143 was 306 Charles Cox Drive Canton, GA 30115 470-758-0799 404-788-1216 __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] R packages to send plottable data to external apps
I put this up on github in current form where it sort of works and I can use it for my needs but if anyone thinks it fills a niche I guess I could clean it up. https://github.com/mmarchywka/mjmdatascope Definitely not ready for users but maybe a deverloper. In terms of the "Trend" package you mentioned maybe one interface would be to emulate that action too as that was my original interest circa 2007 lol. fwiw. Thanks. Mike Marchywka 44 Crosscreek Trail Jasper GA 30143 was 306 Charles Cox Drive Canton, GA 30115 470-758-0799 404-788-1216 From: Iñaki Ucar Sent: Sunday, August 27, 2023 7:12 PM To: Mike Marchywka Cc: r-devel; R Package Development Subject: Re: [Rd] R packages to send plottable data to external apps I think r-package-devel is a better place for this. CC'ing there. On Sun, 27 Aug 2023 at 23:50, Mike Marchywka wrote: > > I was curious what R packages, or indeed any other applications, exist > to plot streamed data from arbitrary data generators. It need not > be publication quality plotting but it should be easy to use like > an oscilloscope. The last time I checked, there wasn't any R package suitable for plotting high-throughput streaming data. There's a nice command-line utility called trend [1] that I extensively used in the past as an oscilloscope to visualize the output from a DAQ card. I don't see any new development there, but it does exactly what it promises; it's easy to use, quite configurable and very fast. Old but gold. I also explored VisPy, which is much more ambitious, but at that time the API had a limitation that didn't allow me to achieve what I required, and I haven't looked at it ever since, but the project seems in good shape. [1] https://www.thregr.org/wavexx/software/trend/ [2] https://vispy.org/ Hope it helps, Iñaki > I was working on something called datascope that I > am using for 1D finite difference monitoring and recently interfaced it > to freefem. I also created an R package. If there is any interest in something > like this I guess I could put it up somewhere when it is more usable > or if you can suggest some similar popular packages that would be good > too. Is there something I could drop-in to the attached code and get > something like the attached output that could also be switched to other > data sources? This right now works via linux fifo and somewhat by UDP. > It can queue data and stop making it if no one seems to be consuming > it depending on the channel. > > Thanks. > > Mike Marchywka > 44 Crosscreek Trail > Jasper GA 30143 > was 306 Charles Cox Drive Canton, GA 30115 > 470-758-0799 > 404-788-1216 > > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel -- Iñaki Úcar __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] R packages to send plottable data to external apps
I was curious what R packages, or indeed any other applications, exist to plot streamed data from arbitrary data generators. It need not be publication quality plotting but it should be easy to use like an oscilloscope. I was working on something called datascope that I am using for 1D finite difference monitoring and recently interfaced it to freefem. I also created an R package. If there is any interest in something like this I guess I could put it up somewhere when it is more usable or if you can suggest some similar popular packages that would be good too. Is there something I could drop-in to the attached code and get something like the attached output that could also be switched to other data sources? This right now works via linux fifo and somewhat by UDP. It can queue data and stop making it if no one seems to be consuming it depending on the channel. Thanks. Mike Marchywka 44 Crosscreek Trail Jasper GA 30143 was 306 Charles Cox Drive Canton, GA 30115 470-758-0799 404-788-1216 __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] R-3.0.1 - transient make check failure in splines-EX.r
I think I just sent a reply to you but you can reply to list if you like. This new menu just has reply and you have hunt for reply all LOL. From: avraham.ad...@guycarp.com To: marchy...@hotmail.com; r-devel@r-project.org Date: Fri, 31 May 2013 10:16:11 -0500 Subject: RE: [Rd] R-3.0.1 - transient make check failure in splines-EX.r Thank you, Mike, I did not know that! I tried to prevent multi-threaded issues by setting the compiler options to be single-threaded, but I know so little about this area that there may be something else going on. Do you think that the same problem may be causing the 64-bit issue I am having (https://stat.ethz.ch/pipermail/r-devel/2013-May/066731.html)? I tend to think not, as I haven't seen changing results in the call to `optim`, and I still don't know what NEW_X means. Once again, thank you. Avraham Adler -Original Message- From: Mike Marchywka [mailto:marchy...@hotmail.com] Sent: Thursday, May 30, 2013 7:21 PM To: Adler, Avraham; 'r-devel@r-project.org' Subject: RE: [Rd] R-3.0.1 - transient make check failure in splines-EX.r From: avraham.ad...@guycarp.com To: r-devel@r-project.org Date: Thu, 30 May 2013 16:17:36 -0500 Subject: Re: [Rd] R-3.0.1 - transient make check failure in splines-EX.r I just found this thread on StackOverflow http://stackoverflow.com/questions/13871818/ns-varies-for-no-apparent-reason/13878936 which had the same problem with the `ns` call changing with Revolution, and the answer given by tech support was that the MKL BLAS sometime returns ever-so-slightly different floating point results than a reference BLAS. The problem with that answer is that if it is true, the runs should not change *on the same machine* but it is another example of this issue. Unfortunately, it seems to dead-end too. Read some of the documents on the Intel site about floating point consistency and compiler optimizations. There are some reasons that you could get a different result from repeated runs on the same machine. One of these would be bugs like unititialized memory but another would be things like state of FPU and issues with multi-threaded code having some order dependencies etc. ( hotmail can not believe I am trying to post text but maybe you can figure it out from whatver this link eds up looking like ) href=http://www.google.com/search?biw=1253bih=542hl=enq=floating+point+low+bits+vary+fpu+prior+state+site%253Aintel.comoq=floating+point+low+bits+vary+fpu+prior+state+site%253Aintel.com; Avraham -Original Message- From: Adler, Avraham Sent: Thursday, May 30, 2013 3:12 PM To: Paul Gilbert Cc: r-devel@r-project.org Subject: RE: R-3.0.1 - transient make check failure in splines-EX.r Thank you very much, Paul. Serendipitously, I seem to have stumbled on a solution. In my parallel (still unsuccessful) attempt to build a BLAS for a 64bit machine (see https://stat.ethz.ch/pipermail/r-devel/2013-May/066731.html) I remembered from ATLAS that under the newer Windows there is a divergence from the standard ABI (see http://math-atlas.sourceforge.net/atlas_install/node57.html). Looking through the various makefiles under OpenBLAS, I found the following: ifeq ($(C_COMPILER), GCC) #Test for supporting MS_ABI GCCVERSIONGTEQ4 := $(shell expr `$(CC) -dumpversion | cut -f1 -d.` \= 4) GCCVERSIONGT4 := $(shell expr `$(CC) -dumpversion | cut -f1 -d.` \ 4) GCCMINORVERSIONGTEQ7 := $(shell expr `$(CC) -dumpversion | cut -f2 -d.` \= 7) ifeq ($(GCCVERSIONGT4), 1) # GCC Majar version 4 # It is compatible with MSVC ABI. CCOMMON_OPT += -DMS_ABI endif I had been building OPBL using gcc4.8.0, which is ostensibly compatible with the newer ABI, but Rtools still lives in 4.6.3, which isn't. Recompiling the BLAS with MinGW32 for 4.6.3 created a file that has passed `make check-all` twice now. I plan on comparing the speed with the ATLAS-based blas, and if it is faster, I hope to e-mail the dll and check results to Dr. Ligges. I say stumbled serendipitously because when using the 64 bit version of MinGw 4.6.3 resulted in the same `optim`-based error in `factanal` which I describe in the thread linked-to above. I will try using different versions of MinGW or even trying under Cygwin, I guess. In any event, Paul, I am curious if when you were trying to compile and had the same issue, were you using a different version or generation of gcc in the BLAS compilation than in the R compilation? Once again, thank you very much. Avraham Adler -Original Message- From: Paul Gilbert Sent: Thursday, May 30, 2013 12:26 AM To: Adler, Avraham Subject: Re: R-3.0.1 - transient make check failure in splines-EX.r Avraham I resolved this only by switching to a different BLAS on the 32 bit machine.Since no one else seemed to be having problems, I considered
Re: [Rd] R-3.0.1 - transient make check failure in splines-EX.r
From: avraham.ad...@guycarp.com To: r-devel@r-project.org Date: Thu, 30 May 2013 16:17:36 -0500 Subject: Re: [Rd] R-3.0.1 - transient make check failure in splines-EX.r I just found this thread on StackOverflow http://stackoverflow.com/questions/13871818/ns-varies-for-no-apparent-reason/13878936 which had the same problem with the `ns` call changing with Revolution, and the answer given by tech support was that the MKL BLAS sometime returns ever-so-slightly different floating point results than a reference BLAS. The problem with that answer is that if it is true, the runs should not change *on the same machine* but it is another example of this issue. Unfortunately, it seems to dead-end too. Read some of the documents on the Intel site about floating point consistency and compiler optimizations. There are some reasons that you could get a different result from repeated runs on the same machine. One of these would be bugs like unititialized memory but another would be things like state of FPU and issues with multi-threaded code having some order dependencies etc. ( hotmail can not believe I am trying to post text but maybe you can figure it out from whatver this link eds up looking like ) href=http://www.google.com/search?biw=1253bih=542hl=enq=floating+point+low+bits+vary+fpu+prior+state+site%253Aintel.comoq=floating+point+low+bits+vary+fpu+prior+state+site%253Aintel.com; Avraham -Original Message- From: Adler, Avraham Sent: Thursday, May 30, 2013 3:12 PM To: Paul Gilbert Cc: r-devel@r-project.org Subject: RE: R-3.0.1 - transient make check failure in splines-EX.r Thank you very much, Paul. Serendipitously, I seem to have stumbled on a solution. In my parallel (still unsuccessful) attempt to build a BLAS for a 64bit machine (see https://stat.ethz.ch/pipermail/r-devel/2013-May/066731.html) I remembered from ATLAS that under the newer Windows there is a divergence from the standard ABI (see http://math-atlas.sourceforge.net/atlas_install/node57.html). Looking through the various makefiles under OpenBLAS, I found the following: ifeq ($(C_COMPILER), GCC) #Test for supporting MS_ABI GCCVERSIONGTEQ4 := $(shell expr `$(CC) -dumpversion | cut -f1 -d.` \= 4) GCCVERSIONGT4 := $(shell expr `$(CC) -dumpversion | cut -f1 -d.` \ 4) GCCMINORVERSIONGTEQ7 := $(shell expr `$(CC) -dumpversion | cut -f2 -d.` \= 7) ifeq ($(GCCVERSIONGT4), 1) # GCC Majar version 4 # It is compatible with MSVC ABI. CCOMMON_OPT += -DMS_ABI endif I had been building OPBL using gcc4.8.0, which is ostensibly compatible with the newer ABI, but Rtools still lives in 4.6.3, which isn't. Recompiling the BLAS with MinGW32 for 4.6.3 created a file that has passed `make check-all` twice now. I plan on comparing the speed with the ATLAS-based blas, and if it is faster, I hope to e-mail the dll and check results to Dr. Ligges. I say stumbled serendipitously because when using the 64 bit version of MinGw 4.6.3 resulted in the same `optim`-based error in `factanal` which I describe in the thread linked-to above. I will try using different versions of MinGW or even trying under Cygwin, I guess. In any event, Paul, I am curious if when you were trying to compile and had the same issue, were you using a different version or generation of gcc in the BLAS compilation than in the R compilation? Once again, thank you very much. Avraham Adler -Original Message- From: Paul Gilbert Sent: Thursday, May 30, 2013 12:26 AM To: Adler, Avraham Subject: Re: R-3.0.1 - transient make check failure in splines-EX.r Avraham I resolved this only by switching to a different BLAS on the 32 bit machine.Since no one else seemed to be having problems, I considered it possible that there was a hardware issue on my old 32 bit machine. The R check test failed somewhat randomly, but often. most disconcertingly, it failed because it gives different answers. If you source the code in an R session a few times you have no trouble reproducing this. It gives the impression of an improperly zeroed matrix. (All this from memory, I'm on the road.) Paul On 13-05-28 06:36 PM, Adler, Avraham wrote: Hello. I seem to be having the same problem that Paul had in the thread titled [Rd] R 2.15.2 make check failure on 32-bit --with-blas=-lgoto2 from October of last year https://stat.ethz.ch/pipermail/r-devel/2012-October/065103.html Unfortunately, that thread ended without an answer to his last question. Briefly, I am trying to compile an Rblas for Windows NT 32bit using OpenBlas (successor to GotoBlas) (Nehalem - corei7), and the compiled version passes all tests except for the splines-Ex test in the exact same place that Paul had issues: stopifnot(identical(ns(x), ns(x, df = 1)), + identical(ns(x, df = 2), ns(x, df = 2, knots = NULL)), # not true till 2.15.2 + !is.null(kk - attr(ns(x),
Re: [Rd] [BioC] enabling reproducible research R package management install.package.version BiocLite
I hate to ask what go this thread started but it sounds like someone was counting on exact numeric reproducibility or was there a bug in a specific release? In actual fact, the best way to determine reproducibility is run the code in a variety of packages. Alternatively, you can do everything in java and not assume that calculations commute or associate as the code is modified but it seems pointless. Sensitivity determination would seem to lead to more reprodicible results than trying to keep a specific set of code quirks. I also seem to recall that FPU may have random lower order bits in some cases, same code/data give different results. Alsways assume FP is stochastic and plan on anlayzing the noise. From: amac...@virginia.edu Date: Mon, 4 Mar 2013 16:28:48 -0500 To: m...@stowers.org CC: r-devel@r-project.org; bioconduc...@r-project.org; r-discuss...@listserv.stowers.org Subject: Re: [Rd] [BioC] enabling reproducible research R package management install.package.version BiocLite On Mon, Mar 4, 2013 at 4:13 PM, Cook, Malcolm m...@stowers.org wrote: * where do the dragons lurk webs of interconnected dynamically loaded libraries, identical versions of R compiled with different BLAS/LAPACK options, etc. Go with the VM if you really, truly, want this level of exact reproducibility. An alternative (and arguably more useful) strategy would be to cache results of each computational step, and report when results differ upon re-execution with identical inputs; if you cache sessionInfo along with each result, you can identify which package(s) changed, and begin to hunt down why the change occurred (possibly for the better); couple this with the concept of keeping both code *and* results in version control, then you can move forward with a (re)analysis without being crippled by out-of-date software. -Aaron -- Aaron J. Mackey, PhD Assistant Professor Center for Public Health Genomics University of Virginia amac...@virginia.edu http://www.cphg.virginia.edu/mackey [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Passing R code from webpage
Date: Sun, 17 Feb 2013 04:27:38 -0800 From: mehrotra.pul...@gmail.com To: matevzpav...@gmail.com CC: r-devel@r-project.org Subject: Re: [Rd] Passing R code from webpage hi , First of all you should as this question on r-help rather than on r-devel as this group deals with development related issues with R. Well, personally this potentially a rather substantial issue and while methods may exist to interface R to webservers, it may be worth discussing if there are any things you could do to R to make it run more easily as an R server so that you need not make a bunch of instance to run in this setting. If 10 people on the same machine use R, what duplication is there? Can I launch a background task within R and still work in foreground? Is there anything anyone could should or wants to change in R to make this work better here? As for the package you can use the shiny package from Rstudio. I have provided the links below. http://shiny.rstudio.org/ http://www.rstudio.com/shiny/ regards, Pulkit On Sat, Feb 16, 2013 at 3:48 PM, Matevz Pavlic matevzpav...@gmail.comwrote: Hi all, Is there a way to pass R code from web page (html file) to do some statistics and than plot the output in web browser. I am looking forever at this, and cant find a way. Regards,m -- View this message in context: http://r.789695.n4.nabble.com/Passing-R-code-from-webpage-tp4658800.html Sent from the R devel mailing list archive at Nabble.com. __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Passing R code from webpage
Date: Mon, 18 Feb 2013 03:24:43 -0800 From: matevzpav...@gmail.com To: r-devel@r-project.org Subject: Re: [Rd] Passing R code from webpage Hi, i think all of this i kinda complicated, even though in all the packages authors are saying that minimum code is required. I mean, i am not an IT engineer , but i have created quite some webpages, so i have some knowledge of HTML5, CSS, C# (limited), and also work quite a lot with R and still installing R to run as a webserver seems to be too complicated. I have look in all packages i can imaggine, Rook, ggoleVis, Shinybut in all i get the problem when i want to connect to a MS SQL database. This is the workflow that i'd like to achieve : 1.) in webrowser connect to MS SQL database 2.) do some R statistics and plots 3.) show that in web browser. Well, if you want a version of R that is a webserver that is one thing, if you want a webserver that can call R and don't care how it scales then any server that can execute arbitrary executables would work and you could ask on an apache list for example. The problem is getting scaling. In the case we had, results were easily cached and we had a facility for invoking bash scripts and so t the time the R instances were not a big deal. There are several ways, like Rserve, to make R more server like. The case we had was mostly loaded with short requests so we needed to fix the server threading model and used netty and java. You may be happy with apache, no idea. This becomes an issue of R devel if anyone things there are ways to make multiple R tasks work together better. Accessing DB from R is probably a help question however :) Although here too scaling can be a matter of caching facilities and a custom server may help there. I am pretty sure it can be very complicated, but it just seems so. any help (still) greatly appreciated. BTW: I HAVE posted in R-help, but no responses were given. m regards, m -- View this message in context: http://r.789695.n4.nabble.com/Passing-R-code-from-webpage-tp4658800p4658909.html Sent from the R devel mailing list archive at Nabble.com. __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Passing R code from webpage
Date: Mon, 18 Feb 2013 04:13:25 -0800 From: matevzpav...@gmail.com To: r-devel@r-project.org Subject: Re: [Rd] Passing R code from webpage Yes, y knowledge). well, this getting off topic but I could suggest you go get cygwin and learn to use linux while still on 'dohs LOL I think there is a 'dohs apache version but yes it is unlikely that IIS would interface to stuff like this easily although i have no idea. I know how to connect to MS SQL from R, i just dont know how to do that from web browser. If you can do this it is likely a security flaw. You should do all of this on the server. This has nothing to do with R... thanks, m -- View this message in context: http://r.789695.n4.nabble.com/Passing-R-code-from-webpage-tp4658800p4658915.html Sent from the R devel mailing list archive at Nabble.com. __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Passing R code from webpage
From: simon.urba...@r-project.org Date: Sun, 17 Feb 2013 09:16:57 -0500 To: matevzpav...@gmail.com CC: r-devel@r-project.org Subject: Re: [Rd] Passing R code from webpage On Feb 16, 2013, at 6:48 PM, Matevz Pavlic wrote: Hi all, Is there a way to pass R code from web page (html file) to do some statistics and than plot the output in web browser. I am looking forever at this, and cant find a way. Typically this is done by sending an AJAX request to R and display the result (or simple forms if you want to). I use FastRWeb for this - it even has an example on how you create plots and other output. There is also rApache if you are using apache web server. Depending on how you want to use it, as others have suggested, there are a lot of approaches. Invoking an R process is a bit expensive and having a single R server available is much more efficient however for what we were doing the bigger gain was from caching results. That is, we have maps by area, say state or zip code, where we can tolerate results a few minutes old. So, we used a custom java server to invoke R as part of a general ability to execute bash scripts and then buffer the resulting images in memory. This approach worked well but did potentially require starting a new Rprocess for each request. I had always wanted to make use of shared R instances but never got around to doing this as the scaling never became relevant. Our interface was something simple, like http://host/Rscriptname?zip=0otherkeys=value; which would return an result with appropriate mime type to keep browser happy and have it fit on page ok etc. The point is that anything that can invoke a bash script can invoke R and if you are concerned about doing this efficiently it is not hard to write a simple java app that listens on a port and can invoke R and then do caching to meet your needs if apache etc does not do this easily. Cheers, Simon Regards,m -- View this message in context: http://r.789695.n4.nabble.com/Passing-R-code-from-webpage-tp4658800.html Sent from the R devel mailing list archive at Nabble.com. __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Randomness not due to seed
Date: Tue, 19 Jul 2011 06:13:01 -0700
From: jeroen.o...@stat.ucla.edu
To: r-devel@r-project.org
Subject: [Rd] Randomness not due to seed
I am working on a reproducible computing platform for which I would like to
be able to _exactly_ reproduce an R object. However, I am experiencing
unexpected randomness in some calculations. I have a hard time finding out
exactly how it occurs. The code below illustrates the issue.
mylm1 - lm(dist~speed, data=cars);
mylm2 - lm(dist~speed, data=cars);
identical(mylm1, mylm2); #TRUE
makelm - function(){
return(lm(dist~speed, data=cars));
}
mylm1 - makelm();
mylm2 - makelm();
identical(mylm1, mylm2); #FALSE
When inspecting both objects there seem to be some rounding differences.
Setting a seed does not make a difference. Is there any way I can remove
this randomness and exactly reproduce the object every time?
I don't know if anyone had a specific answer for this but in general floating
point is not
something for which you want to make bitwise equality tests. You can check the
Intel
website for some references but IIRC the FPU can start your calculation with
bits or
settings ( flushing denorms to zero for example) left over from the last user
although I can't document that.
for example, you can probably find more like this suggesting that changes in
alignmnet
and rounding in preamble code can be significant,
http://software.intel.com/en-us/articles/consistency-of-floating-point-results-using-the-intel-compiler/
and of course if your algorithm is numerically sensitive results could change a
lot. Now
its also possible you have unitiliazed or corrupt memory, but you would need to
consider that you will not get bit wise reproduvibility. You can of course go
to java
if you really want that LOL.
--
View this message in context:
http://r.789695.n4.nabble.com/Randomness-not-due-to-seed-tp3678082p3678082.html
Sent from the R devel mailing list archive at Nabble.com.
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] arbitrary size data frame or other stcucts, curious about issues invovled.
Thanks, http://cran.r-project.org/doc/manuals/R-ints.html#Future-directions Normally I'd take more time to digest these things before commenting but a few things struck me right away. First, use of floating point or double as a replacement for int strikes me as going the wrong way as often to get predictable performance you try to tell the compiler you have ints rather than any floating time for which it is free to round. This is even ignoring any performance issue. The other thing is that scaling should not just be an issue of make everything bigger as the growth in both data needs and computer resources is not uniform. I guess my first thought to these constraints and resource issues is to consider a paged dataframe depending upon the point at which the 32-bit int constraint is imposed. A random access data struct does not always get accessed randomly, and often it is purely sequential. Further down the road, it would be nice if algorithms were implemented in a block mode or could communicate their access patterns to the ds or at least tell it to prefetch things that should be needed soon. I guess I'm thinking mostly along the lines of things I've seen from Intel such as ( first things I could find on their site as I have not looked in detail in quite a while), http://www.google.com/search?hl=ensource=hpq=site%3Aintel.com+performance+optimization as once you get around thrashing virtual memory, you'd like to preserve the lower level memory cache hit rates too etc. These are probably not just niceities, at least with VM, as personally I've seen impl related speed issues make simple analyses impractical. Subject: RE: arbitrary size data frame or other stcucts, curious about issues invovled. From: jayemer...@gmail.com To: marchy...@hotmail.com; r-devel@r-project.org Mike, Neither bigmemory nor ff are drop in solutions -- though useful, they are primarily for data storage and management and allowing convenient access to subsets of the data. Direct analysis of the full objects via most R functions is not possible. There are many issues that could be discussed here (and have, previously), including the use of 32-bit integer indexing. There is a nice section Future Directions in the R Internals manual that you might want to look at. Jay - Original message: We keep getting questions on r-help about memory limits and I was curious to know what issues are involved in making common classes like dataframe work with disk and intelligent swapping? That is, sure you can always rely on OS for VM but in theory it should be possible to make a data structure that somehow knows what pieces you will access next and can keep thos somewhere fast. Now of course algorithms should act locally and be block oriented but in any case could communicate with data structures on upcoming access patterns, see a few ms into the future and have the right stuff prefetched. I think things like bigmemory exist but perhaps one issue was that this could not just drop in for data.frame or does it already solve all the problems? Is memory management just a non-issue or is there something that needs to be done to make large data structures work well? -- John W. Emerson (Jay) Associate Professor of Statistics Department of Statistics Yale University http://www.stat.yale.edu/~jay [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] arbitrary size data frame or other stcucts, curious about issues invovled.
Mike, this is all nice, but AFAICS the first part misses the point that there is no 64-bit integer type in the API so there is simply no alternative at the moment. You just said that you don't like it, but you failed to provide a solution ... As for the second part, the idea is not new and is noble, but AFAIK no one so far was able to draft any good proposal as of what the API would look like. It would be very desirable if someone did, though. (BTW your link is useless - linking google searches is pointless as the results vary by request location, user setting etc.). I guess in reverse order, the google link is intended for convenience for those interested as I could not find a specific link and didn't expect much spam to be there ( its all good ) so results may not be preidctable but just like floating point should be close enough for the curious analyst. I'm not trying to provide a solution until I understand the problem. There are many issues with big data and I'll try to explain my concerns but they require talking about them in a bit of an integrated way to see how they relate and to see if my understandings are correct about R ( before I dig into it, want to look for the right things). The 32 bit int still has cardinality of multi-gigs and there are issues of indexes versus memory size. A typical data frame may point to thousands of rows with many colums of mixed type, non being less than 4 bytes of content. So, to simply avoid using up phyiscal memory I would not think the 32 bit issue is a limitation, certainly a square array already has the 64 bit pointer to a given element ( 32*2LOL). An arbitrary size frame, up to the limits of the indexing, could easily exceed physical memory but as I understand it R can bomb at that point or even with VM have speed issues. Simply being able to select the storage order could be a big deal depending on the access pattern: rows, columns bit reversed, etc. This could prevent VM thrashing well before you hit a 32 bit API limit and be transparent beyond adding a new ctor method. And in fact you may have many larger operands, you may want to tell a give df subclass to ONLY keep so much in physical memory at a time. Resource contention and starvation, fighting for food(data) can be a bottleneck. data.frame( storage=bit_reversed, physical_mem_limit=some absolute or relative thing here). In any case, you may be able to imagine adding something like a paging method to a 32 bit api for example that would be transparent to small data sets although I'd have to give it some thought. This would only make sense in cases where aceesses tend to occur in blocks but this could be a lot of situations. I guess I can look at the big memory and related classes for some idea of what is going on here. For purely sequential access I guess I was looking for some kind of streaming data source and then anything related to size may be well contained. Cheers, Simon On Jun 21, 2011, at 6:33 AM, Mike Marchywka wrote: Thanks, http://cran.r-project.org/doc/manuals/R-ints.html#Future-directions Normally I'd take more time to digest these things before commenting but a few things struck me right away. First, use of floating point or double as a replacement for int strikes me as going the wrong way as often to get predictable performance you try to tell the compiler you have ints rather than any floating time for which it is free to round. This is even ignoring any performance issue. The other thing is that scaling should not just be an issue of make everything bigger as the growth in both data needs and computer resources is not uniform. I guess my first thought to these constraints and resource issues is to consider a paged dataframe depending upon the point at which the 32-bit int constraint is imposed. A random access data struct does not always get accessed randomly, and often it is purely sequential. Further down the road, it would be nice if algorithms were implemented in a block mode or could communicate their access patterns to the ds or at least tell it to prefetch things that should be needed soon. I guess I'm thinking mostly along the lines of things I've seen from Intel such as ( first things I could find on their site as I have not looked in detail in quite a while), http://www.google.com/search?hl=ensource=hpq=site%3Aintel.com+performance+optimization as once you get around thrashing virtual memory, you'd like to preserve the lower level memory cache hit rates too etc. These are probably not just niceities, at least with VM, as personally I've seen impl related speed issues make simple analyses impractical. Subject: RE: arbitrary size data frame or other stcucts, curious about issues invovled. From: jayemer...@gmail.com To: marchy...@hotmail.com; r-devel@r
Re: [Rd] Detecting development environment
From: had...@rice.edu Date: Mon, 13 Jun 2011 09:21:07 -0500 To: r-devel@r-project.org Subject: [Rd] Detecting development environment Hi all, Is there a straight-forward, cross-platform way of determining if a user has all the tools needed to develop R packages (i.e. gcc etc)? It doesn't need to be 100%, but should give a rough idea. One idea I had was simply to see if system(R CMD install --help) worked. You could copy a configure script from just about anywhere and adapt it. Not sure what you would do on 'dohs but the script should be able to support cygwin pretty easily. Hadley -- Assistant Professor / Dobelman Family Junior Chair Department of Statistics / Rice University http://had.co.nz/ __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] arbitrary size data frame or other stcucts, curious about issues invovled.
We keep getting questions on r-help about memory limits and I was curious to know what issues are involved in making common classes like dataframe work with disk and intelligent swapping? That is, sure you can always rely on OS for VM but in theory it should be possible to make a data structure that somehow knows what pieces you will access next and can keep thos somewhere fast. Now of course algorithms should act locally and be block oriented but in any case could communicate with data structures on upcoming access patterns, see a few ms into the future and have the right stuff prefetched. I think things like bigmemory exist but perhaps one issue was that this could not just drop in for data.frame or does it already solve all the problems? Is memory management just a non-issue or is there something that needs to be done to make large data structures work well? Thanks. --- 415-264-8477 marchy...@phluant.com Online Advertising and Analytics for Mobile http://www.phluant.com __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
