On Jul 16, 2014, at 9:51 AM, Prof Brian Ripley <rip...@stats.ox.ac.uk> wrote:
> On 16/07/2014 14:07, Gregory R. Warnes wrote: >> Hi Varadharajan, >> >> Linux uses a copy-on-write for the memory image of forked processes. > > But Linux copied it from Unix and I see no mention of Linux in the posting > being replied to. The reference was indirect. The UKSM documentation indicates that it is implemented as a patch for the Linux kernel. >> Thus, you may also get significant memory savings by launching a single R >> process, loading your large data object, and then using fork::fork() to >> split off the other worker process. > > Or using the more refined interface in package 'parallel' (which is portable, > unlike package 'fork': see its CRAN check results). Thank you for pointing out the issues with fork, I’ll take a look at what is going on with Solaris. -Greg > >> -Greg >> >> Sent from my iPad >> >>> On Jul 16, 2014, at 5:07 AM, Varadharajan Mukundan <srinath...@gmail.com> >>> wrote: >>> >>> [Sending it again in plain text mode] >>> >>> Greetings, >>> >>> We've a fairly large dataset (around 60GB) to be loaded and crunched >>> in real time. The kind of data operations that will be performed on >>> this data are simple read only aggregates after filtering the >>> data.table instance based on the parameters that will passed in real >>> time. We need to have more than one instance of such R process running >>> to serve different testing environments (each testing environment has >>> fairly identical dataset but do have a *small amount of changes*). As >>> we all know, data.table loads the entire dataset into memory for >>> processing and hence we are facing a constraint on number of such >>> process that we could run on the machine. On a 128GB RAM machine, we >>> are coming up with ways in which we could reduce the memory footprint >>> so that we can try to spawn more instances and use the resources >>> efficiently. One of the approaches we tried out was memory >>> de-duplication using UKSM >>> (http://kerneldedup.org/en/projects/uksm/introduction), given that we >>> did have few idle cpu cores. Outcome of the experiment was quite >>> impressive, considering that the effort to set it up was quite less >>> and the entire approach considers the application layer as a black >>> box. >>> >>> Quick snapshot of the results: >>> 1 Instance (without UKSM): ~60GB RAM was being used >>> 1 Instance (with UKSM): ~53 GB RAM was being used >>> >>> 2 Instance (without UKSM): ~125GB RAM was being used >>> 2 Instance (with UKSM): ~81 GB RAM was being used >>> >>> We can see that around 44 GB of RAM was saved after UKSM merged >>> similar pages and all this for a compromise of 1 CPU core on a 48 >>> core machine. We did not feel any noticeable degradation of >>> performance because the data is refreshed by a batch job only once >>> (every morning); UKSM gets in at this time and performs the same page >>> merging and for the rest of day, its just read only analysis. The kind >>> of queries we fire on the dataset at most scans 2-3GB of the entire >>> dataset and hence the query subset spike was low as well. >>> >>> We're interested in knowing if this is a plausible solution to this >>> problem? Any other points/solutions that we should be considering? > > > -- > Brian D. Ripley, rip...@stats.ox.ac.uk > Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ > University of Oxford, Tel: +44 1865 272861 (self) > 1 South Parks Road, +44 1865 272866 (PA) > Oxford OX1 3TG, UK Fax: +44 1865 272595 ______________________________________________ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
