Re: [R] Running R under Sun Grid Engine with OpenMPI tight integration
On Tue, Jan 27, 2009 at 11:04 AM, Marce wrote: > 2009/1/27 Rainer M Krug : >> On Tue, Jan 27, 2009 at 2:30 AM, Peter Waltman >> wrote: >>> Hi - >>> >>> I saw your posting on the R-help mailing list. Were you ever able to get >>> this working? did you end up switching to use the rsge library? >> >> Yes - that is exactly what I did - I am using rsge or, which is in >> most cases sufficient for me, starting several instances of R and run >> the whole simulation (array processing). >> >> But I would still like to know how I can use the Rmpi and snow on the >> Sun Grid Engine. >> >> Please keep me posted, >> >> Rainer >>> >>> I'm trying to do the same, and not having very much luck getting it going. >>> >>> Thanks! >>> >>> Peter Waltman >>> >> >> >> >> -- >> Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation >> Biology, UCT), Dipl. Phys. (Germany) >> >> Centre of Excellence for Invasion Biology >> Faculty of Science >> Natural Sciences Building >> Private Bag X1 >> University of Stellenbosch >> Matieland 7602 >> South Africa >> >> __ >> R-help@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> > > Hi all, > I have been installed R in SGE but with LAM tight integration, I don't > k now if this help you :s . The problem I had with OpenMPI was that > the load in all nodes waiting for a job and with an instance of R. > Their load up to 100 %, and there isn't a job running on them. > Even so, I will try to configure it with OpenMPI..when I have results > I will post them ok? Yes please - could you please provide some info (and a small example) on how you managed this with LAM? I think we should move this thread to r-sig-hpc, as it is more appropriate there. I CC'd it to there - please respond there and not on r-help. Thanks Rainer > > Saludos > > __ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Faculty of Science Natural Sciences Building Private Bag X1 University of Stellenbosch Matieland 7602 South Africa __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Running R under Sun Grid Engine with OpenMPI tight integration
2009/1/27 Rainer M Krug : > On Tue, Jan 27, 2009 at 2:30 AM, Peter Waltman > wrote: >> Hi - >> >> I saw your posting on the R-help mailing list. Were you ever able to get >> this working? did you end up switching to use the rsge library? > > Yes - that is exactly what I did - I am using rsge or, which is in > most cases sufficient for me, starting several instances of R and run > the whole simulation (array processing). > > But I would still like to know how I can use the Rmpi and snow on the > Sun Grid Engine. > > Please keep me posted, > > Rainer >> >> I'm trying to do the same, and not having very much luck getting it going. >> >> Thanks! >> >> Peter Waltman >> > > > > -- > Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation > Biology, UCT), Dipl. Phys. (Germany) > > Centre of Excellence for Invasion Biology > Faculty of Science > Natural Sciences Building > Private Bag X1 > University of Stellenbosch > Matieland 7602 > South Africa > > __ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > Hi all, I have been installed R in SGE but with LAM tight integration, I don't k now if this help you :s . The problem I had with OpenMPI was that the load in all nodes waiting for a job and with an instance of R. Their load up to 100 %, and there isn't a job running on them. Even so, I will try to configure it with OpenMPI..when I have results I will post them ok? Saludos __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Running R under Sun Grid Engine with OpenMPI tight integration
On Tue, Jan 27, 2009 at 2:30 AM, Peter Waltman wrote: > Hi - > > I saw your posting on the R-help mailing list. Were you ever able to get > this working? did you end up switching to use the rsge library? Yes - that is exactly what I did - I am using rsge or, which is in most cases sufficient for me, starting several instances of R and run the whole simulation (array processing). But I would still like to know how I can use the Rmpi and snow on the Sun Grid Engine. Please keep me posted, Rainer > > I'm trying to do the same, and not having very much luck getting it going. > > Thanks! > > Peter Waltman > -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Faculty of Science Natural Sciences Building Private Bag X1 University of Stellenbosch Matieland 7602 South Africa __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Running R under Sun Grid Engine with OpenMPI tight integration
Hi - I saw your posting on the R-help mailing list. Were you ever able to get this working? did you end up switching to use the rsge library? I'm trying to do the same, and not having very much luck getting it going. Thanks! Peter Waltman [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Running R under Sun Grid Engine with OpenMPI tight integration
On Fri, Dec 5, 2008 at 12:11 AM, Sean Davis <[EMAIL PROTECTED]> wrote: > On Thu, Dec 4, 2008 at 5:21 AM, Rainer M Krug <[EMAIL PROTECTED]> wrote: >> Did you make any progress in using R in the SGE / OpenMPI environment? >> I am asking, because I am struggling with the same problem. Using qsub >> does not give me an interactive session which I can use (wherefore I >> don't see a way of using RMPISNOW and getMPIcluster()) and the >> spawning (makeMPIcluster()) when R is running only yields slave >> processes on the node on which the master is running. >> I have the feeling, that I am missing something really basic - but what is >> it... > > No, I haven't gotten things to work. Even under openmpi without SGE, > I have R crashes on a regular basis when using Rmpi-0.5.5, so I > haven't really pursued it further. Thanks - I'll keep you and the list posted when I find something out. I found the package Rsge which looks promising, and I'll try to get it to work. I'll go to the R-sig-hpc mailing list for this topic. Rainer > > Sean > >> On Thu, Oct 9, 2008 at 3:18 PM, Sean Davis <[EMAIL PROTECTED]> wrote: >>> We have spent some time setting up Sun Grid Engine and OpenMPI on a >>> group of linux boxes. I have created a parallel environment and >>> everything seems to be working. I have Rmpi 0.5.5-5 installed on all >>> machines. I would like to start an interactive R session using, say, >>> 8 processors and then start the Rmpi cluster. I haven't been able to >>> find what needs to be done in order to accomplish this task. Is >>> anyone using R in such an environment that can point me in the right >>> direction? >>> >>> Thanks, >>> Sean >>> >>> __ >>> R-help@r-project.org mailing list >>> https://stat.ethz.ch/mailman/listinfo/r-help >>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >>> and provide commented, minimal, self-contained, reproducible code. >>> >> >> >> >> -- >> Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation >> Biology, UCT), Dipl. Phys. (Germany) >> >> Centre of Excellence for Invasion Biology >> Faculty of Science >> Natural Sciences Building >> Private Bag X1 >> University of Stellenbosch >> Matieland 7602 >> South Africa >> > -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Faculty of Science Natural Sciences Building Private Bag X1 University of Stellenbosch Matieland 7602 South Africa __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Running R under Sun Grid Engine with OpenMPI tight integration
On Thu, Dec 4, 2008 at 5:21 AM, Rainer M Krug <[EMAIL PROTECTED]> wrote: > Did you make any progress in using R in the SGE / OpenMPI environment? > I am asking, because I am struggling with the same problem. Using qsub > does not give me an interactive session which I can use (wherefore I > don't see a way of using RMPISNOW and getMPIcluster()) and the > spawning (makeMPIcluster()) when R is running only yields slave > processes on the node on which the master is running. > I have the feeling, that I am missing something really basic - but what is > it... No, I haven't gotten things to work. Even under openmpi without SGE, I have R crashes on a regular basis when using Rmpi-0.5.5, so I haven't really pursued it further. Sean > On Thu, Oct 9, 2008 at 3:18 PM, Sean Davis <[EMAIL PROTECTED]> wrote: >> We have spent some time setting up Sun Grid Engine and OpenMPI on a >> group of linux boxes. I have created a parallel environment and >> everything seems to be working. I have Rmpi 0.5.5-5 installed on all >> machines. I would like to start an interactive R session using, say, >> 8 processors and then start the Rmpi cluster. I haven't been able to >> find what needs to be done in order to accomplish this task. Is >> anyone using R in such an environment that can point me in the right >> direction? >> >> Thanks, >> Sean >> >> __ >> R-help@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> > > > > -- > Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation > Biology, UCT), Dipl. Phys. (Germany) > > Centre of Excellence for Invasion Biology > Faculty of Science > Natural Sciences Building > Private Bag X1 > University of Stellenbosch > Matieland 7602 > South Africa > __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Running R under Sun Grid Engine with OpenMPI tight integration
Did you make any progress in using R in the SGE / OpenMPI environment? I am asking, because I am struggling with the same problem. Using qsub does not give me an interactive session which I can use (wherefore I don't see a way of using RMPISNOW and getMPIcluster()) and the spawning (makeMPIcluster()) when R is running only yields slave processes on the node on which the master is running. I have the feeling, that I am missing something really basic - but what is it... Rainer On Thu, Oct 9, 2008 at 3:18 PM, Sean Davis <[EMAIL PROTECTED]> wrote: > We have spent some time setting up Sun Grid Engine and OpenMPI on a > group of linux boxes. I have created a parallel environment and > everything seems to be working. I have Rmpi 0.5.5-5 installed on all > machines. I would like to start an interactive R session using, say, > 8 processors and then start the Rmpi cluster. I haven't been able to > find what needs to be done in order to accomplish this task. Is > anyone using R in such an environment that can point me in the right > direction? > > Thanks, > Sean > > __ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Faculty of Science Natural Sciences Building Private Bag X1 University of Stellenbosch Matieland 7602 South Africa __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Running R under Sun Grid Engine with OpenMPI tight integration
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, actually there is no perfect way to use R one multiprocessors. There are two options: * using mpi: This means starting several R sessions and using MPI for the communication between the R sessions. Task distribution, fail over solutions, ... have to be implemented by your self. MPI was developed for multicomputer systems. So it is not very efficient to use it at a multiprocessor machine. But it works! * using multiThreads or openMP: There are packages from Luke Tierney using multi threading or openMP for basic R math functions. http://www.cs.uiowa.edu/~luke/R/experimental/ This are especially for multiprocessors designed math functions. You can achieve nice speedups. But unfortunately this is just working for the basic math functions. Have a look to the useR2008 conference and Dirks Tutorium. There you can find a lot of information! Markus Chi Chan wrote: > On Thu, Oct 9, 2008 at 9:02 AM, Dirk Eddelbuettel <[EMAIL PROTECTED]> wrote: >> You could have a look at the slides from my UseR! 2008 tutorial [1] on >> 'high-performance computing with R' which covered Open MPI / Rmpi and >> includes examples. >> >> It may just be that you are expecting something that cannot be done. There >> is no 'R session on 8 processors'. R is still single-threaded, and neither >> Open MPI nor the Sun Grid Engine changes that. But what you can do is to >> start R on a node in your cluster and then use Open MPI to distribute load >> across the 8 processors. There are examples in my slides. > > So R uses MPI for task distribution only? That sounds like not very > fine grained parallelism. > > Can I use rsh to start remote R instead of MPI? > > --Chi > > >> Hope this helps, Dirk >> >> [1] http://dirk.eddelbuettel.com/presentations.html >> >> -- >> Three out of two people have difficulties with fractions. >> >> __ >> R-help@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> > > __ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. - -- Dipl.-Tech. Math. Markus Schmidberger Ludwig-Maximilians-Universität München IBE - Institut für medizinische Informationsverarbeitung, Biometrie und Epidemiologie Marchioninistr. 15, D-81377 Muenchen URL: http://www.ibe.med.uni-muenchen.de Mail: Markus.Schmidberger [at] ibe.med.uni-muenchen.de Tel: +49 (089) 7095 - 4599 -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.4-svn0 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFI7jZPwxqyUctZsggRAm9YAJ0VQ3JwzGfYp7ZmwbOJi7HxCTD/rQCglSUP eKJzO3NelDa55vMD3b+lp3o= =5CyM -END PGP SIGNATURE- __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Running R under Sun Grid Engine with OpenMPI tight integration
On Thu, Oct 9, 2008 at 9:02 AM, Dirk Eddelbuettel <[EMAIL PROTECTED]> wrote: > You could have a look at the slides from my UseR! 2008 tutorial [1] on > 'high-performance computing with R' which covered Open MPI / Rmpi and > includes examples. > > It may just be that you are expecting something that cannot be done. There > is no 'R session on 8 processors'. R is still single-threaded, and neither > Open MPI nor the Sun Grid Engine changes that. But what you can do is to > start R on a node in your cluster and then use Open MPI to distribute load > across the 8 processors. There are examples in my slides. So R uses MPI for task distribution only? That sounds like not very fine grained parallelism. Can I use rsh to start remote R instead of MPI? --Chi > > Hope this helps, Dirk > > [1] http://dirk.eddelbuettel.com/presentations.html > > -- > Three out of two people have difficulties with fractions. > > __ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Running R under Sun Grid Engine with OpenMPI tight integration
On 9 October 2008 at 09:18, Sean Davis wrote: | We have spent some time setting up Sun Grid Engine and OpenMPI on a | group of linux boxes. I have created a parallel environment and | everything seems to be working. I have Rmpi 0.5.5-5 installed on all | machines. I would like to start an interactive R session using, say, | 8 processors and then start the Rmpi cluster. I haven't been able to | find what needs to be done in order to accomplish this task. Is | anyone using R in such an environment that can point me in the right | direction? You could have a look at the slides from my UseR! 2008 tutorial [1] on 'high-performance computing with R' which covered Open MPI / Rmpi and includes examples. It may just be that you are expecting something that cannot be done. There is no 'R session on 8 processors'. R is still single-threaded, and neither Open MPI nor the Sun Grid Engine changes that. But what you can do is to start R on a node in your cluster and then use Open MPI to distribute load across the 8 processors. There are examples in my slides. Hope this helps, Dirk [1] http://dirk.eddelbuettel.com/presentations.html -- Three out of two people have difficulties with fractions. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Running R under Sun Grid Engine with OpenMPI tight integration
We have spent some time setting up Sun Grid Engine and OpenMPI on a group of linux boxes. I have created a parallel environment and everything seems to be working. I have Rmpi 0.5.5-5 installed on all machines. I would like to start an interactive R session using, say, 8 processors and then start the Rmpi cluster. I haven't been able to find what needs to be done in order to accomplish this task. Is anyone using R in such an environment that can point me in the right direction? Thanks, Sean __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
