Thank you Rayson for your answer, Amazon is *the* cloud as you say, but we are working on a private cloud to do our experiment. The different types of instance Eucalyptus provides by default are m1.small, c1.medium, m1.large, m1.xlarge and c1.xlarge, so there is no tiny nor 4xlarge.
I try to remove the features that are not compatible with Euca, (such as describe filters), but it was kind of dirty work. So keeping starcluster in mind, we decided to have a look to slurm perl API, combined to Net::Amazon::EC2. Both of these libs seem to work really fine, so maybe we're gonna write some tools with these. We think, we are gonna let a clean install of slurm with the linear select plugin and manage the queuing in our code. With some of the scheduling strategies we want to implement, we'll sometimes need to move tasks from on node queue to another node. We are currently looking if we can do that with the Perl API. -- Léo On 06/21/2012 10:31 PM, Rayson Ho wrote: > On Thu, Jun 21, 2012 at 6:25 AM, Léo Unbekandt<[email protected]> wrote: >> The slurm starcluster plugin seems really interesting for us. However, our >> cloud is Eucalpytus based, and they use the "boto" lib v2.3 to communicate >> with AWS-EC2, but this version compatible with Eucalyptus 2.0.x, and we are >> not sure it is with the 3.0.x. >> >> But this is not really your stuff there, but if you have some information, >> it would be nice > There were patches that fix OpenStack& Eucalpytus issues floating on > the starcluster mailing list a while ago. > > Also, a few months ago we (ie. Open Grid Scheduler / Grid Engine& > Scalable Logic) got an invite to testdrive the HP Cloud, which is > OpenStack 1.1 based. We were planning to run StarCluster on the HP > Cloud, but as there is a bit more work to get it to work, we decided > to wait and see if there's real interest in using the HP Cloud (IMO, > AWS& EC2 is *the* cloud, all other vendors are lacking too many > things - unless you only use AWS for EC2& S3). > > You can, if you really want, try to run StarCluster with OpenStack& > Eucalpytus. However, since the instance types of your private cloud is > likely to be different than the ones on EC2 (do you have GC1.4xlarge > or t1.micro?), you will need to disable the code that does sanity > check in StarCluster. > > Rayson > > > > > > . >> -- >> Léo Unbekandt >> >> Team ICPS in the LSIIT Labs >> Strasbourg — France >> >> >> On 06/20/2012 03:19 PM, Jharrod W. LaFon wrote: >> >> Hi Léo, >> >> You may want to look at StarCluster (http://web.mit.edu/star/cluster/) and a >> fork of it that features a slurm load balancer: >> https://github.com/jlafon/StarCluster >> >> Getting started: >> https://github.com/jlafon/StarCluster/wiki/Getting-Started-with-SLURM-on-Amazon-EC2 >> >> -- >> Jharrod LaFon >> >> On Wed, Jun 20, 2012 at 4:35 AM, Léo Unbekandt<[email protected]> wrote: >>> >>> Hi, >>> >>> I'm doing an internship in a research team and we are working on >>> interfacing slurm with some cloud platforms (EC2/OpenStack/Eucalyptus,�). >>> >>> Our goal is to optimize the number of virtual machines that are >>> computing our jobs (finally to reduce the price which is about >>> 0.08cts/core/hour). So we need to implement different scheduling >>> algorithms which must dynamically ask our IaaS controller to boot one or >>> several virtual machines (which, once have booted become available slurm >>> nodes, waiting for jobs) >>> >>> The question is, how do you think we must implement these algorithms ? >>> Which kind of plugin must we create ? We first start a new � select � >>> plugin (functions select_p_job_test mainly), but we are not sure that's >>> the good way to do it. >>> >>> Furthermore, in an elastic cloud like Amazon's, we pay per hour, so when >>> the workload decrease we need to shutdown virtual machines. We have seen >>> that the scheduler job is called really often, do you think it's a good >>> idea to check at that time ? (ie. A node is up since 58min and doesn't >>> have anything to do right now so we shutdown it.) >>> >>> Thanks for your answers >>> >>> -- >>> L�o Unbekandt >>> Team ICPS in the LSIIT Labs >>> Strasbourg � France >> >> >> >> -- >> -- >> Jharrod LaFon >> >> >> > > > -- > ================================================== > Open Grid Scheduler - The Official Open Source Grid Engine > http://gridscheduler.sourceforge.net/
