Re: [galaxy-dev] Amazon cloud setup for galaxy training

[Enis Afgan]

Hello Anne,
Overall, what we typically recommend is to launch a new instance and
configure it as desired (eg, with tools, histories, and datasets of
interest for the workshop). Then share that instance (using the CloudMan
share-a-cluster feature
 -> small
green icon next to the cluster name on the main CloudMan console) and
create a number of replicas of the shared instance for use during the
workshop. For RNA-Seq workshops, for example, we generally stick with about
10 participants per instance. Here's a page with some empirical-based
recommendations regarding choosing instance types:
https://wiki.galaxyproject.org/CloudMan/AWS/CapacityPlanning

We are also currently working on a templated training image

that would come populated with a number of training artifacts. This may be
ready for use by Sep 24th.

Finally, Dave Clements (CC'd) has a lot of hands on experience with setups
for various workshops so he may be able to chime in and provide additional
feedback.

Hope this helps and let us know if you have any more questions,
Enis

On Fri, Sep 12, 2014 at 10:15 AM, Anne Pajon 
wrote:

> Dear,
>
> I am setting up with my colleague Jing an Introductory Galaxy training
> course at the University of Cambridge due to on the 24th September.
>
> I am looking for help on how to setup the cloud to have a smooth user
> experience during the workshop. Would you have any links with information
> to recommend or any advices on how to set it up? Many thanks in advance.
>
> Kindest regards,
> Anne.
> --
> Dr Anne Pajon - Bioinformatics Core
> Cancer Research UK - Cambridge Institute
> Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE
> anne.pa...@cruk.cam.ac.uk | +44 (0)1223 769 631
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Amazon cloud setup for galaxy training

[Anne Pajon]

Dear,

I am setting up with my colleague Jing an Introductory Galaxy training course 
at the University of Cambridge due to on the 24th September. 

I am looking for help on how to setup the cloud to have a smooth user 
experience during the workshop. Would you have any links with information to 
recommend or any advices on how to set it up? Many thanks in advance.

Kindest regards,
Anne.
--
Dr Anne Pajon - Bioinformatics Core
Cancer Research UK - Cambridge Institute
Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE
anne.pa...@cruk.cam.ac.uk | +44 (0)1223 769 631


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Amazon EC2 Cloudman: Running parallel Tophats and Master node has 260% load, worker idle and at 46%

[Brian Lin]

Hi guys, I'm fairly new to cloud computing and about 2 days in to using
cloudman for galaxy...

I have setup a m2.4xlarge master node and flexible load for up to 4 workers
of the m2.xlarge type, minimum one.
I was able to upload 6 samples of paired end RNAseq->12 files, gz file
sizes around 5-8 gb.
Grooming files took about a day, but my previous experience was on an
in-house Galaxy install which was pretty small so I didn't think of
anything at the time.
I started 3 Tophat jobs and I noticed the UI being a bit sluggish to
respond, added a 4th one hoping it might push it over the edge for the
worker nodes.

Unfortunately, 12 hours later, the Tophats are still running, the master
node is way over 100%, and the worker is reported idle.
While the cluster log has a few errors in it, it ends saying that the
instance for the worker node is ready, despite it encountering errors
adding it to the SGE host, code1.

Has anyone run into this before or has any insight to fixing this? I'm
pasting the lower portion of the status log below.

Thanks very much for any help! (Also, several other emails about cloud
installs were directed to /dev. If that's not the right place for this
question, I apologize and can change to the /user list.)
-- 
Brian Lin
cont...@brian-lin.com
brian@tufts.edu





13:15:38 - Instance 'i-e2bb0491' reported alive
13:15:38 - Sent master public key to worker instance 'i-e2bb0491'.
13:15:54 - Adding instance i-e2bb0491 as SGE administrative host.
13:16:06 - Adding instance 'i-e2bb0491' to SGE execution host list.
13:16:11 - Process encountered problems adding instance 'i-e2bb0491' as an
SGE execution host. Process returned code 1
13:16:26 - Waiting on worker instance 'i-e2bb0491' to configure itself...
13:17:05 - Instance 'i-e2bb0491' (IP: 23.23.24.81) ready
13:42:23 - Rebooting instance i-e2bb0491 (reboot #3).
13:44:27 - Instance 'i-e2bb0491' reported alive
13:44:27 - Sent master public key to worker instance 'i-e2bb0491'.
13:44:56 - Adding instance i-e2bb0491 as SGE administrative host.
13:44:56 - Adding instance 'i-e2bb0491' to SGE execution host list.
13:44:56 - Process encountered problems adding instance 'i-e2bb0491' as an
SGE execution host. Process returned code 1
13:44:56 - Waiting on worker instance 'i-e2bb0491' to configure itself...
13:46:09 - Instance 'i-e2bb0491' (IP: 23.23.24.81) ready
15:19:37 - Rebooting instance i-e2bb0491 (reboot #4).
15:21:16 - Instance 'i-e2bb0491' reported alive
15:21:16 - Sent master public key to worker instance 'i-e2bb0491'.
15:21:38 - Adding instance i-e2bb0491 as SGE administrative host.
15:21:44 - Adding instance 'i-e2bb0491' to SGE execution host list.
15:21:48 - Process encountered problems adding instance 'i-e2bb0491' as an
SGE execution host. Process returned code 1
15:21:51 - Waiting on worker instance 'i-e2bb0491' to configure itself...
15:22:13 - Instance 'i-e2bb0491' (IP: 23.23.24.81) ready
15:54:32 - Instance i-e2bb0491 not responding after 4 reboots. Terminating
instance.
15:54:32 - Terminating instance i-e2bb0491
15:54:35 - Instance 'i-e2bb0491' removed from the internal instance list.
15:56:10 - Adding 1 on-demand instance(s)
15:56:14 - Cannot get cloud instance object without an instance ID?
15:58:26 - Instance 'i-98932deb' reported alive
15:58:26 - Sent master public key to worker instance 'i-98932deb'.
15:59:02 - Adding instance i-98932deb as SGE administrative host.
15:59:09 - Adding instance 'i-98932deb' to SGE execution host list.
15:59:17 - Successfully added instance 'i-98932deb' to SGE
15:59:17 - Waiting on worker instance 'i-98932deb' to configure itself...
15:59:46 - Instance 'i-98932deb' (IP: 54.234.100.22) ready
16:12:47 - Rebooting instance i-98932deb (reboot #1).
16:12:47 - Rebooting instance i-98932deb (reboot #1).
16:12:47 - Rebooting instance i-98932deb (reboot #1).
16:14:06 - Instance 'i-98932deb' reported alive
16:14:06 - Sent master public key to worker instance 'i-98932deb'.
16:14:29 - Adding instance i-98932deb as SGE administrative host.
16:14:29 - Adding instance 'i-98932deb' to SGE execution host list.
16:14:29 - Process encountered problems adding instance 'i-98932deb' as an
SGE execution host. Process returned code 1
16:14:29 - Waiting on worker instance 'i-98932deb' to configure itself...
16:14:39 - Instance 'i-98932deb' (IP: 54.234.100.22) ready
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Amazon

[Dannon Baker]

For this instance, you'll need to restart using the old method for launching 
via the console, specifying the zone 1b.  Detection of the zone volumes are in 
for existing clusters and specifying those for launch is on the short list of 
things coming up for cloud launch. 

On Oct 31, 2012, at 10:50 AM, Scooter Willis  wrote:

> Tried it again and same error message. The volume was originally created in 
> us-east-1b and newly created instances are being started in us-east-1a. 
> Shouldn't the availability zone be set to us-east-1b when the instance is 
> requested or that info stored in the properties file in the S3 bucket?
> 
> Any suggestions?
> 
> From: Scooter Willis 
> Date: Wednesday, October 31, 2012 10:32 AM
> To: "galaxy-dev@lists.bx.psu.edu" 
> Subject: Amazon
> 
> Started up a cluster on Amazon using the Launch a Galaxy Cloud Instance and 
> got the following message. Since I don't have any control over where the 
> instances are run not sure how I can control this. The last 4 or 5 times I 
> have started up an existing instance has worked with no problem.
> 
> Messages (CRITICAL messages cannot be dismissed.)
> [CRITICAL] Volume 'vol-f882ca85' is located in the wrong availability zone 
> for this instance. You MUST terminate this instance and start a new one in 
> zone 'us-east-1a'. (2012-10-31 14:25:20)
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> 
>  http://lists.bx.psu.edu/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Amazon

[Scooter Willis]

Tried it again and same error message. The volume was originally created in 
us-east-1b and newly created instances are being started in us-east-1a. 
Shouldn't the availability zone be set to us-east-1b when the instance is 
requested or that info stored in the properties file in the S3 bucket?

Any suggestions?

From: Scooter Willis mailto:hwil...@scripps.edu>>
Date: Wednesday, October 31, 2012 10:32 AM
To: "galaxy-dev@lists.bx.psu.edu" 
mailto:galaxy-dev@lists.bx.psu.edu>>
Subject: Amazon

Started up a cluster on Amazon using the Launch a Galaxy Cloud Instance and got 
the following message. Since I don't have any control over where the instances 
are run not sure how I can control this. The last 4 or 5 times I have started 
up an existing instance has worked with no problem.

Messages (CRITICAL messages cannot be dismissed.)

 1.  [CRITICAL] Volume 'vol-f882ca85' is located in the wrong availability zone 
for this instance. You MUST terminate this instance and start a new one in zone 
'us-east-1a'. (2012-10-31 14:25:20)
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] Amazon

[Scooter Willis]

Started up a cluster on Amazon using the Launch a Galaxy Cloud Instance and got 
the following message. Since I don't have any control over where the instances 
are run not sure how I can control this. The last 4 or 5 times I have started 
up an existing instance has worked with no problem.

Messages (CRITICAL messages cannot be dismissed.)

 1.  [CRITICAL] Volume 'vol-f882ca85' is located in the wrong availability zone 
for this instance. You MUST terminate this instance and start a new one in zone 
'us-east-1a'. (2012-10-31 14:25:20)
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Amazon cloud formation for galaxy

[Enis Afgan]

Yeah, also that is is pretty much how CloudMan handles
the infrastructure management now - it has it's own descriptor (i.e.,
template) and from there it handles all the provisioning and coordination of
the underlying resources. Granted, having Amazon do it would add to the
robustness of the overall system (especially because they seem to have also
implemented a rollback option) but it would also tie Galaxy CloudMan to AWS
even more. However, the intent is really to support other
cloud infrastructure providers as more centers roll out their own
cloud deployments...

Enis

On Fri, Feb 25, 2011 at 2:07 PM, Ry4an Brase wrote:

> Today's release of amazon cloud formation has to make the Galaxy Cloud
> stuff a bit easier:
>
> http://aws.typepad.com/aws/2011/02/cloudformation-create-your-aws-stack-from-a-recipe.html
>
> In theory with a description file like this:
>
>
> https://s3.amazonaws.com/cloudformation-templates/CloudFormationSample_WordPress.template
>
> Once can define an entirely cluster to bring up from custom .amis with a
> single click.  Exciting.
>
> --
> Ry4an Brase 612-626-6575
> Software Developer  Application Development
> University of Minnesota Supercomputing Institutehttp://www.msi.umn.edu
> ___
> To manage your subscriptions to this and other Galaxy lists, please use the
> interface at:
>
>  http://lists.bx.psu.edu/
>
___
To manage your subscriptions to this and other Galaxy lists, please use the 
interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] Amazon cloud formation for galaxy

[Ry4an Brase]

Today's release of amazon cloud formation has to make the Galaxy Cloud
stuff a bit easier:
http://aws.typepad.com/aws/2011/02/cloudformation-create-your-aws-stack-from-a-recipe.html

In theory with a description file like this: 

https://s3.amazonaws.com/cloudformation-templates/CloudFormationSample_WordPress.template

Once can define an entirely cluster to bring up from custom .amis with a
single click.  Exciting.

-- 
Ry4an Brase 612-626-6575
Software Developer  Application Development
University of Minnesota Supercomputing Institutehttp://www.msi.umn.edu
___
To manage your subscriptions to this and other Galaxy lists, please use the 
interface at:

  http://lists.bx.psu.edu/