Re: [galaxy-dev] Reducing the size of a volume in Galaxy Cloud
Hi Denise, Although the instructions you found can still be used, a couple of steps can be updated now so I'll to that now and hopefully answer your questions in the process. Updated instructions: 1. Start a brand new Galaxy CloudMan cluster/instance (this will be only a temporary cluster needed for the duration of the following steps) 2. ssh into the master instance and delete whatever genomes you don't need/want (these are all located under /mnt/galaxyIndices; using '*rm -rf dir name*' is the command to use) 3. From CloudMan's Admin page, add a new file system of type 'new volume' and desired size (ie, size of data on /mnt/galaxyIndices after you've deleted what you didn't want). Give this file system a name 'indices'. 4. ssh into the master instance and copy over the data from /mnt/galaxyIndices to /mnt/indices using command: rsync -avz /mnt/galaxyIndices/ /mnt/indices/. (do this as 'galaxy' user) 5. Unmount the new 'indices' file system using command: sudo umount /mnt/indices 6. From the AWS console, create a snapshot of the volume for the 'indices' file system. You can retrieve the volume ID from the file system 'details' on the CloudMan Admin page 7. For the cluster you want to keep around (while it is terminated), edit *persistent_data.yaml* in it's bucket on S3 (you can get the bucket name from the CloudMan Admin page - it will look like so *cm-hash*) and replace the existing snap ID for the *galaxyIndices* with the snapshot ID you got in the previous step 8. Start that cluster and you should have a file system from the new snapshot mounted 9. Terminate delete the cluster you created in step 1 Here are my questions to help me to get through step 4: 1) Step 1: Is a “cluster” the same thing as an “instance”? Yes 2) Step 2: I deleted the directories for individual genomes using rm –rf . Is that the correct approach? Yes 3) Step 3: Do I add the newly created EBS volume to the same instance where I deleted the genomes? Or is it added the instance I want to keep? You add the new file system (ie, EBS volume) to the newly created cluster (and you also delete the extra genomes on that same cluster). 4) Step 3: I can see how to attach this newly created volume using the AWS EC2 management console, but how do I mount it? (and unmount it in Step 5?) With the updated instructions above, mounting the file system by hand is no longer necessary. 5) Step 4: what is the syntax for the rsync (or cp) command to copy directories/files from one volume to another volume (within the same instance, or in if they are in different instances)? I included in the command in the instructions now. Hope this helps. Let us know if you have any trouble or if you have any more questions, Enis ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] My CloudMan EC2 instance won't start Galaxy
Hi Jim, How did you launch your instance? If you still have the instance up, could you please also forward the first ~200 lines of /mnt/cm/paster.log (deleting any identifiable info from the log). Sorry abou the trouble and thanks for reporting it, Enis On Sat, Oct 25, 2014 at 8:58 PM, Jim White jimwh...@uw.edu wrote: When I start CloudMan following the instructions at (https://wiki.galaxyproject.org/CloudMan) (ami-a7dbf6ce, Galaxy CloudMan 2.3) I get pretty far in the process but then Galaxy won't start because the EBS volumes are never mounted. The Volume console shows that the volumes get created based on the public snapshots but they never go to in use which suggest that the call to mount them isn't getting done for some reason. I probably will end up building my own CloudMan AMI (I'm mostly interested in HTCondor DAGman rather than Galaxy) but I figured you'd want to know about this issue. Thanks, Jim Admin Console:https://dl.dropboxusercontent.com/u/73840666/CloudMan%20Trouble/Admin%20Console.png EC2 Volumes:https://dl.dropboxusercontent.com/u/73840666/CloudMan%20Trouble/Volumes%20OK.png Logs: Cluster status log 13:22:17 - Master starting 13:22:18 - Successfully retrieved root user's public key from file. 13:22:18 - Completed the initial cluster startup process. This is a new cluster; waiting to configure the type. 13:22:22 - Migration service prerequisites OK; starting the service 13:22:22 - SGE service prerequisites OK; starting the service 13:22:28 - Setting up SGE... 13:22:36 - Initializing 'Galaxy' cluster type with storage type 'volume'. Please wait... 13:22:40 - For filesystem galaxy, unknown kind: galaxy 13:22:40 - For filesystem galaxyIndices, unknown kind: galaxyIndices 13:22:45 - PostgreSQL data directory '/mnt/galaxy/db' does not exist (yet?) 13:22:46 - Adding volume vol-9fe10c87 (FS object for galaxy)... 13:22:46 - Could not determine next available device from frozenset([]) 13:22:46 - Error adding file system service FS object for galaxy: 'NoneType' object is not iterable 13:22:46 - Adding volume vol-6ae10c72 (FS object for galaxyIndices)... 13:23:32 - Could not determine next available device from frozenset([]) 13:23:32 - Error adding file system service FS object for galaxyIndices: 'NoneType' object is not iterable 13:23:36 - Empty disk usage for FS transient_nfs 13:23:36 - STATUS CHECK: File system named 'galaxy' is not mounted. Error code 0 13:23:36 - STATUS CHECK: File system named 'galaxyIndices' is not mounted. Error code 0 13:23:36 - PostgreSQL data directory '/mnt/galaxy/db' does not exist (yet?) 13:23:49 - Empty disk usage for FS transient_nfs 13:23:49 - STATUS CHECK: File system named 'galaxy' is not mounted. Error code 0 13:23:49 - STATUS CHECK: File system named 'galaxyIndices' is not mounted. Error code 0 13:23:49 - PostgreSQL data directory '/mnt/galaxy/db' does not exist (yet?) 13:24:01 - Empty disk usage for FS transient_nfs 13:24:01 - STATUS CHECK: File system named 'galaxy' is not mounted. Error code 0 13:24:01 - STATUS CHECK: File system named 'galaxyIndices' is not mounted. Error code 0 13:24:01 - PostgreSQL data directory '/mnt/galaxy/db' does not exist (yet?) 13:24:14 - Empty disk usage for FS transient_nfs 13:24:14 - STATUS CHECK: File system named 'galaxy' is not mounted. Error code 0 13:24:14 - STATUS CHECK: File system named 'galaxyIndices' is not mounted. Error code 0 Admin Console log 2014-10-25 13:27:42,910 DEBUG master:2445 Monitor adding service 'Galaxy' 2014-10-25 13:27:42,910 DEBUG __init__:300 Galaxy service prerequisites are not yet satisfied, waiting for: [, , , , ]. Setting Galaxy service state to 'Unstarted' 2014-10-25 13:27:42,911 DEBUG master:2445 Monitor adding service 'GalaxyReports' 2014-10-25 13:27:42,911 DEBUG __init__:300 GalaxyReports service prerequisites are not yet satisfied, waiting for: []. Setting GalaxyReports service state to 'Unstarted' 2014-10-25 13:27:47,307 DEBUGsge:552 qstat: ['al...@ip-10-158-92-92.ec2.inte BIP 0/0/1 0.33 lx24-amd64 '] 2014-10-25 13:27:47,344 WARNING filesystem:518 Empty disk usage for FS transient_nfs 2014-10-25 13:27:47,366 ERROR filesystem:582 STATUS CHECK: File system named 'galaxy' is not mounted. Error code 0 2014-10-25 13:27:47,390 ERROR filesystem:582 STATUS CHECK: File system named 'galaxyIndices' is not mounted. Error code 0 2014-10-25 13:27:47,391 WARNING postgres:199 PostgreSQL data directory '/mnt/galaxy/db' does not exist (yet?) 2014-10-25 13:27:47,404 DEBUG master:2534 SS: Migration..Completed; SGE..OK; FS object for transient_nfs..OK; PSS..Unstarted; FS object for galaxy..Error; FS object for galaxyIndices..Error; Postgres..Unstarted; ProFTPd..Unstarted; Galaxy..Unstarted; GalaxyReports..Unstarted; 2014-10-25 13:27:47,404 DEBUG master:2445 Monitor adding service 'PSS' 2014-10-25 13:27:47,404 DEBUG
Re: [galaxy-dev] Galaxy instance file upload problem
As Hans said, this might reguire a change in the proxy server configuration as discussed here: http://gmod.827538.n3.nabble.com/Data-uploaded-with-new-upload-tool-doesn-t-get-added-to-history-td4046375.html On Fri, Oct 17, 2014 at 2:43 AM, Hans-Rudolf Hotz h...@fmi.ch wrote: Hi Bongsoo we had once a similar situation with one of our development Galaxy installations. This was due to a change in the proxy settings of the server. Regards, Hans-Rudolf On 10/17/2014 01:06 AM, Bongsoo Park wrote: Aysam, Thanks for your reply. It works well on usegalaxy.org http://usegalaxy.org, but it doesn't work in the Galaxy instance I've developed. I downloaded the latest Galaxy version, and installed on the Redhat 6.5 system. I created a galaxy user, and just ran it as usual. I have to update any part of the configuration? I have to allow any specific port to use file upload function? Thanks! Best, Bongsoo On Thu, Oct 16, 2014 at 6:25 PM, Aysam Guerler aysam.guer...@gmail.com mailto:aysam.guer...@gmail.com wrote: Hey Bongsoo, Does this happen on usegalaxy.org http://usegalaxy.org? Also does this happen with other files too? Thanks, Sam On Thu, Oct 16, 2014 at 2:49 PM, Bongsoo Park bx...@psu.edu mailto:bx...@psu.edu wrote: Hi folks, I encountered the problem in file upload. The error message is like below 'Failed: Not found (404)'. Attached is the screen shot for this error message. Any idea? Thanks! Best, Bongsoo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] CloudMan auto-scaling - Use spot instance for worker node?
No it is not. The thinking was that autoscaling should be responsive and spot instance are not necessarily predictable so we decided against it. We can revisit that idea though if you're keen? Cheers, Enis On Wed, Oct 15, 2014 at 11:05 AM, Petit III, Robert A. robert.pe...@emory.edu wrote: Hi all, I realize when you manually add a node you can request that it be a spot instance. Is it also possible to use spot instances for auto-scaling? Thank you very much, Robert -- This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] CloudMan with HTCondor and Slurm mounting pathes
Changing the default job manager is not interchangeable in an automated fashion. SGE is currently the main job manager that Galaxy has been configured to be used with. Slurm will replace SGE in the next Galaxy CloudMan release. HTCondor has been added as an experimental module but due to the lack of interest from the community (ie, no interest until now), it hasn't seen any development and I'm really not even sure if it's working as expected any more. Relevant documentation is available here https://wiki.galaxyproject.org/CloudMan/HTCondor so if you can try this out first and see what happens; we can then go from there. Having said that, if you install any of the desired job managers manually, Galaxy can be configured to use that one by setting appropriate configuration option in */mnt/galaxy/galaxy-app/universe_wsgi.ini*. See this page for more details on configuring Galaxy: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster Regarding the use of an S3 bucket as a data source - the intent (largely mandated by the technical characteristics of an object store) is for this to be used as a source of data rather than the default data store for Galaxy. With that, it is advisable to use it as a source location for importing shared datasets and ultimately storing the resulting ones. What this means from within Galaxy is that you could create a data library with the mounted S3 bucket being a data source and then utilize it for the analyses. Take a look at slides #25-28 in this presentation on how to do this: http://www.slideshare.net/afgane/data-analysis-with-galaxy-on-the-cloud If you would still like to use the S3 bucket as a comprehensive data store, this can be done by setting *file_path* in */mnt/galaxy/galaxy-app/universe_wsgi.ini* to the desired path. Hope this clarifies some questions. Let us know what you decide to do and how things develop, Enis On Tue, Oct 14, 2014 at 6:35 AM, Abdularahman Azab a...@ifi.uio.no wrote: Dear all, I've installed a galaxy cloud cluster using usergalaxy.org/cloudlaunch It runs fine, but I need to know how to let the cluster run on THCondor or slurm instead of SGE. I can see that Condor is already installed on the galaxy-master node since the condor_status command is there. But how can I switch the default galaxy job-runner between SGE, HTCondor, and slurm Also regarding the mounting option of a e.g. s3 bucket on the master node. It works fine and it can be accessed from the command line. But how to let galaxy see this mounted path for e.g, reading input files and storing resulted history elements in it? Thank you, Yours sincerely, Abdulrahman Azab Head engineer, ELIXIR.NO / The Genomic HyperBrowser team Department of Informatics, University of Oslo, Boks 1072 Blindern, NO-0316 OSLO, Norway Email: a...@ifi.uio.no http://owa/abdul...@ifi.uio.no, Cell-phone: +47 46797339 Senior Lecturer in Computer Engineering Faculty of Engineering, University of Mansoura, 35516-Mansoura, Egypt Email: abdulrahman.a...@mans.edu.eg http://owa/abdulrahman.a...@mans.edu.eg ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] CloudMan: Autoscaling =Unable to run this job due to a cluster error
Hi Robert, Can you check on CloudMan's Admin page if the following is shown (towards the bottom of the page): Switch master to not run jobs If it says, Switch master to run jobs, just click on the text and the master will become an execution host on the cluster, which should then give Galaxy some resources to run the jobs on. Sorry for the trouble and let us know if this does not fix it, Enis On Wed, Oct 8, 2014 at 6:34 PM, Petit III, Robert A. robert.pe...@emory.edu wrote: Hello Galaxy Team! I've set up a vanilla CloudMan instance using AWS. I've set the head node to not handle jobs and have set AutoScaling to a minimum of 0 and a maximum of 4 worker nodes. Upon submitting a job, it fails with the following error: Unable to run this job due to a cluster error, please retry it later Now if I set AutoScaling to a minimum of 1 worker node it works fine. But I would rather not always have 1 worker node up. Any recommendations? Thank you very much, Robert -- This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] CloudMan: Autoscaling =Unable to run this job due to a cluster error
There needs to be at least one exec host for jobs to actually run, irrespective of whether auto-scaling is on or off. In an attempt to offload work from the master instance when there are workers, CloudMan will automatically toggle the master as an exec host a the first worker is added or the last one removed. However, I'm starting to think there's a bug in how that's implemented. If this causes you more problems, let us know. On Thu, Oct 9, 2014 at 11:35 AM, Petit III, Robert A. robert.pe...@emory.edu wrote: Hi Enis, I have indeed set the master node to not run jobs. Does there has to be at least one execution host (either master or always on worker) auto scaling to work? Thanks for the help, Robert -- *From:* Enis Afgan [afg...@gmail.com] *Sent:* Thursday, October 09, 2014 11:30 AM *To:* Petit III, Robert A. *Cc:* galaxy-dev@lists.bx.psu.edu *Subject:* Re: [galaxy-dev] CloudMan: Autoscaling =Unable to run this job due to a cluster error Hi Robert, Can you check on CloudMan's Admin page if the following is shown (towards the bottom of the page): Switch master to not run jobs If it says, Switch master to run jobs, just click on the text and the master will become an execution host on the cluster, which should then give Galaxy some resources to run the jobs on. Sorry for the trouble and let us know if this does not fix it, Enis On Wed, Oct 8, 2014 at 6:34 PM, Petit III, Robert A. robert.pe...@emory.edu wrote: Hello Galaxy Team! I've set up a vanilla CloudMan instance using AWS. I've set the head node to not handle jobs and have set AutoScaling to a minimum of 0 and a maximum of 4 worker nodes. Upon submitting a job, it fails with the following error: Unable to run this job due to a cluster error, please retry it later Now if I set AutoScaling to a minimum of 1 worker node it works fine. But I would rather not always have 1 worker node up. Any recommendations? Thank you very much, Robert -- This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Cloudman indices installation/configuration
Hi Iry, Yeah, I see what you mean about that *env.sh* file not being in the GATK2 repo the readme states so. I'm not sure what's exactly supposed to be in that file for GATK2 in particular so perhaps one of the wrapper authors can jump in. For majority of tools, you'd just need something like this in there: *PATH=/mnt/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/gatk2/8bcc13094767/gatk2/bin:$PATH http://toolshed.g2.bx.psu.edu/repos/iuc/gatk2/8bcc13094767/gatk2/bin:$PATH* and have placed the binaries for the tool in that directory. If nobody else jumps in, I'll poke around more in the coming days. On Tue, Oct 7, 2014 at 11:58 AM, Iry Witham iry.wit...@jax.org wrote: Hi Enis, Thanks for that information. Now I am getting an error with the Unified_Genotyper failing to locate the GenomeAnalysisTK.jar. I discovered that gatk2 needs to be downloaded and installed. I have done that, but can't seem to figure out where the env.sh file reference below exists. Can you point me to the correct proximity of that file? Or do I need to create the file and if so where? Thanks, Iry Galaxy wrapper for GATK2 This wrapper is copyright 2013 by Björn Grüning, Jim Johnson the Galaxy Team. The Genome Analysis Toolkit or GATK is a software package developed at the Broad Institute to analyse next-generation resequencing data. The toolkit offers a wide variety of tools, with a primary focus on variant discovery and genotyping as well as strong emphasis on data quality assurance. Its robust architecture, powerful processing engine and high-performance computing features make it capable of taking on projects of any size. http://www.broadinstitute.org/gatk http://www.broadinstitute.org/gatk/about/citing-gatk GATK is Free for academics, and fee for commercial use. Please study the GATK licensing website: http://www.broadinstitute.org/gatk/about/#licensing Installation The recommended installation is by means of the toolshed http://toolshed.g2.bx.psu.edu/view/iuc/gatk2. Galaxy should be able to install samtools dependencies automatically for you. GATK2, and its new licence model, does not allow us to distribute the GATK binaries. As a consequence you need to install GATK2 by your own, please see the GATK website for more information: http://www.broadinstitute.org/gatk/download Once you have installed GATK2, you need to edit the env.sh files that are installed together with the wrappers. You must edit the GATK2_PATH environment variable in the file: tool_dependency_dir/environment_settings/GATK2_PATH/iuc/gatk2/hash_string/env.sh to point to the folder where you have installed GATK2. Optionally, you may also want to edit the GATK2_SITE_OPTIONS environment variable in the file: tool_dependency_dir/environment_settings/GATK2_SITE_OPTIONS/iuc/gatk2/hash_string/env.sh to deactivate the 'call home feature' of GATK with something like: GATK2_SITE_OPTIONS='-et NO_ET -K /data/gatk2_key_file' GATK2_SITE_OPTIONS can be also used to insert other specific options into every GATK2 wrapper at runtime, without changing the actual wrapper. Read more about the Phone Home problem at: http://www.broadinstitute.org/gatk/guide/article?id=1250 Optionally, you may also want to add some commands to be executed before GATK (e.g. to load modules) to the file: tool_dependency_dir/gatk2/default/env.sh Finally, you should fill in additional information about your genomes and annotations in the gatk2_picard_index.loc and gatk2_annotations.txt. You can find them in the tool-data/ Galaxy directory. From: Enis Afgan afg...@gmail.com Date: Saturday, October 4, 2014 6:10 AM To: Iry Witham iry.wit...@jax.org Cc: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Cloudman indices installation/configuration Hi Iry, Try adding the following to your */mnt/galaxy/galaxy-app/tool_data_table_conf.xml*, populating the referenced files (tool-data/gatk2_picard_index.loc and tool-data/gatk2_annotations.txt) as desired and restarting Galaxy: !-- Location of Picard dict files valid for GATK -- table name=gatk2_picard_indexes comment_char=# columnsvalue, dbkey, name, path/columns file path=tool-data/gatk2_picard_index.loc / /table !-- Available of GATK references -- table name=gatk2_annotations comment_char=# columnsvalue, name, gatk_value, tools_valid_for/columns file path=tool-data/gatk2_annotations.txt / /table Hope this gets you going. Let us know if it doesn't, Enis On Fri, Oct 3, 2014 at 1:36 PM, Iry Witham iry.wit...@jax.org wrote: It looks like I need to generate the dict file for the mm10 reference as well as add the reference to the srma_index.loc. My question is where do these need to exist? Do they belong in the repo directory structure or or in the primary tool-data directory? The hg19.fa, hg19.fa.fia, hg19.dict as well as these same files
Re: [galaxy-dev] Cloudman indices installation/configuration
Hi Iry, Try adding the following to your */mnt/galaxy/galaxy-app/tool_data_table_conf.xml*, populating the referenced files (tool-data/gatk2_picard_index.loc and tool-data/gatk2_annotations.txt) as desired and restarting Galaxy: !-- Location of Picard dict files valid for GATK -- table name=gatk2_picard_indexes comment_char=# columnsvalue, dbkey, name, path/columns file path=tool-data/gatk2_picard_index.loc / /table !-- Available of GATK references -- table name=gatk2_annotations comment_char=# columnsvalue, name, gatk_value, tools_valid_for/columns file path=tool-data/gatk2_annotations.txt / /table Hope this gets you going. Let us know if it doesn't, Enis On Fri, Oct 3, 2014 at 1:36 PM, Iry Witham iry.wit...@jax.org wrote: It looks like I need to generate the dict file for the mm10 reference as well as add the reference to the srma_index.loc. My question is where do these need to exist? Do they belong in the repo directory structure or or in the primary tool-data directory? The hg19.fa, hg19.fa.fia, hg19.dict as well as these same files for the mm9 GRCh37. However, the .dict does not exist for mm10. Even though that is the case the references do not appear in the gatk2 tools. Any ideas? Thanks, Iry From: Daniel Blankenberg d...@bx.psu.edu Date: Thursday, October 2, 2014 1:57 PM To: Iry Witham iry.wit...@jax.org Cc: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Cloudman indices installation/configuration Hi Iry, First thing to check is that your fields are tab delimited — they appear to be spaces instead of tabs in this email, but copy and pasting into email can munge things sometimes (also “gh19.fa” is probably a typo, but that wouldn’t prevent the selection option from showing up). Thanks for using Galaxy, Dan On Oct 2, 2014, at 1:49 PM, Iry Witham iry.wit...@jax.org wrote: Hi Team, I have a new instance of galaxy cloudman running and have added tools from the toolshed to it. When I attempt to run tools like sam-to-bam or any gatk tool I am prompted for a reference genome. However, indices/references not available for these tools. I have added the following line to the sam_fa_indices.loc, but that did nothing: index hg19/mnt/galaxyIndices/genomes/Hsapiens/hg19/seq/gh19.fa I have also added the following three lines to the gatk2_picard_index.loc: hg19hg19Human (hg19) /mnt/galaxyIndices/genomes/Hsapiens/hg19/seq/hg19.fa GRCh37 GRCh37 Human (GRCh37) /mnt/galaxyIndices/genomes/Hsapiens/GRCh37/seq/GRCh37.fa mm10mm10Mouse (mm10) /mnt/galaxyIndices/genomes/Mmusculus/mm10/seq/mm10.fa I know I have missed something, but can't seem to figure it out. Could someone point me in the right direction? Regards, __ Iry T. Witham Scientific Applications Administrator Computational Sciences Group The Jackson Laboratory 600 Main Street Bar Harbor, ME 04609 Phone: 207-288-6744 email: iry.wit...@jax.org 372D007A-1B00-4668-BA6B-F0527C1F24BE[34][3].png The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Amazon cloud setup for galaxy training
Hello Anne, Overall, what we typically recommend is to launch a new instance and configure it as desired (eg, with tools, histories, and datasets of interest for the workshop). Then share that instance (using the CloudMan share-a-cluster feature http://www.biomedcentral.com/1471-2105/13/315 - small green icon next to the cluster name on the main CloudMan console) and create a number of replicas of the shared instance for use during the workshop. For RNA-Seq workshops, for example, we generally stick with about 10 participants per instance. Here's a page with some empirical-based recommendations regarding choosing instance types: https://wiki.galaxyproject.org/CloudMan/AWS/CapacityPlanning We are also currently working on a templated training image https://trello.com/c/Uc1Gd5Vv/157-cloudmain-train-training-shareable-instance that would come populated with a number of training artifacts. This may be ready for use by Sep 24th. Finally, Dave Clements (CC'd) has a lot of hands on experience with setups for various workshops so he may be able to chime in and provide additional feedback. Hope this helps and let us know if you have any more questions, Enis On Fri, Sep 12, 2014 at 10:15 AM, Anne Pajon anne.pa...@cruk.cam.ac.uk wrote: Dear, I am setting up with my colleague Jing an Introductory Galaxy training course at the University of Cambridge due to on the 24th September. I am looking for help on how to setup the cloud to have a smooth user experience during the workshop. Would you have any links with information to recommend or any advices on how to set it up? Many thanks in advance. Kindest regards, Anne. -- Dr Anne Pajon - Bioinformatics Core Cancer Research UK - Cambridge Institute Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE anne.pa...@cruk.cam.ac.uk | +44 (0)1223 769 631 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy and object stores
Hi Inge, There is an implementation for using the AWS S3 object store as the data store for a given Galaxy instance. The implementation is located here https://bitbucket.org/galaxy/galaxy-central/src/3a51eaf209f2502bf32dbb421ecabb7fe46243ea/lib/galaxy/objectstore/s3.py?at=default and it offers several config options in universe_wsgi.ini. The data stored in S3 is locally cached while it's being operated on but always synced with the back end object store. Pulsar seems to have some support for S3 but, as the docs say in the implementation, it's explicitly beta: https://github.com/galaxyproject/pulsar/blob/b32b7caafc6582a3a28e694e2dbb75e7a8f2bffc/galaxy/objectstore/pulsar.py As a side note, there are some planned enhancements to how the object store implementation is handled and there will hopefully be quite a bit of activity on this topic in the near future (eg, https://trello.com/c/YynQKq8m ). Hope this at least clarifies the state of object store support, Enis On Mon, Aug 25, 2014 at 10:24 AM, Raknes Inge Alexander inge.a.rak...@uit.no wrote: I have a few questions about object stores in Galaxy: 1: Can all Galaxy data sets be stored in an object store? 2: If so, does Galaxy still need to maintain a local copy of the data? 3: Is LWR or Pulsar able to get the data directly from the object store, or does it still have to go through Galaxy? We are planning to let users of our Galaxy installation handle large input/output files (~30G) and we expect that the VM containing our Galaxy installation will become a bottleneck if all data needs to travel through that node. - Inge Alexander Raknes ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] setting SGE arguments in galaxy (cloudman)
Hi Jim Try setting it like this: default_cluster_job_runner = drmaa://-w n -l mem_free=16G/ I've just tried it and worked fine. Sorry for a delay in getting back to you, Enis On Tue, Aug 5, 2014 at 6:44 PM, Jim McCusker jmccus...@5amsolutions.com wrote: I have a galaxy/cloudman cluster where I would like to pass a -l mem_free=16G parameter when galaxy submits the job to SGE. I've tried adding it like this: default_cluster_job_runner = drmaa://-l mem_free=16G/ (it was: default_cluster_job_runner = drmaa:/// ) but now the jobs don't even submit. Am I wiping out other important parameters? How should I do this? Thanks, Jim ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] setting SGE arguments in galaxy (cloudman)
If I'm understanding your question correctly (ie, have the ability to specify SGE requirements on a per-tool basis), then yes. You'd do something like this in universe_wsgi.ini: [galaxy:tool_runners] bwa_wrapper = drmaa://-pe smp 2/ bowtie2 = drmaa://-pe smp 2/ On Thu, Aug 14, 2014 at 2:26 PM, Jim McCusker jmccus...@5amsolutions.com wrote: thanks, I ended up doing subinvocations (submitting a job from within a job) because one tool in particular needs lots of memory. Is there a way to set these things on a per-tool basis (which would let me remove the subinvocation)? Jim On Thu, Aug 14, 2014 at 2:09 PM, Enis Afgan afg...@gmail.com wrote: Hi Jim Try setting it like this: default_cluster_job_runner = drmaa://-w n -l mem_free=16G/ I've just tried it and worked fine. Sorry for a delay in getting back to you, Enis On Tue, Aug 5, 2014 at 6:44 PM, Jim McCusker jmccus...@5amsolutions.com wrote: I have a galaxy/cloudman cluster where I would like to pass a -l mem_free=16G parameter when galaxy submits the job to SGE. I've tried adding it like this: default_cluster_job_runner = drmaa://-l mem_free=16G/ (it was: default_cluster_job_runner = drmaa:/// ) but now the jobs don't even submit. Am I wiping out other important parameters? How should I do this? Thanks, Jim ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] setting SGE arguments in galaxy (cloudman)
The default args will apply to all the tools. Tool-specific args will apply only for the tool you specify and those overwrite the default (ie, arguments will not add up). On Thu, Aug 14, 2014 at 2:53 PM, Jim McCusker jmccus...@5amsolutions.com wrote: Perfect, thanks! That's in addition to the default arguments? On Thu, Aug 14, 2014 at 2:52 PM, Enis Afgan afg...@gmail.com wrote: If I'm understanding your question correctly (ie, have the ability to specify SGE requirements on a per-tool basis), then yes. You'd do something like this in universe_wsgi.ini: [galaxy:tool_runners] bwa_wrapper = drmaa://-pe smp 2/ bowtie2 = drmaa://-pe smp 2/ On Thu, Aug 14, 2014 at 2:26 PM, Jim McCusker jmccus...@5amsolutions.com wrote: thanks, I ended up doing subinvocations (submitting a job from within a job) because one tool in particular needs lots of memory. Is there a way to set these things on a per-tool basis (which would let me remove the subinvocation)? Jim On Thu, Aug 14, 2014 at 2:09 PM, Enis Afgan afg...@gmail.com wrote: Hi Jim Try setting it like this: default_cluster_job_runner = drmaa://-w n -l mem_free=16G/ I've just tried it and worked fine. Sorry for a delay in getting back to you, Enis On Tue, Aug 5, 2014 at 6:44 PM, Jim McCusker jmccus...@5amsolutions.com wrote: I have a galaxy/cloudman cluster where I would like to pass a -l mem_free=16G parameter when galaxy submits the job to SGE. I've tried adding it like this: default_cluster_job_runner = drmaa://-l mem_free=16G/ (it was: default_cluster_job_runner = drmaa:/// ) but now the jobs don't even submit. Am I wiping out other important parameters? How should I do this? Thanks, Jim ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] August 2014 Galaxy CloudMan release
*We just released an update to Galaxy CloudMan.* CloudMan offers an easy way to get a personal and completely functional instance of Galaxy in the cloud in just a few minutes, without any manual configuration. This is mostly an incremental bug fix release with the following summary of changes: - On AWS, updated *galaxyFS* snapshot (snap-e6e1c04a), which includes the June 2, 2014 Galaxy release with the July 30th security fix. All the tools installed via the Tool Shed have been updated and a number of new tools added, most notably: Tophat2, Bowtie2, FastQC, several FASTQ manipulation tools, several QC tools. - For AWS, added support for VPC - For OpenStack clouds, added the ability to automatically recover worker instances on cluster reboot - Added support for creating a file system based on a downloadable archive - Do not run Galaxy with multiple processes by default. This is because Tool Shed installs do not work properly in the multi-process mode. This feature can be enabled by setting user data option configure_multiple_galaxy_processes to True when launching an instance. - Set SGE slots in each queue to be equal to the number of cores on the instance - Set instance IP in the Galaxy's FTP data upload tool message - Added support for Nginx v1.4 and allow it (with the PAM module) to used as the authentication mechanism when accessing Galaxy Reports app - Fixed cluster deletion when performed via the API - No longer automatically start Hadoop and HTCondor services - On manually-invoked instance reboots, do not increment the instance reboot count that otherwise eventually leads to instance termination - Limit the size of the log message buffer used in the UI to 1000 lines. Long-running instances had issues with this log growing large and that led to poor UI performance. The complete log is still available from the Admin page (or the command line). - Automatically delete the bucket/container for Test type (*ie*, 'SGE only') clusters on cluster termination For complete details on implemented changes, please see the source code commits https://bitbucket.org/galaxy/cloudman/commits/all?search=835%3A903 . Enjoy and please let us know what you think, Enis Dannon The Galaxy Team https://wiki.galaxyproject.org/Cloud ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] AWS CloudFormation for Galaxy ?
Someone else might have but we have not (the same argument still holds). Please let us know if you go down that path, Enis On Fri, Apr 25, 2014 at 12:35 PM, Luca Toldo lucato...@gmail.com wrote: dear Galaxians, AWS CloudFormation is now mature, and I have interest in exploiting it for extending my local grid installation with SGE-based execute nodes with galaxy on top. This is something that Enis and Ryan already discussed more than 3 years ago https://www.mail-archive.com/galaxy-dev@lists.bx.psu.edu/msg00427.html however I couldn't find any followup to that discussion. Has anyone already done such script ? ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Help Galaxy
Hello, Galaxy workflow and history structure is stored in Galaxy's database vs. flat files so you can't just find the representation of those on the file system. I actually don't know if there is any documentation about the JSON structure for those artifacts - I've CC'd the -dev mailing list so someone there might know (btw, emailing the -dev list is preferred with questions like these because your question get more exposure). In the mean time, you can explore the API documentation (available herehttp://bioblend.readthedocs.org/en/latest/api_docs/galaxy/all.html#module-bioblend.galaxy.workflowsand herehttps://galaxy-central.readthedocs.org/en/latest/lib/galaxy.webapps.galaxy.api.html#galaxy.webapps.galaxy.api.workflows.WorkflowsAPIController) because that's where JSON objects can be retrieved. Hope this helps, Enis On Thu, Apr 24, 2014 at 3:33 PM, yattara Moussa moussayatta...@yahoo.frwrote: Hello, I'm working at South Paris University (Université Paris-Sud) Computer Science Research Laboratory (LRI), France. I'm working on Galaxy. I need to know about Galaxy syntax for worklfow internal structure description with JSON, and Galaxy workflow history. *if you have any technical document describing the JSON syntax used for the storage of Galaxy workflow and Galaxy history, it will be very nice if you can send it to me. *I installed Galaxy on ubuntu, In which directory can I find workflow histories files ?. Thank you for your considération, I look forward to hearing from you. Moussa YATTARA. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] launching CloudMan 2.0 on AWS
Hi Vipin, It seems there are too many volumes (or too much volume storage) in your AWS account and AWS is preventing any more from being created. You can either delete some volumes if you don't need them or send a request to AWS to increase your volume limit: https://aws.amazon.com/contact-us/ebs_volume_limit_request/ Cheers, Enis On Mon, Oct 14, 2013 at 10:06 PM, Vipin TS vipin...@gmail.com wrote: Hi dev-team, I am trying to launch cloudman on AWS us-east-1a and I am getting the message *All cluster services started; the cluster is ready for use. (2013-10-14 19:45:52)* I am seeing some error message in the log. * 19:05:33 - Master starting* *19:05:35 - Could not find service class matching userData service entry: PSS* *19:05:35 - Completed the initial cluster startup process. This is a new cluster; waiting to configure the type.* *19:06:11 - Migration service prerequisites OK; starting the service* *19:06:11 - SGE service prerequisites OK; starting the service* *19:06:26 - Setting up SGE...* *19:06:40 - HTCondor service prerequisites OK; starting the service* *19:06:48 - Hadoop service prerequisites OK; starting the service* *19:07:04 - Done adding Hadoop service; service running.* *19:08:13 - Initializing 'Galaxy' cluster type. Please wait...* *19:08:14 - Error creating volume: EC2ResponseError: 400 Bad Request VolumeLimitExceededMaximum number of active volumes bytes, 20, exceeded.66659429-b0dd-4f88-b763-36fd9232d3c4* *19:08:17 - Adding volume vol-f9c4728e (FS object for galaxy)...* *19:08:42 - Successfully grew file system FS object for galaxy* *19:08:42 - Successfully mounted file system /mnt/galaxy from /dev/xvdf* It seems that the galaxyIndices filesytem is not mounted to the instance. Also I am not able to start the postgres or Galaxy service through cloudman interface, both says that Cannot find the service. Please find the attached image. I think I am missing the galaxyIndices volume from the image,. any idea about the failure to launch the instance. Thanks, Vipin http://galaxy.cbio.mskcc.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [galaxy-user] Connecting to new Galaxy Cloudman by FTP
FTP issues are fixed now so things will be functional out of the box without any of these workarounds. On Tue, Jul 16, 2013 at 4:04 PM, Daniel Blankenberg d...@bx.psu.edu wrote: Hey Mo, You can use ssh to connect to the Galaxy machine. If you used cloudlaunch to create your instance, it should display an example connection string that will work from e.g. a linux/mac shell, something like: 'ssh -i cloudman_keypair.pem ubuntu@IP', after your instance launched. Once inside of the machine, you can do something like: 'sudo -iu galaxy', to switch to the galaxy user and then have a look at the mount points under /mnt/. The cloudman_keypair.pem file is the key file that you (should have) downloaded the first time that you launched a cloudman instance, or when you manually generated a keypair. You can create additional keypairs in your aws console if you need to download a new one to use (you can probably delete the existing cloudman_keypair and have it regenerated automatically by cloudman, but I haven't tested this and I wouldn't recommend doing this if an instance is running). You'll need to use the correct .pem file for the keypair that you specified during launch of the instance. See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/generating-a-keypair.htmlfor Amazon's info on creating keypairs (especially if you are using e.g. Windows and putty: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html). Once you make the changes to the files locally, you can use the Cloudman Admin web UI to restart the Galaxy instance. Let us know if you encounter any issues. Thanks for using Galaxy, Dan On Jul 15, 2013, at 7:22 PM, Mohammad Heydarian wrote: Hey Nate, Thanks for the response and instructions. I understand the last three steps of your protocol, but the first step is difficult for me to understand. I'm guessing that, 1. Set 'use_pbkdf2 = False' in universe_wsgi.ini anywhere in the [app:main] section, is telling me to change a setting of Cloudman by command line. This is generally where we get stuck in using Cloudman, because we aren't familiar with command line (we get cold sweats and light palpitations) and are weary of making changes to the Cloudman code. I would ask our IT guys for help, but their expertise ends at updating Office tools. I would bother a programmer or bioinformatician, but most of them are so busy you need a formal collaboration to get on their radar. I would ask people who vaguely know command line for help, but most of the time their knowledge of command line is just higher than mine and we end up troubleshooting an issue neither of us can really grasp. So, is there a webcast, or video, or slideshow, that can show a newbie how to command line into Cloudman and make the changes you outline in step 1? Cheers, Mo Heydarian On Mon, Jul 15, 2013 at 4:22 PM, Nate Coraor n...@bx.psu.edu wrote: On Jul 8, 2013, at 3:50 PM, Mohammad Heydarian wrote: Hi, I am having trouble setting up a FTP connection with the recently released version of Galaxy Cloudman (ami-118bfc78). I have instantiated the new version of Galaxy Cloudman with CloudLaunch and also through the AWS EC2 wizard (using the same security group settings as the previous versions) and neither instance will connect to my FTP connection. Has anyone else had this problem? Does anyone know what is preventing the FTP connection? Any help would be greatly appreciated. Hey Mo, This may be a case of the new password hashing algorithm's incompatibility with the provided ProFTPD config. Could you try the following: 1. Set 'use_pbkdf2 = False' in universe_wsgi.ini anywhere in the [app:main] section 2. Restart Galaxy 3. Reset your password in the Galaxy UI 4. Test FTP again --nate Cheers, Mo Heydarian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ The Galaxy User list should be
Re: [galaxy-dev] Galaxy CloudMan release - Tophat does not work
Not quite next week, but an update to the tools volume and CloudMan itself was just released so things should be working now. Sorry for the trouble and the delay and let us know if how things are going for you now. Cheers, Enis On Sat, Jul 13, 2013 at 9:27 AM, Enis Afgan afg...@gmail.com wrote: Next week, we'll update the underlying snapshot to correct those symlinks and get Tophat working. Sorry for the trouble. On Fri, Jul 12, 2013 at 9:23 PM, Ravpreet Setia ravpreet.se...@oicr.on.ca wrote: I see you are experiencing the same problem as I am. The reason these programs cannot be found is because the default symlink is broken. The following page describes how the symlink is used by Galaxy: http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies Suppose you wish to fix Tophat, then you will need to go to /mnt/galaxy/tools/tophat. Type 'ls –l' and you will notice that the 'default' symlink points to a directory in /mnt/galaxyTools. Recall that this update merged galaxyTools and galaxyData into a single volume – but the symlink was not modified to account for this. Therefore, the symlink should instead point to /mnt/galaxy/tools/tophat/VERSION. Then, go into the directory of the version you specified and modify the env.sh file so it also has the proper path. Let me know if Tophat works for you, even after doing this it fails to run properly for me. However, at least now Galaxy can find the programs. From: Mohammad Heydarian mheyd...@jhmi.edu Date: Fri, 12 Jul 2013 14:21:23 -0400 To: Microsoft Office User ravpreet.se...@oicr.on.ca Cc: Enis Afgan afg...@gmail.com, Galaxy-dev galaxy-...@bx.psu.edu Subject: Re: [galaxy-dev] Galaxy CloudMan release - Tophat does not work Cufflinks, Cuffcompare, and Cuffdiff are also failing to run because they can not be found. I am using the newly released version of Cloudman 2.0 (ami-118bfc78). Any help would be appreciated. Cheers, Mo Heydarian On Mon, Jul 8, 2013 at 5:02 PM, Ravpreet Setia ravpreet.se...@oicr.on.ca wrote: Some tools (tophat is one) are failing to run because they cannot be found. I have noticed that after launching a new instance with the latest AMI, the package/version/default sym-links under /mnt/galaxy/tools point to a directory in /mnt/galaxyTools, which does not exist since this new AMI merged galaxyTools and galaxyData. Additionally, even after manually fixing the symlink and the path in the env.sh file (it also references galaxyTools), running tophat would fail, although now, at least, it can find the program. This document explains how the default symlink is used by Galaxy: http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies Thanks for any suggestions. From: Enis Afgan afg...@gmail.com Date: Sun, 30 Jun 2013 07:37:46 +0200 To: Galaxy-dev galaxy-...@bx.psu.edu Subject: [galaxy-dev] Galaxy CloudMan release *Last night, we released an update to Galaxy CloudMan.* CloudMan offers an easy way to get a personal and completely functional instance of Galaxy in the cloud in just a few minutes, without any manual configuration. *IMPORTANT - please read* Any new cluster will automatically start using this version of CloudMan. Existing clusters will be given an option to do an automatic update once the main interface page is refreshed. Note that this upgrade is a major version upgrade and thus the migration is rather complicated. The migration process has been automated but will take a little while to complete. If you have made customizations to your cluster in terms of adding file systems, upgrading the database, or similar, we do not recommend you perform the upgrade. Note that this upgrade comes with (and requires) a new AMI (ami-118bfc78), which will be automatically used if starting an instance via CloudLaunch http://usegalaxy.org/cloudlaunch. *This update brings a large number of updates and new features, the most prominent ones being:* - Unification of *galaxyTools* and *galaxyData* file systems into a single *galaxy* filesystem. This change makes it possible to utilize the Galaxy Tool Shed when installing tools into Galaxy. - Added initial support for Hadoop-type workloads - Added initial support for cluster federation via HTCondor - Added a new file system service for instance's transient storage, allowing it to be used across the cluster over NFS - Added a service for Galaxy Reports webapp - Added optional Loggly (loggly.com) based off-site logging support - Added tags to all resources utilized by CloudMan For more details on the new features, see the the CHANGELOGhttps://bitbucket.org/galaxy/cloudman/src/tip/CHANGELOG.md?at=defaultand for even more details see, all of 291 commit messageshttps://bitbucket.org/galaxy/cloudman/commits/all?search=35baec1%3A8bbae3f from 7 contributors. Enjoy and please let us know what you think, Enis P.S. We also now have a logo for CloudMan [image: Inline image
Re: [galaxy-dev] Cloudman database issue (?)
Hi Mo, Just like in the email I just sent, the new release should fix this. Let us know if you have any more issues. On Tue, Jul 16, 2013 at 6:29 PM, Dannon Baker dannon.ba...@gmail.comwrote: Hey Mo, The new volume we pushed out for the conference has several known issues. Enis and I are both away from the office on travel right now, but updating the volume and fixing these issues is the first thing I'll be doing when I'm back next week. For now, you can launch using the pre-conference AMI and cloudman bucket using something like https://main.g2.bx.psu.edu/cloudlaunch?ami=ami-da58aab3bucket_default=gxy-workshop -Dannon On Tue, Jul 16, 2013 at 12:24 PM, Mohammad Heydarian mheyd...@jhmi.eduwrote: Hello, I found a couple of issues in using the latest version of Cloudman (revision:10201:ebe87051fadf). The Extract Genomic DNA tool returns an error: No sequences are available for 'mm9', request them by reporting this error. Upon trying to report the error in Galaxy (on the page that comes up when you click the bug icon) I get the error: Mail is not configured for this galaxy instance Any help on fixing the Extract Genomic DNA tool would be great. Thanks. Cheers, Mo Heydarian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy CloudMan release - Tophat does not work
Next week, we'll update the underlying snapshot to correct those symlinks and get Tophat working. Sorry for the trouble. On Fri, Jul 12, 2013 at 9:23 PM, Ravpreet Setia ravpreet.se...@oicr.on.cawrote: I see you are experiencing the same problem as I am. The reason these programs cannot be found is because the default symlink is broken. The following page describes how the symlink is used by Galaxy: http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies Suppose you wish to fix Tophat, then you will need to go to /mnt/galaxy/tools/tophat. Type 'ls –l' and you will notice that the 'default' symlink points to a directory in /mnt/galaxyTools. Recall that this update merged galaxyTools and galaxyData into a single volume – but the symlink was not modified to account for this. Therefore, the symlink should instead point to /mnt/galaxy/tools/tophat/VERSION. Then, go into the directory of the version you specified and modify the env.sh file so it also has the proper path. Let me know if Tophat works for you, even after doing this it fails to run properly for me. However, at least now Galaxy can find the programs. From: Mohammad Heydarian mheyd...@jhmi.edu Date: Fri, 12 Jul 2013 14:21:23 -0400 To: Microsoft Office User ravpreet.se...@oicr.on.ca Cc: Enis Afgan afg...@gmail.com, Galaxy-dev galaxy-...@bx.psu.edu Subject: Re: [galaxy-dev] Galaxy CloudMan release - Tophat does not work Cufflinks, Cuffcompare, and Cuffdiff are also failing to run because they can not be found. I am using the newly released version of Cloudman 2.0 (ami-118bfc78). Any help would be appreciated. Cheers, Mo Heydarian On Mon, Jul 8, 2013 at 5:02 PM, Ravpreet Setia ravpreet.se...@oicr.on.ca wrote: Some tools (tophat is one) are failing to run because they cannot be found. I have noticed that after launching a new instance with the latest AMI, the package/version/default sym-links under /mnt/galaxy/tools point to a directory in /mnt/galaxyTools, which does not exist since this new AMI merged galaxyTools and galaxyData. Additionally, even after manually fixing the symlink and the path in the env.sh file (it also references galaxyTools), running tophat would fail, although now, at least, it can find the program. This document explains how the default symlink is used by Galaxy: http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies Thanks for any suggestions. From: Enis Afgan afg...@gmail.com Date: Sun, 30 Jun 2013 07:37:46 +0200 To: Galaxy-dev galaxy-...@bx.psu.edu Subject: [galaxy-dev] Galaxy CloudMan release *Last night, we released an update to Galaxy CloudMan.* CloudMan offers an easy way to get a personal and completely functional instance of Galaxy in the cloud in just a few minutes, without any manual configuration. *IMPORTANT - please read* Any new cluster will automatically start using this version of CloudMan. Existing clusters will be given an option to do an automatic update once the main interface page is refreshed. Note that this upgrade is a major version upgrade and thus the migration is rather complicated. The migration process has been automated but will take a little while to complete. If you have made customizations to your cluster in terms of adding file systems, upgrading the database, or similar, we do not recommend you perform the upgrade. Note that this upgrade comes with (and requires) a new AMI (ami-118bfc78), which will be automatically used if starting an instance via CloudLaunch http://usegalaxy.org/cloudlaunch. *This update brings a large number of updates and new features, the most prominent ones being:* - Unification of *galaxyTools* and *galaxyData* file systems into a single *galaxy* filesystem. This change makes it possible to utilize the Galaxy Tool Shed when installing tools into Galaxy. - Added initial support for Hadoop-type workloads - Added initial support for cluster federation via HTCondor - Added a new file system service for instance's transient storage, allowing it to be used across the cluster over NFS - Added a service for Galaxy Reports webapp - Added optional Loggly (loggly.com) based off-site logging support - Added tags to all resources utilized by CloudMan For more details on the new features, see the the CHANGELOGhttps://bitbucket.org/galaxy/cloudman/src/tip/CHANGELOG.md?at=defaultand for even more details see, all of 291 commit messageshttps://bitbucket.org/galaxy/cloudman/commits/all?search=35baec1%3A8bbae3f from 7 contributors. Enjoy and please let us know what you think, Enis P.S. We also now have a logo for CloudMan [image: Inline image 2] ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:http://lists.bx.psu.edu/ To search Galaxy mailing lists use
Re: [galaxy-dev] Cloudman - Software does not work due to broken symlinks
Is this a migrated cluster or a brand new one? I'm guessing you're using the new AMI? The migration process supposed to have updated those env.sh paths. More to the point, is your tophat job failing due to bowtie not being found? Try adding the path to bowtie executables to tophat's env.sh file as well and see if that fixes the issue. On Fri, Jul 12, 2013 at 5:26 PM, Ravpreet Setia ravpreet.se...@oicr.on.cawrote: This may be a problem with the latest Galaxy Cloudman AMI. Some tools (tophat is one) are failing to run because they cannot be found. I have noticed that after launching a new instance with the latest AMI, the package/version/default sym-links under /mnt/galaxy/tools point to a directory in /mnt/galaxyTools, which does not exist since this new AMI merged galaxyTools and galaxyData. Additionally, even after manually fixing the symlink and the path in the env.sh file (it also references galaxyTools), running tophat would fail, although now, at least, it can find the program. This may be an issue with tophat's XML or python script though, since I fixed Velvet's symlink and env.sh and it worked perfectly. This document explains how the default symlink and env.sh are used by Galaxy: http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies Thanks for any suggestions. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] CloudMan Error
Hi Greg, Sorry for replying really late. So, I'm guessing this was an old cluster that was shared and is now being derived on a new cluster? There was a large number of paths we explored while getting ready for the upgrade and I was of the opinion we covered that path but it seems things are not working as expected. Can look at the more detailed log on the Admin page (under CloudMan log) and see if there are more details about what's going on and why it's failing? On Thu, Jul 11, 2013 at 3:14 PM, greg margeem...@gmail.com wrote: Hi guys, I just thought I'd check in again. None of the researches that want to run out genotyping program can do so until I figure this out. Any help or advice at all would be greatly appreciated. Thanks, Greg On Mon, Jul 8, 2013 at 8:43 AM, greg margeem...@gmail.com wrote: Hi guys, Any thoughts on this? I'm kind of stuck. (Even some pointers on where to look for more clues would be extremely helpful.) Thanks, Greg On Fri, Jul 5, 2013 at 11:10 AM, greg margeem...@gmail.com wrote: Hi guys, I'm hitting an error using CloudMan using the Share-an-Instance option. It says: Error creating volume from shared cluster's snapshot '['snap-cfa775ba']': 'filesystems'. Also disk stats says 0 /0 and the Applications light is yellow while the data light is green. I'm using the share string cm-808d863548acae7c2328c39a90f52e29/shared/2012-09-17--19-47 It's always worked in the past. Thanks, Greg Here's the full log: 14:58:18 - Master starting 14:58:20 - Completed the initial cluster startup process. This is a new cluster; waiting to configure the type. 14:58:24 - Migration service prerequisites OK; starting the service 14:58:24 - SGE service prerequisites OK; starting the service 14:58:31 - Setting up SGE... 14:58:51 - HTCondor service prerequisites OK; starting the service 14:58:51 - HTCondor config file /etc/condor/condor_config not found! 14:58:59 - Hadoop service prerequisites OK; starting the service 14:59:48 - Done adding Hadoop service; service running. 15:01:45 - Error creating volume from shared cluster's snapshot '['snap-cfa775ba']': 'filesystems' ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Galaxy CloudMan release
*Last night, we released an update to Galaxy CloudMan.* CloudMan offers an easy way to get a personal and completely functional instance of Galaxy in the cloud in just a few minutes, without any manual configuration. *IMPORTANT - please read* Any new cluster will automatically start using this version of CloudMan. Existing clusters will be given an option to do an automatic update once the main interface page is refreshed. Note that this upgrade is a major version upgrade and thus the migration is rather complicated. The migration process has been automated but will take a little while to complete. If you have made customizations to your cluster in terms of adding file systems, upgrading the database, or similar, we do not recommend you perform the upgrade. Note that this upgrade comes with (and requires) a new AMI (ami-118bfc78), which will be automatically used if starting an instance via CloudLaunch http://usegalaxy.org/cloudlaunch. *This update brings a large number of updates and new features, the most prominent ones being:* - Unification of *galaxyTools* and *galaxyData* file systems into a single * galaxy* filesystem. This change makes it possible to utilize the Galaxy Tool Shed when installing tools into Galaxy. - Added initial support for Hadoop-type workloads - Added initial support for cluster federation via HTCondor - Added a new file system service for instance's transient storage, allowing it to be used across the cluster over NFS - Added a service for Galaxy Reports webapp - Added optional Loggly (loggly.com) based off-site logging support - Added tags to all resources utilized by CloudMan For more details on the new features, see the the CHANGELOGhttps://bitbucket.org/galaxy/cloudman/src/tip/CHANGELOG.md?at=defaultand for even more details see, all of 291 commit messageshttps://bitbucket.org/galaxy/cloudman/commits/all?search=35baec1%3A8bbae3f from 7 contributors. Enjoy and please let us know what you think, Enis P.S. We also now have a logo for CloudMan [image: Inline image 2] cloudman_logo_black_long.png___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Cloudman - Can you label volumes after user specifies the volume size they want?
Yes, this is good suggestion. The good news is that the functionality is already implemented in the soon-to-be-released version of CloudMan. On Fri, Jun 14, 2013 at 6:52 PM, Ravpreet Setia ravpreet.se...@oicr.on.cawrote: Initially only one volume is shown on Amazon's EC2 management console. Then, after the user goes into the instance's URL and logs into Cloudman, they are prompted with a box called Initial cluster configuration where they specify the size of the volume they want. THEN, three nameless volumes are produced. The problem here is that the volumes are nameless, and this makes cleanup on Amazon's EC2 console difficult since we have no way of knowing these volumes are related to the Galaxy cloudman AMI. I am wondering if there is a way to pass a script to Cloudman that it will run once the volumes are created (ie. After the user goes to the URL and specifies the volume size they want). The script will be used to add name= tags for each of the three additional volumes. Thanks. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] bowtie2 on galaxy cloudman?
Hi Teresa, It's not available on the curent release but we are finalizing a new release that will make addition of tools very straightforward so, if not included by default, it will be easy to do. Thanks for using Galaxy. On Sun, Jun 16, 2013 at 4:25 PM, Murphy, Theresa tmur...@pathology.wustl.edu wrote: Is bowtie2 available on galaxy cloudman? If not, when will it be available? Thanks, Theresa Murphy The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tools and Data
Correct; certain NSG tools require additional configuration for their indices (http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup) On Tue, Mar 5, 2013 at 1:07 PM, shenwiyn shenw...@gmail.com wrote: ** Hi, I've installed Galaxy in local as what Rafael does.I want to make sure that the basic functions work successfully just depending on the necessary tools in Tool dependencieshttp://wiki.galaxyproject.org/Admin/Tools/Tool%20Dependencies manual,and the NGS tools not only depend on the necessary tools but also some another needed data(such as indexes,and so on). thakks -- shenwy ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] GALAXY LOCAL INSTALLATION: Tools and Data
Hi Rafa, The .loc files in 'tool-data' directory link the index files with the actual paths so you need to configure each of the .loc files to match where the index files are placed. With that, you can put the index files anywhere you'd like on the file system (preferably outside Galaxy's home directory). As a reference for a working setup, you can rsync (portion) of the data from GalaxyMain (http://wiki.galaxyproject.org/Admin/Data%20Integration) and see how that's setup. Also, you can fire up a cloud instance and see how it's setup. Hope this helps, Enis On Thu, Feb 21, 2013 at 12:14 AM, Rafael Hernández rafah...@gmail.comwrote: Hi, I've installed Galaxy in local and following the Tool dependencieshttp://wiki.galaxyproject.org/Admin/Tools/Tool%20Dependencies manual I installed all necessary tools. However for some of then such as BWA, Bowtie,... it's usually needed to create indexes. Where should I store those indexes? In the galaxy directory or in each tool directory? How can I tell galaxy where the indexes are? I find a manual's page talking about it / http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup), suggesting to organize the various index files for each build, but I don't know if these folders should be stored into Galaxy directory (eg. in tool-data directory) or what. Could you please explain this a little? Thank you and regards, Rafa Hernández de Diego Genomics of Gene Expression Lab. Bioinformatics and Genomics Department Centro de Investigación Príncipe Felipe (CIPF) C/ Eduardo Primo Yúfera 3 46012 Valencia, Spain ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Questions regarding updating the Galaxy on Amazon AWS
Hi Chun-Yuan, Sorry to see you're running running into so much trouble. The reality of the situation is that the Update Galaxy button in CloudMan is currently broken due to the changes in Galaxy that make the updates a rather manual process. For the past several weeks, the team has been working on sorting this out and we are getting closer to having that resolved. The upcoming upgrade will include an update to Galaxy itself, a number of tools, and the machine image. Also, a change in the architecture of the cloud deployment will be necessary to accomplish this. Specifically, galaxyTools and galaxyData volumes (ie, file systems) will be merged into a single file system. As far as mi-deployment goes - it has been deprecated at this point in favor of CloudBioLinux (cloudbiolinux.org). Over the past 6 months or so, the functionality from mi-deployment has been merged with CBL and is the preferred way of building images and tools. As far as getting a volume attached to a specific instance, the AWS console allows you to attach a volume to an instance at a given device. Then, the device becomes available on the attached instance. Note that the device ID may differ from the one you used to attach the device as - this is noted in the AWS console at the time of attaching a volume so you should look for a device with that ID. As far as support goes - unfortunately, we do not have resources to offer phone support. Instead, you should subscribe and send emails to galaxy-dev mailing list (http://lists.bx.psu.edu/listinfo/galaxy-dev). I have CC'd that list now and posting to that list in the future should give the biggest exposure to the questions you may have. Hope this helps. Let us know if you have any more questions, Enis On Tue, Feb 12, 2013 at 9:04 AM, Chun-Yuan Huang hua...@uwm.edu wrote: Dear Dr. Afgan, I am in a lab that focuses on NGS analysis for Zebrafish Bis-seq dataset, and is about to use the Bismark tool (http://www.bioinformatics.** babraham.ac.uk/projects/**bismarkhttp://www.bioinformatics.babraham.ac.uk/projects/bismark) for that purpose. We were very excited to learn that Bismark works in the Galaxy project that you are the main author of, and have been trying to install Bismark into Galaxy for the past few weeks. We have tried and received the following: 1. Start up our own instance on Amazon AWS using ami-da58aab3, galaxy-cloudman-2011-03-22, and following the instructions from Galaxy Wiki to install Bismark manually, including the necessary modifications on the files universe_wsgi.ini, tool_conf.xml, tool_data_table_conf.xml. However, the installed Bismark kept generating error messages that we could not fully resolve. 2. We later noticed that Bismark can be installed from Tool Shed in a rather automatic way. But in order to take advantage of that, we need to update the galaxy system into more recent version in order to have the recent/automatical Tool Shed. Oddly the link in the Cloudman Admin page for Update Galaxy from a provided repository doesn't work for us. So we tried to update the galaxy manually, then the Bismark from Tool Shed. It worked partially, as some tools are functional but not others, including the Bismark. We are in the process of configuring individual tools as well as Bismark. 3. During the frustration, I kept wondering whether there is a more systemic and better documented way of upgrading Galaxy and installing its tools such as Bismark. Although we may be able to figure out all the problems eventually, the time and work spent on it is rather expensive as compared to a stand-alone Linux box. 4. So in looking out for a more systemic and better documented solution, I just came across your site for the mi-deployment ( https://bitbucket.org/afgane/**mi-deploymenthttps://bitbucket.org/afgane/mi-deployment). By reading its overview, it seems this is exactly the solution I have been looking for (sorry for my ignorance on this matter, as I should have tried it from the beginning). I am in the process of trying mi-deployment. But in the meantime, I would like to ask a couple of questions: 1. Is mi-deployment the right track I should be following, or I am still in the wrong place for my situation? Are there more detailed instructions on doing it? Do you have any other suggestions? I consider myself and my group with fair literacy on bioinformatic tools, NGS tools, and how Linux system works (but not experts on it). 2. Per your instruction on mi-deployment, we have established a CloudBioLinux Ubuntu 12.04 instance (version dated to December 2012) and have set environment variables for both access key and secret access key, in addition to using pip to install boto and fabric. We've attempted to create an EBS volume and link it to our EC2 instance, but it does not show up within the /dev directory following a kernel restart. How may we get it to display such that we can then mount it with sudo mount /dev/VOLUME
Re: [galaxy-dev] [galaxy-user] Why SGE needed for galaxy ?
Here's a link to the architecture paper: http://onlinelibrary.wiley.com/doi/10.1002/cpe.1836/full Building CloudMan image from the repo you mention will not work - that repo is for cloudman itself while you need an image capable of running cloudman. For that, you should use CBL. Also, here are some instructions about setting up cloudman and galaxy on OpenNebula cloud: https://www.cloud.sara.nl/projects/mattiasdehollander-project/wiki (note that this mentions use of mi-deployment set of scripts; since that, mi-deployment has been merged into CBL). On Mon, Feb 11, 2013 at 8:21 PM, Zeeshan Ali Shah zas...@pdc.kth.se wrote: Hi, Thanks for answers , Actually i tried to understand cloudman role in galaxy , do you have an architecture paper which i should read ? for e.g. what i unserstood si that cloud man runs a python server and manage SGE through it via some python script . (may be i am wrong) Our proposed installation is like this: Users launch cloud man from Open nebula cloud, when cloudman is running they can add more nodes which are endup in same private cloud. As you suggested in this particular case I should have same images both for (master cloudman) and workers , am i right ? actually i built one image via https://bitbucket.org/galaxy/cloudman . DO you suggest that I should built image from CBL tree you mentioned below ? which wd have both cloudman and galaxy together. BR Zeeshan On W6-Feb 8, 2013, at 10:21 PM, Enis Afgan wrote: As far as actually building the image, the recommended method is to use CloudBioLinux build scripts: https://github.com/chapmanb/cloudbiolinux There is a CloudMan flavor of CBL that allows you to build only CloudMan- and Galaxy-required parts: https://github.com/chapmanb/cloudbiolinux/tree/master/contrib/flavor/cloudman On Sat, Feb 9, 2013 at 12:24 AM, Dannon Baker dannonba...@me.com wrote: The workers don't need their own copy of galaxy installed, but a shared filesystem is a requirement for galaxy (in any cluster environment -- see the galaxy wiki for more http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster). Cloudman handles managing NFS for you and sharing the galaxy/tools/index/data volumes. In order for workers to communicate with the master instance, they'll need the cloudman installation as well, so you should use the same image. Now that I've answered that, I'm not sure I totally understand your proposed installation yet, but if you're suggesting bypassing cloudman for installation on a private cloud it should be possible. You'd want the master instance up full time running as the galaxy front end, dispatching jobs to a separate cluster managed by SGE/PBS/whatever. Basically the standard cluster configuration outlined in the wiki above, but you'd want your worker nodes automatically configured to mount the shared directories and join the PBS/SGE queue so they could handle jobs. Depending on what type of private cloud you're working with, it might be easier to just see if you can get cloudman to work :) Lastly, I swapped this message to galaxy-dev since it's about installation nuts and bolts. -Dannon On Feb 8, 2013, at 3:02 AM, Zeeshan Ali Shah zas...@pdc.kth.se wrote: Dear Enis, thanks for reply and being you as cloudman developer it is good to see you in the list . Q2: On Workers node we need galaxy installed with its shared directories ? like galaxyindices , galaxydata Q3: For a private cloud setup do you prefare to have a master image with cloudman and galaxy and use the same image for workers as well ? or worker images can be vanilla OS ? BR Zeeshan On W6-Feb 7, 2013, at 11:50 PM, Enis Afgan wrote: Hi Zeeshan, In order to gain from the scalability of the cloud, SGE does need to run. However, CloudMan sets all that up and manages it going forward. Enis On Fri, Feb 8, 2013 at 8:59 AM, Zeeshan Ali Shah zas...@pdc.kth.se wrote: Hi, It seems that cloud man need SGE for scaling . Does SGE need also when run cloud on private cloud ? Zeeshan ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy
Re: [galaxy-dev] [galaxy-user] Why SGE needed for galaxy ?
As far as actually building the image, the recommended method is to use CloudBioLinux build scripts: https://github.com/chapmanb/cloudbiolinux There is a CloudMan flavor of CBL that allows you to build only CloudMan- and Galaxy-required parts: https://github.com/chapmanb/cloudbiolinux/tree/master/contrib/flavor/cloudman On Sat, Feb 9, 2013 at 12:24 AM, Dannon Baker dannonba...@me.com wrote: The workers don't need their own copy of galaxy installed, but a shared filesystem is a requirement for galaxy (in any cluster environment -- see the galaxy wiki for more http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster). Cloudman handles managing NFS for you and sharing the galaxy/tools/index/data volumes. In order for workers to communicate with the master instance, they'll need the cloudman installation as well, so you should use the same image. Now that I've answered that, I'm not sure I totally understand your proposed installation yet, but if you're suggesting bypassing cloudman for installation on a private cloud it should be possible. You'd want the master instance up full time running as the galaxy front end, dispatching jobs to a separate cluster managed by SGE/PBS/whatever. Basically the standard cluster configuration outlined in the wiki above, but you'd want your worker nodes automatically configured to mount the shared directories and join the PBS/SGE queue so they could handle jobs. Depending on what type of private cloud you're working with, it might be easier to just see if you can get cloudman to work :) Lastly, I swapped this message to galaxy-dev since it's about installation nuts and bolts. -Dannon On Feb 8, 2013, at 3:02 AM, Zeeshan Ali Shah zas...@pdc.kth.se wrote: Dear Enis, thanks for reply and being you as cloudman developer it is good to see you in the list . Q2: On Workers node we need galaxy installed with its shared directories ? like galaxyindices , galaxydata Q3: For a private cloud setup do you prefare to have a master image with cloudman and galaxy and use the same image for workers as well ? or worker images can be vanilla OS ? BR Zeeshan On W6-Feb 7, 2013, at 11:50 PM, Enis Afgan wrote: Hi Zeeshan, In order to gain from the scalability of the cloud, SGE does need to run. However, CloudMan sets all that up and manages it going forward. Enis On Fri, Feb 8, 2013 at 8:59 AM, Zeeshan Ali Shah zas...@pdc.kth.se wrote: Hi, It seems that cloud man need SGE for scaling . Does SGE need also when run cloud on private cloud ? Zeeshan ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] galaxy api documentation
We also have a python library for dealing with the API, the documentation is available at http://bioblend.readthedocs.org/ The library source code also contains an example very similar to what you're trying to achieve: imports a workflow, imports data, and runs the workflow. Here's that script: https://github.com/afgane/bioblend/blob/master/docs/examples/run_imported_workflow.py On Mon, Feb 4, 2013 at 2:38 AM, Dannon Baker dannonba...@me.com wrote: Sure, automated data import and workflow execution is certainly possible. In addition to the documentation there are several example scripts in the /scripts/api directory of your galaxy install. In particular, for your use case, you may want to have a look at example_watch_folder.py. This script watches a specified folder for any new files, uploads them to a galaxy data library, and then triggers the execution of a predefined workflow on the data. I wouldn't implement your pipeline exactly like in that script, but it's designed to illustrate a few specific examples without being overly complicated. -Dannon On Feb 3, 2013, at 10:22 AM, mark.r...@syngenta.com wrote: Hey Dannon Thanks, the web page I was finding earlier was mostly just an outline. This is much better. At this point I'm just interested in learning what general capabilities the API can provide. I'm sure after I read this and think more about it I'll have questions. One thing I am thinking about is what role the API might play in automatically analyzing data coming off an Illumina MiSeq. This would seem to be attractive in that it might proceed in a timely way, offer the possibility of a greater variety of analyses than come onboard the MiSeq, and place the results in a history that might readily be shared with multiple users. Does this seem possible? Do you know of anyone doing this now? Thanks again for your help Mark -Original Message- From: Dannon Baker [mailto:dannonba...@me.com] Sent: Saturday, February 02, 2013 10:57 PM To: Rose Mark USRE Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] galaxy api documentation Mark, There's documentation currently available here: https://galaxy-dist.readthedocs.org/en/latest/lib/galaxy.webapps.galaxy.api.html Is there anything else in particular you're looking for? -Dannon On Feb 2, 2013, at 9:15 PM, mark.r...@syngenta.com wrote: Hi All When is it anticipated that the documentation for the API will be available? Thanks Mark This message may contain confidential information. If you are not the designated recipient, please notify the sender immediately, and delete the original and any copies. Any use of the message by you is prohibited. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ This message may contain confidential information. If you are not the designated recipient, please notify the sender immediately, and delete the original and any copies. Any use of the message by you is prohibited. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] RFE: Adding tags to AWS resources created by Cloudman
Hi Shea, This is a reasonable feature so we'll add it on the todo list. To make the search for all untagged resources easier, did you have a list of the ones that are currently not getting tagged? Thanks, Enis On Tue, Dec 4, 2012 at 7:34 AM, Shea Lovan shea.lo...@lscg.ucsb.edu wrote: We are managing Galaxy/Cloudman clusters on AWS for a few research groups on campus. As is so often the case, we need to separate usage for billing purposes. This this is fairly simple to do with Amazon's CSV billing report with the clusterName tag included. However, it doesn't appear that I can configure Cloudman to set that tag on every resource it creates. This is forcing us to keep the scaling of the clusters under our control so that we can set the tags manually. What I was hoping, is that this feature could be added. It seems like a fairly simple patch; once the ID (instance, volume, or snapshot) is available, it's one more API call to create/set the clusterName tag. Any chance of getting this included? Thanks! -Shea -- Shea A. Lovan Manager, Server Operations and Architecture UCSB Life Sciences Computing Group Santa Barbara, CA 93106 805.893.2405 __**_ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Importing files into Libraries
Hi Niel, I'm assuming the data uploaded to a library on a galaxy cloudman instance. In that case, all the data that's uploaded to a data library (or a history) goes on an nfs-exported file system and is thus available to all the worker nodes. Cheers, Enis On Wed, Oct 24, 2012 at 3:51 PM, neil.burd...@csiro.au wrote: Hi, If we upload data and then import the data into libraries, how is that data then accessed when running a tool in the cloud. For example if we have hundreds of template files that we need to analyse the data, and we have a tool that is instantiated on a number of nodes in the clous, is the data in the libraries copied to each node? Or is the data access via nfs, symbolic link or some other method? Thanks Neil ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy in cloud (not Amazon)
Yes there is, the setup is just not documented very well at this point... Also, are you familiar with the cloud concepts or just looking to use Galaxy? The basic approach is to setup a CloudMan (usecloudman.org) machine image on the local cloud (using the image building process from https://github.com/chapmanb/cloudbiolinux is the recommended method). Install Galaxy as well as all of its tools, dependencies, and reference data on the appropriate block storage volumes and turn those into snapshots (see this paper for the architecture overview http://onlinelibrary.wiley.com/doi/10.1002/cpe.1836/full and then Galaxy's wiki (http://wiki.g2.bx.psu.edu/Admin) for the details on how to set everything up). Beyond that, it's a matter of making sure it all works as desired on your setup. You'll probably also want to use a version of the code similar to https://github.com/chapmanb/biocloudcentral to launch instances because for the non-amazon case, the user data ( http://wiki.g2.bx.psu.edu/CloudMan/UserData) required by an instance is a bit tedious to compose by hand (the user data link just explains what the user data is but the full set of user data fields required for non-amazon clouds is not yet documented - I'll do that soon). Hope this helps. I'm also CCing the galaxy-dev mailing list because others may be interested in this topic as well. Enis On Tue, Oct 2, 2012 at 9:37 PM, pandorin...@gmail.com wrote: Hi Enis! In our institute we process biological data. And we have cloud, based on OpenStack. We want to use the Galaxy in the cloud (OpenStack). Are there any solutions for this? (or just Amazon) Thank you so much for the Galaxy! :) And sorry for my English. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Starting a Cloud Instance
Hi Mark, You can always confirm the zone you started an instance in by logging in to the AWS console and, under Instances, find the instance you're wanting to check. After selecting it, under the instance properties, the zone will be displayed for the given instance. The security group and the key pair are rather important to get right for being able to access the instance. However, it seems you're connecting to an existing cluster and are able to access the web console, so I would think the problem is really in having started the instance in the wrong zone vs. the security key issue? Let us know if you have any more issues, Enis On Wed, Aug 8, 2012 at 9:44 PM, Jennifer Jackson j...@bx.psu.edu wrote: Re-Post to galaxy-dev On 8/7/12 3:01 AM, Mark Lindsay wrote: I wondered if you might be able to help. I have very little computing experience but have been trying to launch a cloud instance for the last 24 hrs. Initially I had problems launching the instance in Sahari i.e. it kept saying that it could not connect to the relevant page. When I have finally got top the CloudMan launch page it is giving the following message [Critical] Volume 'vol-dd6397db' is located in the wrong availability zone for this instance. You MUST terminate this instance and start a new one in zone 'us-east-1a' despite the fact that I am sure I have started in zone 'us-east-1a. How important are the security codes and keys? BW Mark __**_ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/**listinfo/galaxy-devhttp://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Jennifer Jackson http://galaxyproject.org __**_ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] adding IDR to Amazon AMI
Hi Quang, We have no immediate plans to extend the toolset of the tools available on the default image so I'd suggest one of two options to get this tool out: 1. Create a Galaxy Tool wrapper for it and upload it to the Tool Shed at http://toolshed.g2.bx.psu.edu/ 2. Install it on an Galaxy CloudMan instance and use the 'share-a-cluster' feature to make a copy of the entire instance available to others Hope this helps, Enis On Sun, Jun 17, 2012 at 10:04 AM, Quang Trinh quang.tr...@gmail.com wrote: Hi dev, Just wondering if there is plan to add IDR ( please see the link below ) to Galaxy Amazon AMI? https://sites.google.com/site/anshulkundaje/projects/idr IDR is being used by both modENCODE and ENCODE projects for ChiP-seq analysis. Thanks, Q ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Python revs for latest Galaxy and new Cloudman ?
Hi Greg, On my laptop, I run Python 2.7.1 and as much as one can test things locally, things are working fine. The version on the AMI you mention is obviously working fine so I'd say that there shouldn't be any surprises arising from the different Python versions. Cheers, Enis On Fri, Jun 15, 2012 at 6:21 PM, Greg Edwards gedwar...@gmail.com wrote: Hi, Just checking before I plunge in and upgrade various things. Is the latest Galaxy and Cloudman ( http://wiki.g2.bx.psu.edu/News/NewCloudManRelease ) ok on Python 2.7.3 ? And can I ask what Python rev is incorporated in the new Cloudman AMI ? Presumably a little earlier than 2.7.3. My current 861460482541/galaxy-cloudman-2011-03-22 is Python 2.6.5. Current local Mac is 2.6.1. Sorry about somewhat dim questions. I'm starting a big Python learning binge and want to set up a local Galaxy on Mac, an AWS Galaxy, and a learning environment in the most useful way. Thanks. -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] CloudMan: Command not found
Jose, Did you see the 'Share-an-instance' option within CloudMan? It's the green icon next to the cluster name in the console. It should do exactly what you're describing. Cheers, Enis On Thu, Jun 14, 2012 at 11:46 AM, Jose Navas josenavasmol...@hotmail.comwrote: Ok, I will change my XML files. I've another question. When I customize my cluster, I can make this customizations permanent through the CloudMan management console. We are trying to customize the Galaxy instance with our tools and then make an AMI public where the users can use our tools through the Galaxy interface. What should I do to create this AMI? Thanks, Jose Subject: Re: [galaxy-dev] CloudMan: Command not found From: dannonba...@me.com Date: Wed, 13 Jun 2012 21:32:24 -0400 CC: galaxy-dev@lists.bx.psu.edu To: josenavasmol...@hotmail.com You can probably modfiy /home/galaxy/.sge_request to make it work for now, but for portability of tools I'd still recommend the requirements tag. Changes to .sge_request will not be persisted after shutdown, so you'll have to redo it with each new cluster. On Jun 13, 2012, at 9:25 PM, Jose Navas wrote: Hi Dannon, thank you for your help. I've tried the requirements tag option with the 'env.sh' script. I have added ~100 XML files... there isn't any option to modify the environment variables directly? If not I will modify all my XML files thanks! Jose Subject: Re: [galaxy-dev] CloudMan: Command not found From: dannonba...@me.com Date: Wed, 13 Jun 2012 21:05:02 -0400 CC: galaxy-dev@lists.bx.psu.edu To: josenavasmol...@hotmail.com Instead of modifying the environment variables directly, I've found it easier to use Galaxy's built in tool dependency framework. All you'd have to do is add a requirements type=package tag to your custom tools to specify the package directory that needs to be checked at runtime. See http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies, specifically the Managed Tool Dependencies section. Many of the other tools on the cloud volume use the 'env.sh' method discussed there. For example, blast has the 'blast' package requirement, and in the /mnt/galaxyTools/blast/default directory you'll find an 'env.sh' file that configures the environment at runtime. -Dannon On Jun 13, 2012, at 8:49 PM, Jose Navas wrote: Hi Galaxy Team, I'm modifying my Galaxy CloudMan instance by adding custom tools. I've installed my tools under the /mnt/galaxyTools/tools folder and I've modified the .bashrc files from the sgeadmin and galaxy users to add the needed paths to the PATH and PYTHONPATH variables. When I'm in Galaxy and I try to launch one of my tools, it fails and shows the 'Command not found' error. Where I should add the paths to make Galaxy now where are my executables? Whan I log into the isntance through ssh and I use the galaxy user, it knows where are my executables. What I'm missing? Thanks, Jose ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Public IP Address
Hi Madeleine, By design (ie, for your own benefit), there is no way for anyone but yourself to check on the status of an AWS instance once started. In order to check what happened and if an instance got started, you should log into https://console.aws.amazon.com/ec2/home with the same AWS credentials you used on biocloudcentral.org and check your instance. There, under instance properties, you should also be able to get the public IP of an instance. This is not the expected behavior of the above portal so if happens again, please let us know again so we can dig a deeper. Enis On Sat, May 19, 2012 at 8:33 AM, Madeleine Matias madeleine.mat...@gmail.com wrote: Hello, I recently launched a new instance about 6 minutes ago using BioCloud Central (https://biocloudcentral.herokuapp.com/launch) but did not receive a Public IP address. I'm not sure if the instance was fully launched. Here is the Instance and Image ID I received: Instance ID i-6789fa01 Image ID (AMI) ami-500cd139 Can you please verify if an instance is in fact running and what the Public IP address is? Thank you, Madeleine ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Moving data about the EC2/EBS/S3 spaces on the Amazon cloud.
Hi Mo, On Wed, Apr 18, 2012 at 4:53 AM, Mohammad Heydarian mheyd...@jhmi.eduwrote: Hello, I am new to cloud computing and am trying to use the Galaxy cloudman service through Amazon to analyse NGS data. I have a couple of questions regarding data transfer: If I am running an EC2 instance, with an EBS volume attached, how can I retrieve the data from the EBS volume? If the data is not accessible via Galaxy (in which case you can use the Save icon for a given dataset in your History), you will need to scp the files over to your local instance. A command like the following should work: [local] $ scp -i path to private key ubuntu@ec2-rest of instance DNS:path to files (see below) path on your local machine In addition, how do I view what files are in the EBS volume? If using CloudMan to setup a data volume, all the user data will be stored under /mnt/galaxyData/. There will be more directories there with the Galaxy ones under the 'files' subdir. Can I send the data to an S3 bucket? And how would I move data from the S3 bucket to a fresh EC2 instance? This needs to be done manually via ec2 tools, a combnation of boto and pyhton or a combination of scp and a GUI tools such as cyberduck. Assuming I can store my files on EBS (or S3), what is the best way to download them? From EBS, you must use scp; from S3 you can use the AWS console or a 3rd party tool (cyberduck) Can I set up a ftp transfer to (quickly) download my potentially very large files? CloudMan sets up an FTP server on the instance for use with Galaxy ( http://wiki.g2.bx.psu.edu/FTPUpload). For downloading files, you could probably leverage that server but it's likely to take some playing around with (here are instructions mirrored on the cloud instance that may be useful http://wiki.g2.bx.psu.edu/Admin/Config/Upload%20via%20FTP). Hope this clears up some of your questions, Enis Many thanks for any insight and direction! Cheers, Mo Heydarian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] CloudMan/Galaxy wiki suggestion
Hi Gordon,
On Sat, Apr 14, 2012 at 8:06 AM, Assaf Gordon gor...@cshl.edu wrote:
Hello,
Great work with the Galaxy on the Cloud - it's the first time I've used it
and it looks very impressive (and I'm not a big cloud fan :) ).
Thank you very much for the comment!
Once I got it working it was very nice, but before it worked I encountered
two issues (not sure if those were voodoo or not) that could be perhaps
documented in the Wiki (under troubleshooting?):
1. the User Data is sensitive to trailing white-space, or senstive to
copypaste in the edit box.
I accidentally had:
access_key: XXspacespace
and it seems it completely prevent the Cloudman from starting.
when the stared, I couldn't connect to it with a web-browser.
I logged on using SSH, and saw that there was nothing running/listening on
port 80 (no nginx, no python).
Not sure exactly why it happened, but it did.
removing white-space (and also saving the user-data to a file instead of
pasting it directly in the editbox) solved the problem.
Noted and will be fixed before the next update.
You may also consider using biocloudcentral.org as a quick way to starting
up CloudMan clusters without having to worry about the formatting of the
user data.
2. If the user-data password is all numeric, there's no way to login to
the cloudman.
I suspect the reason is that an all numeric password is treated as a
number, not a string.
Then, the log shows:
$ cat /opt/galaxy/pkg/ec2autorun.py.log
[...]
[DEBUG] ec2autorun:337 2012-04-13 20:22:50,983: Composed user data:
{'access_key': 'XX', 'cloudman_home': '/mnt/cm',
'cluster_name': 'galaxy-TGAC-workshop', 'bucket_default': 'cloudman',
'role': 'master', 'bucket_cluster': 'cm-169ca4be88c386ce8ec3078d69a6dc3a',
'boot_script_path': '/tmp/cm', 'boot_script_name': 'cm_boot.py',
'secret_key': 'X', 'password': 12345678}
[...]
you'll notice that the password key is a numeric value, not a string - and
login wasn't possible (perhaps it was voodoo and I'm wrong in my guess, but
once I changed it to alphanumeric password, with the same user data
settings, it workd).
Will be fixed as well.
Thanks again for reporting these issues,
Enis
-gordon
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Reducing costs in Cloud Galaxy
Awesome! Also glad to hear your rate of success can be quantified as a 700% improvement. Amazing! :) Best, Enis On Wed, Mar 28, 2012 at 2:57 PM, Greg Edwards gedwar...@gmail.com wrote: Enis, Thanks, your instructions below worked ok and I have reduced my 700GB to 1 GB. No doubt a pile of genome tools don't work now but I don't need them. I see this is described in the Wiki as well, but it was more concise below, thanks. cp -r was fine for copying the remnants of /mnt/galaxyIndices. Cheers, Greg E 2. The vast majority of the storage costs are fro the Gemome databases in the 700GB /mnt/galaxyIndices, which I don't need. Can this be reduced to the bare essentials ? You can do this manually: 1. Start a new Galaxy cluster (ie, one you can easily delete later) 2. ssh into the master instance and delete whatever genomes you don't need/want (these are all located under /mnt/galaxyIndices) 3. Create a new EBS volume of size that'll fit whatever's left on the original volume, attach it and mount it 4. Copy over the data from the original volume to the new one while keeping the directory structure the same (rsync is probably the best tool for this) 5. Unmount detach the new volume; create a snapshot from it 6. For the cluster you want to keep around (while it is terminated), edit persistent_data.yaml in it's bucket on S3 and replace the existing snap ID for the galaxyIndices with the snapshot ID you got in the previous step 7. Start that cluster and you should have a file system from the new snapshot mounted. 8. Terminate delete the cluster you created in step 1 If you don't want to have to do this the first time around on your custom cluster, you can first try it with another temporary cluster and make sure it all works as expected and then move on to the real cluster. Best, Enis ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Reducing costs in Cloud Galaxy
Greg, Regarding the performance of different types of instances, I came across this and thought you might potentially find it useful: http://cloudharmony.com/benchmarks Enis On Mon, Mar 19, 2012 at 7:49 PM, Greg Edwards gedwar...@gmail.com wrote: Enis, Thanks. Will try that re the storage. Greg E On Mon, Mar 19, 2012 at 4:49 PM, Enis Afgan eaf...@emory.edu wrote: Hi Greg, On Mon, Mar 19, 2012 at 11:01 AM, Greg Edwards gedwar...@gmail.comwrote: Hi, I've got an implementation of some proteomics tools going well in Galaxy on AWS EC2 under Cloudman. Thanks for the help along the way. I need to drive the costs down a bit. I'm using an m1.large AMI and it's costing about $180 - $200 / month. This is about 55% storage and 45% instance costs. That's peanuts in some senses but for now we need to get it down so that it comes out of petty cash for the department, while the case is proven for it's use. I have a few questions and would appreciate ny insights .. 1. AWS has just released an m1.medium and m1.small instance type, which are 1/2 and 1/4 the cost of m1.large. http://aws.amazon.com/ec2/instance-types/ http://aws.amazon.com/ec2/pricing/ I tried the m1.small and m1.medium with the latest Cloudman AMI * *galaxy-cloudman-2011-03-22 (ami-da58aab3) All seemed to install ok, but the Tools took up tp 30 minutes to start execution on m1.medium, and never started on m1.small. m1.medium only added about 15% to run times compared with m1.large, can't say for m1.small. t1.micro does run (and for free in my Free Tier first year) but blows execution times out by a factor of about 3 which is too much. Has anyone tried these new Instance Types ? (m1.small/medium) I have no real experience with these instance types yet either so maybe someone else can chime in on this? 2. The vast majority of the storage costs are fro the Gemome databases in the 700GB /mnt/galaxyIndices, which I don't need. Can this be reduced to the bare essentials ? You can do this manually: 1. Start a new Galaxy cluster (ie, one you can easily delete later) 2. ssh into the master instance and delete whatever genomes you don't need/want (these are all located under /mnt/galaxyIndices) 3. Create a new EBS volume of size that'll fit whatever's left on the original volume, attach it and mount it 4. Copy over the data from the original volume to the new one while keeping the directory structure the same (rsync is probably the best tool for this) 5. Unmount detach the new volume; create a snapshot from it 6. For the cluster you want to keep around (while it is terminated), edit persistent_data.yaml in it's bucket on S3 and replace the existing snap ID for the galaxyIndices with the snapshot ID you got in the previous step 7. Start that cluster and you should have a file system from the new snapshot mounted. 8. Terminate delete the cluster you created in step 1 If you don't want to have to do this the first time around on your custom cluster, you can first try it with another temporary cluster and make sure it all works as expected and then move on to the real cluster. Best, Enis Using m1.small/medium and getting rid of the 700GB would being my costs down to say $50 / month which is ok. Thanks ! Greg E -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Reducing costs in Cloud Galaxy
Hi Greg, On Mon, Mar 19, 2012 at 11:01 AM, Greg Edwards gedwar...@gmail.com wrote: Hi, I've got an implementation of some proteomics tools going well in Galaxy on AWS EC2 under Cloudman. Thanks for the help along the way. I need to drive the costs down a bit. I'm using an m1.large AMI and it's costing about $180 - $200 / month. This is about 55% storage and 45% instance costs. That's peanuts in some senses but for now we need to get it down so that it comes out of petty cash for the department, while the case is proven for it's use. I have a few questions and would appreciate ny insights .. 1. AWS has just released an m1.medium and m1.small instance type, which are 1/2 and 1/4 the cost of m1.large. http://aws.amazon.com/ec2/instance-types/ http://aws.amazon.com/ec2/pricing/ I tried the m1.small and m1.medium with the latest Cloudman AMI * *galaxy-cloudman-2011-03-22 (ami-da58aab3) All seemed to install ok, but the Tools took up tp 30 minutes to start execution on m1.medium, and never started on m1.small. m1.medium only added about 15% to run times compared with m1.large, can't say for m1.small. t1.micro does run (and for free in my Free Tier first year) but blows execution times out by a factor of about 3 which is too much. Has anyone tried these new Instance Types ? (m1.small/medium) I have no real experience with these instance types yet either so maybe someone else can chime in on this? 2. The vast majority of the storage costs are fro the Gemome databases in the 700GB /mnt/galaxyIndices, which I don't need. Can this be reduced to the bare essentials ? You can do this manually: 1. Start a new Galaxy cluster (ie, one you can easily delete later) 2. ssh into the master instance and delete whatever genomes you don't need/want (these are all located under /mnt/galaxyIndices) 3. Create a new EBS volume of size that'll fit whatever's left on the original volume, attach it and mount it 4. Copy over the data from the original volume to the new one while keeping the directory structure the same (rsync is probably the best tool for this) 5. Unmount detach the new volume; create a snapshot from it 6. For the cluster you want to keep around (while it is terminated), edit persistent_data.yaml in it's bucket on S3 and replace the existing snap ID for the galaxyIndices with the snapshot ID you got in the previous step 7. Start that cluster and you should have a file system from the new snapshot mounted. 8. Terminate delete the cluster you created in step 1 If you don't want to have to do this the first time around on your custom cluster, you can first try it with another temporary cluster and make sure it all works as expected and then move on to the real cluster. Best, Enis Using m1.small/medium and getting rid of the 700GB would being my costs down to say $50 / month which is ok. Thanks ! Greg E -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy Cloudman - How to analyse 1TB data ?
Hi Yves, When you create the LVM file system - are you composing it from the volume that already contains the data (i.e., directory structure++ created by CloudMan) and then adding another volume into the LVM or starting with 2 new, clean volumes? Maybe trying again and not messing with SGE at all would at least resolve the SGE issue. Namely, SGE is on the root file system so it should be fine as is. I'd suggest stopping Galaxy and PostgreSQL services (from the CloudMan Admin), from the CLI, unmount galaxyData file system and proceed to create the LVM. Mount the file system and ensure the directories and the data that were there are still there. The start back PostgreSQL and Galaxy services. See if it all comes up fine and try adding a worker node if it does. Currently, CloudMan does not support composition of a file system from multiple volumes but I would think that as long as you did not restart the cluster and created the file system manually, things would work fine. I've been thinking about why you're seeing the described behavior and am not really sure so please let me know how the above process works out. On Thu, Feb 16, 2012 at 7:37 PM, Wetzels, Yves [JRDBE Extern] ywet...@its.jnj.com wrote: Hi Brad I did not restart the master CloudMan node. I only restarted the services (Galaxy, PostgreSQL and SGE). I do not have these problems without creating the logical volume. Kind Regards Yves Yves; I'm hoping Enis can jump in here since he is more familiar with the internals of CloudMan and may be able to offer better advice. I can tell you what I see from your error messages. I used LVM2 to create the logical volume. Does this involve stopping and restarting the master CloudMan node? The error messages you are seeing look like SGE is missing or not properly configured on the master node: 02/15/2012 11:22:08| main|domU-12-31-39-0A-62-12|E|error opening file /opt/sge/default/common/./sched_configuration for reading: No such file or directory [...] DeniedByDrmException: code 17: error: no suitable queues which is causing the job submission to fail since it can't find the SGE cluster environment to submit to. The strange thing is that SGE is present in /opt on the main EBS store, so I wouldn't expect your modified /mnt/galaxyData volume to influence this. Since starting worker nodes appears to be fine, I'd focus on the main instance manipulations you are doing. Perhaps some of the setup causes the problem without creating the logical volume? This could help narrow down the issue and hopefully get you running again. Hope this helps, Brad ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Expanding galaxyTool volume?
No problem; this should get you there: 1. With CloudMan running, go to the Admin console and stop Galaxy and PostgreSQL services (in that order) 2. From instance's CLI: sudo umount /mnt/galaxyTools 3. From the AWS console, detach the tools volume (but remember as which device it was attached) 4. From the AWS console, create a snapshot of the detached volume 5. From the AWS console, create a new volume from the newly created snapshot of the desired size. 6. From the AWS console, attach the new larger volume to the running instance (attach it as a different device then the original volume that was attached) 7. From instance's CLI: sudo mount device /mnt/galaxyTools 8. From instance's CLI: sudo xfs_growfs /mnt/galaxyTools 9. From instance's CLI: sudo umount /mnt/galaxyTools 10. From the AWS console, detach the volume and create a snapshot 11. From the AWS console, attach the original volume as the same device as it was attached 12. From instance's CLI: sudo mount device /mnt/galaxyTools 13. From CloudMan Admin, 'File systems' service should be running now. If so, start PostgresSQL and Galaxy services (in that order) 14. From CloudMan, Terminate cluster 15. From the AWS S3 console, in the cluster's bucket, edit the 'persistent_data.yaml' galaxyTools file system to point to the new snapshot and its size is properly set (snapshot from step 10) 16. Start the cluster back up using the same user data. Now you should have the new file system there and any changes you want to make can be done from CLI. Then persist file system changes from the CloudMan Admin to keep those around after you restart the cluster. I have not actually tried this but am speaking from memory so there may be things that do not end up working quite like this but the general concept is there. Good luck and let us know how it goes. Enis On Fri, Feb 17, 2012 at 12:34 AM, Dave Lin d...@verdematics.com wrote: Hi Enis, I installed a new test cluster earlier today and did notice that the new clusters magically now have galaxyTool volumes with 10GB. That is a good change. However, you are correct. I have an existing cluster (that had the old 2 GB volume size) that I'm trying to expand. With additional tools and log files, that volume keeps getting full. Can you help guide me through this process? Thanks again, Dave On Thu, Feb 16, 2012 at 2:36 PM, Enis Afgan eaf...@emory.edu wrote: Hi Dave, Are you trying to modify the size of the tools volume for a cluster that's been around for a while and you customized already or could this be a new cluster? The reason I'm asking is because as of Tuesday (3 days ago), the default tools volume for any new cluster will be 10GB (vs 2GB previously) and only 1.7GB are taken. I would hope that gives plenty of storage space for majority of anyone's needs. Let me know if you need to modify an existing cluster and I'll guide you through the process then. Enis On Thu, Feb 16, 2012 at 9:31 PM, Dave Lin d...@verdematics.com wrote: Hi All, What is the recommend process for expanding the galaxyTool volume for an existing galaxy instance (using EC2/cloudman)? I tried the following, but it didnt' work for me. 0) Terminate cluster. 1) Amazon EC2- create snapshot of current galaxyTools volume 2) Amazon EC2- create volume from step 1 + specify desired volume size. 3) Amazon EC2- create new snapshot from Step 2. 4) Amazon S3- identify S3 bucket for this cluster. Modify persistent_data.yaml. Modify size and snap_id to correspond with step #3 5) Amazon EC2- Start new instance-- using same AmazonID + ClusterName I was expecting the new instance to startup and create a galaxyTools volume based on the snapshot identified in the persistent_data.yaml file, but that didn't seem work. Thanks in advance for any pointers. Dave ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] Using Galaxy Cloudman for a workshop
Hi Clare, The share string is generated when you share a cluster. The string is accessible on the shared cluster, when you click the green 'Share a cluster' icon next to the cluster name and then the top link Shared instances. You will get a list of the point in time shares of the cluster you have created. The share string will look something like this cm-cd53Bfg6f1223f966914df347687f6uf32/shared/2011-10-19--03-14 You simply paste that string into new cluster box you mentioned. Enis On Thu, Dec 1, 2011 at 6:31 AM, Clare Sloggett s...@unimelb.edu.au wrote: Hi Enis, Jeremy, and all, Thanks so much for all your help. I have another question which I suspect is just me missing something obvious. I'm guessing that when you cloned the cluster for your workshop, you used CloudMan's 'share-an-instance' functionality? When I launch a new cluster which I want to be a copy of an existing cluster, and select the share-an-instance option, it asks for the cluster share-string. How can I find this string for my existing cluster? Or have I got completely the wrong idea - did you actually clone the instance using AWS functionality? Thanks, Clare On Mon, Nov 21, 2011 at 5:37 PM, Enis Afgan eaf...@emory.edu wrote: Hi Clare, I don't recall what instance type we used earlier, but I think an Extra Large Instance is going to be fine. Do note that the master node is also being used to run jobs. However, if it's loaded by just the web server, SGE will typically just not schedule jobs to it. As far as the core/thread/slot concerns goes, SGE sees each core as a slot. Each job in Galaxy simply requires 1 slot, even if it uses multiple threads (i.e., cores). What this means is that nodes will probably get overloaded if only the same type of job is being run (BWA), but if analyses are being run that use multiple tools, jobs will get spread over the cluster to balance the overal load a bit better than by simply looking at the number of slots. Enis On Mon, Nov 21, 2011 at 4:34 AM, Clare Sloggett s...@unimelb.edu.au wrote: Hi Jeremy, Also if you do remember what kind of Amazon node you used, particularly for the cluster's master node (e.g. an 'xlarge' 4-core 15GB or perhaps one of the 'high-memory' nodes?), that would be a reassuring sanity chech for me! Cheers, Clare On Mon, Nov 21, 2011 at 10:37 AM, Clare Sloggett s...@unimelb.edu.au wrote: Hi Jeremy, Enis, That makes sense. I know I can configure how many threads BWA uses in its wrapper, with bwa -t. But, is there somewhere that I need to tell Galaxy the corresponding information, ie that this command-line task will make use of up to 4 cores? Or, does this imply that there is always exactly one job per node? So if I have (for instance) a cluster made of 4-core nodes, and a single-threaded task (e.g. samtools), are the other 3 cores just going to waste or will the scheduler allocate multiple single-threaded jobs to one node? I've cc'd galaxy-dev instead of galaxy-user as I think the conversation has gone that way! Thanks again, Clare On Fri, Nov 18, 2011 at 2:36 PM, Jeremy Goecks jeremy.goe...@emory.edu wrote: On Fri, Nov 18, 2011 at 12:56 AM, Jeremy Goecks jeremy.goe...@emory.edu wrote: Scalability issues are more likely to arise on the back end than the front end, so you'll want to ensure that you have enough compute nodes. BWA uses four nodes by default--Enis, does the cloud config change this parameter?--so you'll want 4x50 or 200 total nodes if you want everyone to be able to run a BWA job simultaneously. Actually, one other question - this paragraph makes me realise that I don't really understand how Galaxy is distributing jobs. I had thought that each job would only use one node, and in some cases take advantage of multiple cores within that node. I'm taking a node to be a set of cores with their own shared memory, so in this case a VM instance, is this right? If some types of jobs can be distributed over multiple nodes, can I configure, in Galaxy, how many nodes they should use? You're right -- my word choices were poor. Replace 'node' with 'core' in my paragraph to get an accurate suggestion for resources. Galaxy uses a job scheduler--SGE on the cloud--to distribute jobs to different cluster nodes. Jobs that require multiple cores typically run on a single node. Enis can chime in on whether CloudMan supports job submission over multiple nodes; this would require setup of an appropriate parallel environment and a tool that can make use of this environment. Good luck, J. -- E: s...@unimelb.edu.au P: 03 903 53357 M: 0414 854 759 -- E: s...@unimelb.edu.au P: 03 903 53357 M: 0414 854 759 -- E: s...@unimelb.edu.au P: 03 903 53357 M: 0414 854 759
Re: [galaxy-dev] Removing nodes from a CloudMan instance
Unfortunately not. When removing nodes, CloudMan chooses from the nodes that are currently not being used but among those the choice is random. You can manually terminate a particular worker instance from the AWS console and thus remove it from your cluster by force. CloudMan will then reconfigure the cluster. Although this has been implemented and tested, it is not really the recommended behavior, especially not repeatedly. I'll look into how to implement this option. Enis On Thu, Dec 1, 2011 at 6:20 AM, Clare Sloggett s...@unimelb.edu.au wrote: Hi galaxy-devs, Quick question: when using the cloud console on CloudMan, it's possible to add different types of nodes (large, micro, etc) to the virtual cluster using the 'Add Nodes' option at the top. I can also remove a given number of nodes using the 'Remove Nodes' option at the top. However, is there any way to control exactly which node (or more importantly just which type of node) gets removed? Thanks for any help! Clare -- E: s...@unimelb.edu.au P: 03 903 53357 M: 0414 854 759 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] GalaxyCloudman + CADDSuite
Hi Marcel, However, when I create an AMI, terminate the cluster and create a new cluster using the new AMI, both /mnt/galaxyData and /mnt/galaxyTools do not exist anymore, i.e. /dev/sdg3 and /dev/sdg4 are not mounted automatically. If I mount those two devices manually, everything runs smoothly again. So, is there anything that I might have forgotten to do while creating the AMI? Is there a way to make sure that those devices will be mounted automatically? It is not necessary to create a new AMI when wanting to customize your cluster. Instead, on the admin interface - after you have modified the file systems, there is an option to persist static file systems (galaxyTools galaxyIndices). Once the process is completed and you restart the cluster, just continue to use the same AMI. CloudMan will use the new, customized, data snapshots at runtime. Let us know how it goes, Enis Regards, Marcel On 11/30/11 3:57 PM, Enis Afgan wrote: Hi Marcel, It would be best to use 'galaxy' user to add any tools. To do so, after you've logged in as ubuntu user, simply execute: sudo su galaxy and you will become galaxy user. You can then make the desired modifications. Good luck, Enis On Wed, Nov 30, 2011 at 3:42 PM, Greg Von Kusterg...@bx.psu.edu wrote: Hello Marcel, In the future, please send all questions like this to the galaxy-dev mailing list, as doing so will streamline the process of getting a timely answer. I believe Enis is best able to answer your questions. Thanks! On Nov 30, 2011, at 9:29 AM, Marcel Schumann wrote: Hi Greg, I'm currently trying to create a GalaxyCloudman version that includes CADDSuite. Thus, I launched GalaxyCloudman as described in your wiki and tried to modify it afterwards. Well, starting cloudman worked without any problems... so far, so good :-) As described on http://wiki.g2.bx.psu.edu/**Admin/Cloud/Customize%** 20Galaxy%20Cloudhttp://wiki.g2.bx.psu.edu/Admin/Cloud/Customize%20Galaxy%20Cloud I could then log-in via ssh as user 'ubuntu' (not as user 'galaxy'). However, all files of the galaxy installation belong to user and group 'galaxy'. Thus my question: How should users be able to customize cloudman? Is there some trick by which I can log-in as 'galaxy' or do you have any other idea how to make this work ? ;-) Sorry Greg, if you are not the correct contact in this case, but I found not specific contact or mailing list for cloudman. Perhaps, you could just forward this mail in that case ... Cheers, Marcel -- Marcel Schumann University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schum...@informatik.uni-**tuebingen.deschum...@informatik.uni-tuebingen.de Greg Von Kuster Galaxy Development Team g...@bx.psu.edu -- Marcel Schumann University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schum...@informatik.uni-**tuebingen.deschum...@informatik.uni-tuebingen.de ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] GalaxyCloudman + CADDSuite
Hi Marcel, On Thu, Dec 1, 2011 at 11:28 AM, Marcel Schumann schum...@informatik.uni-tuebingen.de wrote: Hi Enis, well, I know I do not have to create a new AMI if I want to reuse an instance myself. However, I would like to share the modified GalaxyCloudman version with other people and therefore I do have to create an AMI. Unless you modify the system packages (i.e., your customizations are not self contained), you still don't have to create a new AMI to share a cluster. There is the share-a-cluster option (icon next to the cluster name). Just wanted to make sure you were aware of the functionality... Ok, I will try to make this work somehow ... but I guess there are no immediate clues as to what could have gone wrong? Or do you have any ideas what I should try? CloudMan sets up the system at runtime so it performs changes that then get persisted when you create the AMI. So, it is necessary to reverse those changes before creating the AMI so that next time a cluster is started, the startup procedure proceeds as before. Did you see what's in the cloudman log (/mnt/cm/paster.log) on your customized AMI? That's probably the easiest place to start and we can work from there. Enis Cheers, Marcel On 12/1/11 10:45 AM, Enis Afgan wrote: Hi Marcel, However, when I create an AMI, terminate the cluster and create a new cluster using the new AMI, both /mnt/galaxyData and /mnt/galaxyTools do not exist anymore, i.e. /dev/sdg3 and /dev/sdg4 are not mounted automatically. If I mount those two devices manually, everything runs smoothly again. So, is there anything that I might have forgotten to do while creating the AMI? Is there a way to make sure that those devices will be mounted automatically? It is not necessary to create a new AMI when wanting to customize your cluster. Instead, on the admin interface - after you have modified the file systems, there is an option to persist static file systems (galaxyTools galaxyIndices). Once the process is completed and you restart the cluster, just continue to use the same AMI. CloudMan will use the new, customized, data snapshots at runtime. Let us know how it goes, Enis Regards, Marcel On 11/30/11 3:57 PM, Enis Afgan wrote: Hi Marcel, It would be best to use 'galaxy' user to add any tools. To do so, after you've logged in as ubuntu user, simply execute: sudo su galaxy and you will become galaxy user. You can then make the desired modifications. Good luck, Enis On Wed, Nov 30, 2011 at 3:42 PM, Greg Von Kusterg...@bx.psu.edu wrote: Hello Marcel, -- Marcel Schumann University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schum...@informatik.uni-**tueb**ingen.de http://tuebingen.de schumann@informatik.**uni-tuebingen.deschum...@informatik.uni-tuebingen.de -- Marcel Schumann University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schum...@informatik.uni-**tuebingen.deschum...@informatik.uni-tuebingen.de ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] GalaxyCloudman + CADDSuite
Hi Marcel, On Thu, Dec 1, 2011 at 10:35 PM, Marcel Schumann schum...@informatik.uni-tuebingen.de wrote: Hi Brad, hi Enis, ok, so one possible way should thus be to use the _cleanup_ec2() function before creating an AMI via the amazon console and the second option perhaps is this share-a-cluster functionality. About the latter: sorry, I tried to find out what it really does, but did not find any documentation of it. So, how does this share-a-cluster functionality work in principle? If it does not create AMI, does it just create snapshots of the individual disks that users have to mount (manually) or ... ? My point here is: I need to make a modified version of GalaxyCloudman available (i.e. one that contains the CADDSuite tools). I do not just want to share an _instance_ (a running cluster) but a cloudman image (as an AMI or by some other means) that includes all my tools, so that other users can easily start their own server. The share-an-instance functionality creates snapshot the data volume (/mnt/galaxyData) so any tool that is installed on that file system will get bundled into the shared instance. Also, any modifications that you make to you galaxy configuration (e.g., galaxy_tools.xml) become part of the shared instance. When a user then starts a new cluster and provides the cluster share string (this is available on the shared cluster after you share it), they will get access to an exact replica of your cluster. Like, I mentioned earlier, unless you are modifing system libraries there is absolutely no reason to create a new AMI. Simply contain your customizations to the file system and use cluster sharing functionality. Try it and see if it does what you need it to do. If you do indeed have a description of this somewhere that I just did not find, then I am sorry but would be grateful for a link ;-) Sorry but there's no functionality about this feature on the wiki. However, there is a pretty detailed description of the functionality and the process on the cloudman console after you click the green share a cluster button. I'll work on preparing some additional documentation on this topic. Enis Cheers, Marcel On 12/1/11 10:04 PM, Brad Chapman wrote: Marcel; well, I know I do not have to create a new AMI if I want to reuse an instance myself. However, I would like to share the modified GalaxyCloudman version with other people and therefore I do have to create an AMI. What Enis was suggesting is using the share-a-cluster functionality built into CloudMan. This bundles your data volumes as snapshots and prepares a sharable cluster than anyone can initiate. In the CloudMan interface, there is a little green button next to the cluster name that enables this. That is definitely the easiest way to share and distribute modified CloudMan versions. Ok, I will try to make this work somehow ... but I guess there are no immediate clues as to what could have gone wrong? Or do you have any ideas what I should try? After CloudMan boots once, you need to clean up some files before preparing an AMI. This is the automated code we use to clean up for prepping CloudMan compatible AMIs: https://github.com/chapmanb/**cloudbiolinux/blob/master/** cloudbio/cloudman.py#L87https://github.com/chapmanb/cloudbiolinux/blob/master/cloudbio/cloudman.py#L87 Be careful if you run that directly. It runs immediately before bundling and removes the ssh keys (so you don't have a backdoor to the AMI you are distributing) so you want to do it as the last thing. It also assumes you have unmounted all of the associated Galaxy data libraries. Hope this helps, Brad -- Marcel Schumann University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schum...@informatik.uni-**tuebingen.deschum...@informatik.uni-tuebingen.de ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] GalaxyCloudman + CADDSuite
Hi Marcel, It would be best to use 'galaxy' user to add any tools. To do so, after you've logged in as ubuntu user, simply execute: sudo su galaxy and you will become galaxy user. You can then make the desired modifications. Good luck, Enis On Wed, Nov 30, 2011 at 3:42 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Marcel, In the future, please send all questions like this to the galaxy-dev mailing list, as doing so will streamline the process of getting a timely answer. I believe Enis is best able to answer your questions. Thanks! On Nov 30, 2011, at 9:29 AM, Marcel Schumann wrote: Hi Greg, I'm currently trying to create a GalaxyCloudman version that includes CADDSuite. Thus, I launched GalaxyCloudman as described in your wiki and tried to modify it afterwards. Well, starting cloudman worked without any problems... so far, so good :-) As described on http://wiki.g2.bx.psu.edu/Admin/Cloud/Customize%20Galaxy%20Cloud I could then log-in via ssh as user 'ubuntu' (not as user 'galaxy'). However, all files of the galaxy installation belong to user and group 'galaxy'. Thus my question: How should users be able to customize cloudman? Is there some trick by which I can log-in as 'galaxy' or do you have any other idea how to make this work ? ;-) Sorry Greg, if you are not the correct contact in this case, but I found not specific contact or mailing list for cloudman. Perhaps, you could just forward this mail in that case ... Cheers, Marcel -- Marcel Schumann University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schum...@informatik.uni-tuebingen.de Greg Von Kuster Galaxy Development Team g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Configuration of a local install - mi-deploy ?
Hi Clare, As a technical add-on - you didn't mention what type OS you're looking to install this on so I just wanted to mention that mi-deployment scripts are tested primarily on Ubuntu 10.04 LTS and thus most likely to work properly there. If you're also looking to setup the data, unfortunately the data script is fairly well out of date at this point so I am not sure how it will work. Instead, you can try using this script https://github.com/chapmanb/cloudbiolinux/blob/master/data_fabfile.py. Enis On Fri, Nov 11, 2011 at 2:09 AM, Clare Sloggett s...@unimelb.edu.au wrote: Hi Greg Ross, Thanks for this! Yes, I confused the issue by mentioning the Tool Shed, sorry - it's the binaries themselves I need to install. Essentially I think I need to follow the steps at http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup , but I was wondering about mi-deployment scripts as a better way to do this, and whether that's standard practice. After looking through the scripts they really seem like the best option to me - it looks like they are set up so you mostly only need to change configuration variables to get this to work. The scripts are mentioned on the wiki instructions but don't seem to be the default option (they are not bundled in galaxy-dist), so I wondered if people are usually doing it this way? Thanks, Clare On Fri, Nov 11, 2011 at 2:06 AM, Ross ross.laza...@gmail.com wrote: Clare, As Greg says, the tool wrappers exposed on Main all come with a mercurial checkout - but all the binary dependencies, genomic data and indexes do not. The tool shed is really the tool-wrapper shed - binary and data dependencies aren't there either. I haven't tried them but I'd guess that the deploy scripts take care of a lot of messy details but will probably need some work to fit your local setup. As to email for admin - my advice would be don't worry about it - if one user forgets their password you can reset it from the admin interface - that's the main use and if this is really a test instance, it's not a show stopper. On Thu, Nov 10, 2011 at 9:14 AM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Clare, On Nov 9, 2011, at 11:11 PM, Clare Sloggett wrote: Hi all, Most of our playing around with Galaxy has been in getting it working on our local cloud, but now for the first time I'm configuring a non-cloud local install of galaxy-dist (set up as per http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Production%20Server ) So I have some naive questions! Would it be a sensible approach to grab the tools_fabfile script from mi-deployment and use it in this case? Or should I be using the Tool Shed for installing the base set of tools? The Galaxy distribution includes all of the tools you see on both the Penn State test and main instances. You can also get tools form the tool shed if you want - see our wiki at http://wiki.g2.bx.psu.edu/Tool%20Shedfor information about how to do this. I don't think you'll need to use the tools_fabfile script from mi_deployment repo for your local instance. Also, if I have problems with this server sending out emails (which may be the case) am I going to run into trouble with user/password management or can I just admin everything manually? There will only be a very small number of users on this server. I'm not clear on this question, but you certainly shouldn't run into any problems with user / password management within Galaxy. From the Galaxy Admin interface you have the ability to Manage users where you can reset passwords if necessary. If I have missed any good 'getting started' documentation on config/admin of a local install please point me in the right direction. I've been looking at http://wiki.g2.bx.psu.edu/Admin . You found the right wiki. Thanks, Clare -- E: s...@unimelb.edu.au P: 03 903 53357 M: 0414 854 759 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ Greg Von Kuster Galaxy Development Team g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444; -- E: s...@unimelb.edu.au P: 03 903 53357 M: 0414 854 759 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the
Re: [galaxy-dev] Some questions about cloudman
On Fri, Oct 28, 2011 at 5:50 PM, Cittaro Davide cittaro.dav...@hsr.itwrote: On Oct 28, 2011, at 5:35 PM, James Taylor wrote: It currently uses a tiny amount of S3 storage just to save configuration information about your instance. Ok.. never used AWS, actually, I didn't know S3 holds the information. I guess I will have to read some how-to This configuration is something cloudman does behind the scenes so nothing there to worry about much. Long term though we plan to move dataset storage over to S3 as well. Mmm... I've just had a chat with an AWS engineer, he told me that every operation on S3-stored data goes through a download/upload process... isn't that a PITA for data analysis? I guess S3 is for static data EBS has limits, S3 is more durable and scalable. Which limits (in addition to the 1 Tb size)? I know these are not Galaxy-related questions but you are the best people I can ask :-) The 1TB size is the primary issue, especially with NGS data. That's why we're looking into S3 as a way to offload some of the data size issues while handling it all behind the scenes. Other than that, the only other comment is that these instances are independent and self standing so if any customizations are required, manual effort will be required (but this applies to any local instance). Let us know if you have any more questions, Enis d /* Davide Cittaro, PhD Head of Bioinformatics Core Center for Translational Genomics and Bioinformatics San Raffaele Scientific Institute Via Olgettina 58 20132 Milano Italy Office: +39 02 26439140 Mail: cittaro.dav...@hsr.it Skype: daweonline */ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] cloudman on private cloud
Hi Rodrigo, I'm currently working on support for Eucalyptus clouds, so if that's the cloud infrastructure you have setup locally, support will be available soon. However, it is not working yet. Enis 2011/10/10 Rodrigo García Herrera rgar...@inmegen.gob.mx Hello! If I setup a private cloud infrastructure with ubuntu, can I setup galaxy cloudman there? Thanks! Rodrigo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] CloudMan on Eucalyptus
Hi Andrey, CloudMan does not currently support eucalyptus but I am looking into it now so the support is coming (sorry but I don't have a timeline yet, still working on setting up the infrastructure). Enis On Thu, Oct 6, 2011 at 5:13 PM, Andrey Tovchigrechko atovt...@jcvi.orgwrote: We want to use CloudMan to provision and control a cluster with our own VM image (with Galaxy front-end as well). Our plan was to make the VMs for both our internal Eucalyptus cloud as well as EC2. Will CloudMan work with Eucalyptus? If not, how much would our effort have be in order to extend the source code to make it work? I am seeing in the repo an empty file https://bitbucket.org/galaxy/**cloudman/src/1440e0147c85/cm/** clouds/eucalyptus.pyhttps://bitbucket.org/galaxy/cloudman/src/1440e0147c85/cm/clouds/eucalyptus.py Thanks, Andrey __**_ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] cloud instance missing /opt/sge/default/common directory
Are you continuously running this instance and just restarting it or do you start a clean instance each time? I'd suggest ensuring the persistent_data.yaml in your cluster's bucket is correct (see below), terminating this instance, and starting a new one with the same user data as used for this cluster. That way you should get a clean (i.e., standardized) start and that is something that should follow the process correctly. persistent_data.yaml contains the data required to reestablish an existing cluster and must follow the following format but contain proper values (especially if you customized your cluster via custom snapshots). This file is in your cluster's bucket on S3 so you can download it, check it, and upload if anything was changed or got out of sync. data_filesystems: galaxyData: - size: 50 vol_id: vol-909342f8 services: - service: SGE - service: Postgres - service: Galaxy static_filesystems: - filesystem: galaxyIndices size: 700 snap_id: !!python/unicode 'snap-5b030634' - filesystem: galaxyTools size: 2 snap_id: !!python/unicode 'snap-1688b978' On Fri, Sep 23, 2011 at 1:09 PM, Joseph Hargitai joseph.hargi...@einstein.yu.edu wrote: Hi, the mount is no longer there: ubuntu@ip-10-68-42-15:~$ more /etc/hosts 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts ubuntu@ip-10-68-42-15:~$ df -h FilesystemSize Used Avail Use% Mounted on /dev/sda1 15G 9.5G 5.6G 63% / devtmpfs 7.3G 128K 7.3G 1% /dev none 7.6G 0 7.6G 0% /dev/shm none 7.6G 96K 7.6G 1% /var/run none 7.6G 0 7.6G 0% /var/lock none 7.6G 0 7.6G 0% /lib/init/rw /dev/sdb 414G 201M 393G 1% /mnt ubuntu@ip-10-68-42-15:~$ cd /mnt ubuntu@ip-10-68-42-15:/mnt$ ls cm lost+found ubuntu@ip-10-68-42-15:/mnt$ more /etc/fstab # /etc/fstab: static file system information. # file system mount point type options dump pass proc/proc proc nodev,no exec,nosuid 0 0 /dev/sda1 / xfs defaults 0 0 /dev/sdb/mntautodefaults,nobootwait,comment=cloudconfig0 0 -- *From:* Enis Afgan [eaf...@emory.edu] *Sent:* Thursday, September 22, 2011 8:48 AM *To:* Joseph Hargitai *Cc:* galaxy-dev@lists.bx.psu.edu *Subject:* Re: [galaxy-dev] cloud instance missing /opt/sge/default/common directory Hi Joe, And this is happening on a freshly booted instance (from a previously existing cluster) using the same AMI? The order of execution seems a bit odd, seeing Galaxy being removed before SGE is setup; SGE should be the first thing that gets setup so I'm wondering... If you log into the instance, what is in /etc/hosts? Does it match the instance DNS? And if you try executing that same command (cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf) by hand (as root), is any more info produced? Also, qmaster log should be available under /opt/sge/ge6 (or something like this) /default/spool/qmaster/ so please take a look there as well and see if more info is available. On Thu, Sep 22, 2011 at 12:46 AM, Joseph Hargitai joseph.hargi...@einstein.yu.edu wrote: the error is ' [DEBUG] galaxy:139 2011-09-22 00:03:21,055: Galaxy UI does not seem to be accessible. [DEBUG] master:1491 2011-09-22 00:03:21,055: SS: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] root:354 2011-09-22 00:03:24,724: Managing services: [] [INFO] galaxy:30 2011-09-22 00:03:24,724: Removing 'Galaxy' service [INFO] galaxy:122 2011-09-22 00:03:24,724: Shutting down Galaxy... [DEBUG] misc:511 2011-09-22 00:03:26,067: Successfully stopped Galaxy. [DEBUG] root:354 2011-09-22 00:03:33,936: Managing services: [] [DEBUG] sge:61 2011-09-22 00:03:33,937: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-09-22 00:03:33,937: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-09-22 00:03:34,117: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-09-22 00:03:35,557: Configuring SGE... [DEBUG] sge:104 2011-09-22 00:03:35,558: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-09-22 00:03:35,558: Setting up SGE. [ERROR] misc:514 2011-09-22 00:03:35,651: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't
Re: [galaxy-dev] cloud instance missing /opt/sge/default/common directory
Hi Joe, And this is happening on a freshly booted instance (from a previously existing cluster) using the same AMI? The order of execution seems a bit odd, seeing Galaxy being removed before SGE is setup; SGE should be the first thing that gets setup so I'm wondering... If you log into the instance, what is in /etc/hosts? Does it match the instance DNS? And if you try executing that same command (cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf) by hand (as root), is any more info produced? Also, qmaster log should be available under /opt/sge/ge6 (or something like this) /default/spool/qmaster/ so please take a look there as well and see if more info is available. On Thu, Sep 22, 2011 at 12:46 AM, Joseph Hargitai joseph.hargi...@einstein.yu.edu wrote: the error is ' [DEBUG] galaxy:139 2011-09-22 00:03:21,055: Galaxy UI does not seem to be accessible. [DEBUG] master:1491 2011-09-22 00:03:21,055: SS: SGE..Shut down; FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; Galaxy..Starting; [DEBUG] root:354 2011-09-22 00:03:24,724: Managing services: [] [INFO] galaxy:30 2011-09-22 00:03:24,724: Removing 'Galaxy' service [INFO] galaxy:122 2011-09-22 00:03:24,724: Shutting down Galaxy... [DEBUG] misc:511 2011-09-22 00:03:26,067: Successfully stopped Galaxy. [DEBUG] root:354 2011-09-22 00:03:33,936: Managing services: [] [DEBUG] sge:61 2011-09-22 00:03:33,937: Unpacking SGE from '/opt/galaxy/pkg/ge6.2u5' [DEBUG] sge:76 2011-09-22 00:03:33,937: Cleaning '/opt/sge' directory. [DEBUG] sge:82 2011-09-22 00:03:34,117: Unpacking SGE to '/opt/sge'. [INFO] sge:96 2011-09-22 00:03:35,557: Configuring SGE... [DEBUG] sge:104 2011-09-22 00:03:35,558: Created SGE install template as file '/opt/sge/galaxyEC2.conf' [DEBUG] sge:112 2011-09-22 00:03:35,558: Setting up SGE. [ERROR] misc:514 2011-09-22 00:03:35,651: Setting up SGE did not go smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator [: 359: 11: unexpected operator error resolving local host: can't resolve host name (h_errno = HOST_NOT_FOUND) j -- *From:* Enis Afgan [afg...@gmail.com] *Sent:* Tuesday, September 13, 2011 4:20 AM *To:* Joseph Hargitai *Cc:* galaxy-dev@lists.bx.psu.edu *Subject:* Re: [galaxy-dev] cloud instance missing /opt/sge/default/common directory Hi Joe, If you look in /mnt/cm/paster.log on the instance, are there any indications as to what went wrong? It should be toward the top of the log after the server gets started. SGE gets installed each time an instance is rebooted so simply rebooting it again may do the trick. You can also chose to manually remove/clean SGE before rebooting. To do so, you can follow the basic approach captured in this method: https://bitbucket.org/galaxy/cloudman/src/862d1087080f/cm/services/apps/sge.py#cl-26 Enis On Sun, Sep 11, 2011 at 12:05 AM, Joseph Hargitai joseph.hargi...@einstein.yu.edu wrote: Hi, Upon restarting a saved cloud instance I am missing: -bash: /opt/sge/default/common/settings.sh: No such file or directory -bash: /opt/sge/default/common/settings.sh: No such file or directory all the other mounts are there and well preserved. Is this pulled from a special place i may have not saved? The instance now does not boot beyond this point. Have login and admin console access. joe ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] ProFTPd on Ubuntu system.
I forgot to mention earlier that you'll still need to do the database GRANT etc. as described on the wiki page you pointed to. The mentioned method does not do that. Enis On Tue, Sep 13, 2011 at 8:10 AM, Enis Afgan eaf...@emory.edu wrote: You can try extracting the ProFTP install method from mi-deployment and running only it via Fabric? We use this method to install ProFTP on the cloud and galaxy VM, both of which are based on Ubuntu 10.04. The method is available here: https://bitbucket.org/afgane/mi-deployment/src/d894af37a83b/mi_fabfile.py#cl-442 (also take a look at the _required_packages method and _setup_users method because some additional configuration is done there). Hope this helps, Enis On Mon, Sep 12, 2011 at 8:05 PM, Luobin Yang yangl...@isu.edu wrote: Hi, Has anyone set up ProFTPd successfully on Ubuntu 10.04 to enable FTP upload on Galaxy? I followed the instructions on this link ( http://wiki.g2.bx.psu.edu/Admin/Config/Upload%20via%20FTP) but it doesn't work. Thanks, Luobin ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] installing software for Galaxy
Hi Mattias, We use a combination of mi-deployment and Brad's cloudbiolinux script ( https://github.com/chapmanb/cloudbiolinux) to install tools (including the biolinux ones). It requires some manual effort but has worked for us. The approach you describe sounds pretty good for packaged tools, I would just suggest to maybe create a script to automate the process as you go about installing all of the tools. If you decide to create the script, please conside sharing it. Good luck, Enis On Mon, Sep 5, 2011 at 2:22 PM, Mattias de Hollander m.dehollan...@nioo.knaw.nl wrote: Hello Galaxy-developers and community, I would like to ask your advice on installing software. I used the cloudman scripts to install galaxy on a multi-core server. I use the fabric scripts from cloudman/mi-diployment to install some software but I am also looking at repositories like biolinux and the NBIC RPM repository. There has been a thread about that over here: http://gmod.827538.n3.nabble.com/RPM-repository-for-NGS-tools-in-Galaxy-tp2617820p2617820.html As James says in that thread I also prefer to install software in isolated directories to keep track of different version. Did someone use the biolinux repository to install software in a galaxy accepted path: $GALAXY_APPS/package/version/? How does the Galaxy Cloud team does that? (Enis?) I was thinking to use 'sudo apt-get --download-only' to first get the debs and then install them using 'dpkg --instdir' and then specifying the directory. Or is there a smarter way to do this? Any thoughts on this issue are appreciated. Thanks. -- Bioinformatician Netherlands Institute of Ecology (NIOO-KNAW) Wageningen, the Netherlands ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] mi-tools- tools_fabfile.py permissions requirements (local install)
Hi Joe, I just updated the code for the Galaxy installation process as part of mi-deployment to use the user defined via env.galaxy_user. There is still a (slightly guarded) chown called at the end of the script run due to the way other tools are being installed. We keep talking about rewriting this code entirely so it hasn't been generalized terribly much, which means that it is likely that some customizations for the current deployment will be desired. Overall, if you have suggestions and/or would like to fork the project and issue pull requests with code changes, please do, I'd appreciate all of that a lot. Let me know if this change helps. Enis On Mon, Aug 1, 2011 at 2:44 PM, Joseph Hargitai joseph.hargi...@einstein.yu.edu wrote: Cool, one question remains: since the cloud instance works and installs - there needs to be a script that is fully functional - a - installs dependencies b - installs tools c - creates indexes d - points tools to indexes j -- *From:* Enis Afgan [eaf...@emory.edu] *Sent:* Monday, August 01, 2011 12:54 PM *To:* Joseph Hargitai *Cc:* Galaxy Dev *Subject:* Re: [galaxy-user] mi-tools- tools_fabfile.py permissions requirements (local install) Well, you can give it a shot like that and see if it still works. I quit using (and thus updating) the R-installation methods a while ago now so not sure if the process was changed. There is high likelihood that R will work (you should probably update the version number in the method itself). rpy however, will probably not work - rpy was very hard to install to begin with and that is why we moved away from this approach. Alternatively, you can add packages I listed in my last email to the list of packages under _required_packages method. Enis On Mon, Aug 1, 2011 at 12:49 PM, Joseph Hargitai joseph.hargi...@einstein.yu.edu wrote: Hi, so for local install i can uncomment the R and rpy from your script as far as at the _install_R.. part? j -- *From:* Enis Afgan [eaf...@emory.edu] *Sent:* Monday, August 01, 2011 12:37 PM *To:* Joseph Hargitai *Cc:* Galaxy Dev *Subject:* Re: [galaxy-user] mi-tools- tools_fabfile.py permissions requirements (local install) Hi Joe, mi-deployment scripts are currently being used to create cloud images so this is the correct code your looking at. However, when it comes to R and rpy, those tools are (as you discovered) not being installed via those scripts; they are installed from packages but, in the case of Galaxy cloud images, they are inherited from the CloudBioLinux images that the Galaxy cloud images build on top of so not even included in the mi-deployment scripts. If you would like to follow suit and install those tools from packages, these are the ones (for debian): r-base, r-base-core, r-base-core-ra, r-base-dev, r-base-html, python-rpy As far as the permissions issues goes, I'll look (over the next couple of days) more closely at those and focus on consistency. Enis On Mon, Aug 1, 2011 at 11:04 AM, Peter Cock p.j.a.c...@googlemail.comwrote: Hi Jeseph, I see now you are talking about Enis' code for automated building of virtual machine Galaxy images, https://bitbucket.org/afgane/mi-deployment/src - mentioned briefly here: http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup I know very little about this area of Galaxy - we install any tools needed for Galaxy by hand (and try to document how we did it for future reference). On Mon, Aug 1, 2011 at 3:50 PM, Joseph Hargitai joseph.hargi...@einstein.yu.edu wrote: R - not the correct download url, Which URL are you talking about? The wiki page on dependencies http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Dependencies correctly links to the R project as http://www.r-project.org/ you are correct, however the tools_fabfile.py has version = 2.11.1 url = http://mira.sunsite.utk.edu/CRAN/src/base/R-2/R-%s.tar.gz; % version That was a CRAN mirror, which seems to have gone now. Not listed here: http://cran.r-project.org/mirrors.html rpy - will this be fixed? What needs to be fixed? Moving Galaxy to rpy2? https://bitbucket.org/galaxy/galaxy-central/issue/103/upgrade-rpy-to-latest-version again script has a comment: def _install_rpy(): # *Does not work in reality* Do you mean installing Galaxy without admin rights (i.e. without using sudo)? either way - but should work. The fab script for instance adds some jar files to the galaxy-dist structure - then more added via the same script as sudo, and when it wants to change permissions for the entire folder for the next application the script fails. Hence just curious what a smoother solution would be. In general - is the cloud install using the mi- script to install the tools? If so, where is it located? I'd like to compare it to the one i use for the local install. If it uses
Re: [galaxy-dev] [galaxy-user] mi-tools- tools_fabfile.py permissions requirements (local install)
Hi Joe, mi-deployment scripts are currently being used to create cloud images so this is the correct code your looking at. However, when it comes to R and rpy, those tools are (as you discovered) not being installed via those scripts; they are installed from packages but, in the case of Galaxy cloud images, they are inherited from the CloudBioLinux images that the Galaxy cloud images build on top of so not even included in the mi-deployment scripts. If you would like to follow suit and install those tools from packages, these are the ones (for debian): r-base, r-base-core, r-base-core-ra, r-base-dev, r-base-html, python-rpy As far as the permissions issues goes, I'll look (over the next couple of days) more closely at those and focus on consistency. Enis On Mon, Aug 1, 2011 at 11:04 AM, Peter Cock p.j.a.c...@googlemail.comwrote: Hi Jeseph, I see now you are talking about Enis' code for automated building of virtual machine Galaxy images, https://bitbucket.org/afgane/mi-deployment/src - mentioned briefly here: http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup I know very little about this area of Galaxy - we install any tools needed for Galaxy by hand (and try to document how we did it for future reference). On Mon, Aug 1, 2011 at 3:50 PM, Joseph Hargitai joseph.hargi...@einstein.yu.edu wrote: R - not the correct download url, Which URL are you talking about? The wiki page on dependencies http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Dependencies correctly links to the R project as http://www.r-project.org/ you are correct, however the tools_fabfile.py has version = 2.11.1 url = http://mira.sunsite.utk.edu/CRAN/src/base/R-2/R-%s.tar.gz; % version That was a CRAN mirror, which seems to have gone now. Not listed here: http://cran.r-project.org/mirrors.html rpy - will this be fixed? What needs to be fixed? Moving Galaxy to rpy2? https://bitbucket.org/galaxy/galaxy-central/issue/103/upgrade-rpy-to-latest-version again script has a comment: def _install_rpy(): # *Does not work in reality* Do you mean installing Galaxy without admin rights (i.e. without using sudo)? either way - but should work. The fab script for instance adds some jar files to the galaxy-dist structure - then more added via the same script as sudo, and when it wants to change permissions for the entire folder for the next application the script fails. Hence just curious what a smoother solution would be. In general - is the cloud install using the mi- script to install the tools? If so, where is it located? I'd like to compare it to the one i use for the local install. If it uses another script where/what is it? thanks, joe Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] mi-tools- tools_fabfile.py permissions requirements (local install)
Well, you can give it a shot like that and see if it still works. I quit using (and thus updating) the R-installation methods a while ago now so not sure if the process was changed. There is high likelihood that R will work (you should probably update the version number in the method itself). rpy however, will probably not work - rpy was very hard to install to begin with and that is why we moved away from this approach. Alternatively, you can add packages I listed in my last email to the list of packages under _required_packages method. Enis On Mon, Aug 1, 2011 at 12:49 PM, Joseph Hargitai joseph.hargi...@einstein.yu.edu wrote: Hi, so for local install i can uncomment the R and rpy from your script as far as at the _install_R.. part? j -- *From:* Enis Afgan [eaf...@emory.edu] *Sent:* Monday, August 01, 2011 12:37 PM *To:* Joseph Hargitai *Cc:* Galaxy Dev *Subject:* Re: [galaxy-user] mi-tools- tools_fabfile.py permissions requirements (local install) Hi Joe, mi-deployment scripts are currently being used to create cloud images so this is the correct code your looking at. However, when it comes to R and rpy, those tools are (as you discovered) not being installed via those scripts; they are installed from packages but, in the case of Galaxy cloud images, they are inherited from the CloudBioLinux images that the Galaxy cloud images build on top of so not even included in the mi-deployment scripts. If you would like to follow suit and install those tools from packages, these are the ones (for debian): r-base, r-base-core, r-base-core-ra, r-base-dev, r-base-html, python-rpy As far as the permissions issues goes, I'll look (over the next couple of days) more closely at those and focus on consistency. Enis On Mon, Aug 1, 2011 at 11:04 AM, Peter Cock p.j.a.c...@googlemail.comwrote: Hi Jeseph, I see now you are talking about Enis' code for automated building of virtual machine Galaxy images, https://bitbucket.org/afgane/mi-deployment/src - mentioned briefly here: http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup I know very little about this area of Galaxy - we install any tools needed for Galaxy by hand (and try to document how we did it for future reference). On Mon, Aug 1, 2011 at 3:50 PM, Joseph Hargitai joseph.hargi...@einstein.yu.edu wrote: R - not the correct download url, Which URL are you talking about? The wiki page on dependencies http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Dependencies correctly links to the R project as http://www.r-project.org/ you are correct, however the tools_fabfile.py has version = 2.11.1 url = http://mira.sunsite.utk.edu/CRAN/src/base/R-2/R-%s.tar.gz; % version That was a CRAN mirror, which seems to have gone now. Not listed here: http://cran.r-project.org/mirrors.html rpy - will this be fixed? What needs to be fixed? Moving Galaxy to rpy2? https://bitbucket.org/galaxy/galaxy-central/issue/103/upgrade-rpy-to-latest-version again script has a comment: def _install_rpy(): # *Does not work in reality* Do you mean installing Galaxy without admin rights (i.e. without using sudo)? either way - but should work. The fab script for instance adds some jar files to the galaxy-dist structure - then more added via the same script as sudo, and when it wants to change permissions for the entire folder for the next application the script fails. Hence just curious what a smoother solution would be. In general - is the cloud install using the mi- script to install the tools? If so, where is it located? I'd like to compare it to the one i use for the local install. If it uses another script where/what is it? thanks, joe Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Installing tools with dependencies using mi-deployment scripts
Each use of fabric's `run` method is independent so they cannot be chained;
you would either need to prepend the export command to the command that
depends on the fact PYTHONPATH is set (e.g., run(export PYTHONPATH=...;
command that requires given PP) or (better approach) create an env.sh
file in the Qiime's install dir so that it includes the required 'export
PYTHONPATH=...'. When Galaxy invokes Qiime, it will automatically source
that env.sh (assuming you have config variable `tool_dependency_dir =
/mnt/galaxyTools/tools` set in Galaxy's universe_wsgi.ini).
Enis
On Fri, Jul 15, 2011 at 12:19 PM, Mattias de Hollander
m.dehollan...@nioo.knaw.nl wrote:
Hello Enis,
I am trying to add PyCogent to the PYTHONPATH, but it seems the Fabric
run command does not recognize it.
I tried: run(export
PYTHONPATH=/mnt/galaxyTools/tools/PyCogent/1.5.1/lib/python2.6/site-packages/:$PYTHONPATH)
and then use cmd_install() to install Galaxy, but I keep getting the
message that PyCogent is not installed.
The normal strategy to use sys.path will not work because that will
adjust the PYTHONPATH on my local machine, not in the image if I am
correct.
Do you know a way to set the PYTHONPATH for Fabric' run() command?
Thanks,
Mattias
On Thu, 2011-07-14 at 09:05 +, Enis Afgan wrote:
Hi Mattias,
PyCogent is not currently being installed as part of mi-deployment, so
the method for installing it should be added (if you do so, please
consider issuing a pull request on bitbucket so it can be added to the
project).
When it comes to composing $PYTHONPATH to include the reference to
PyCogent via Qiime's env.sh - this should be done as part of the Qiime
installation method in mi-deployment (something like export
PYHTONPATH=path where PyCogent was installed; export PATH=...).
Sourcing of the tool's env.sh script is prepended to the execution
command when running the tool so the appropriate environment settings
will be loaded before the tool is run.
Enis
On Wed, Jul 13, 2011 at 12:09 PM, Mattias de Hollander
m.dehollan...@nioo.knaw.nl wrote:
Hello,
I am trying to add tools (PyCogent, Qiime) to a Cloudman image
using the
mi-deployment scripts. Qiime depends on PyCogent and it needs
to be
accessible on the python path, otherwise I get an error:
PyCogent not installed but required. (Is it installed? Is it
in the
current user's $PYTHONPATH or site-packages?)
Is there a way to add the paths in env.sh to the sys.path
during
installation using tools_fabfile.py?
All the tools are in /mnt/galaxyTools/tools.
I tried the DependencyManager from galaxy but then I only
receive the
path to the env.sh file:
In [43]: dependency_manager.find_dep('blast')
Out[43]:
('/mnt/galaxyTools/tools/blast/2.2.25/env.sh',
'/mnt/galaxyTools/tools/blast/2.2.25',
'2.2.25')
Is there a smarter way to add files to the PYTHONPATH during
the
execution of tools_fabfile.py?
Thanks,
Mattias
--
Bioinformatician
Netherlands Institute of Ecology (NIOO-KNAW)
Wageningen, the Netherlands
--
Bioinformatician
Netherlands Institute of Ecology (NIOO-KNAW)
Wageningen, the Netherlands
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Installing tools with dependencies using mi-deployment scripts
Hi Mattias,
PyCogent is not currently being installed as part of mi-deployment, so the
method for installing it should be added (if you do so, please consider
issuing a pull request on bitbucket so it can be added to the project).
When it comes to composing $PYTHONPATH to include the reference to PyCogent
via Qiime's env.sh - this should be done as part of the Qiime installation
method in mi-deployment (something like export PYHTONPATH=path where
PyCogent was installed; export PATH=...).
Sourcing of the tool's env.sh script is prepended to the execution command
when running the tool so the appropriate environment settings will be loaded
before the tool is run.
Enis
On Wed, Jul 13, 2011 at 12:09 PM, Mattias de Hollander
m.dehollan...@nioo.knaw.nl wrote:
Hello,
I am trying to add tools (PyCogent, Qiime) to a Cloudman image using the
mi-deployment scripts. Qiime depends on PyCogent and it needs to be
accessible on the python path, otherwise I get an error:
PyCogent not installed but required. (Is it installed? Is it in the
current user's $PYTHONPATH or site-packages?)
Is there a way to add the paths in env.sh to the sys.path during
installation using tools_fabfile.py?
All the tools are in /mnt/galaxyTools/tools.
I tried the DependencyManager from galaxy but then I only receive the
path to the env.sh file:
In [43]: dependency_manager.find_dep('blast')
Out[43]:
('/mnt/galaxyTools/tools/blast/2.2.25/env.sh',
'/mnt/galaxyTools/tools/blast/2.2.25',
'2.2.25')
Is there a smarter way to add files to the PYTHONPATH during the
execution of tools_fabfile.py?
Thanks,
Mattias
--
Bioinformatician
Netherlands Institute of Ecology (NIOO-KNAW)
Wageningen, the Netherlands
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] changing file storage location
Hi Ravi, Just a quick check - did you restart Galaxy after saving the changes? What you did should direct any new files to the specified location. Enis On Wed, Jun 29, 2011 at 10:43 AM, Sanka, Ravi rsa...@jcvi.org wrote: Greetings, ** ** My name is Ravi Sanka. I have a question regarding Galaxy. Recently, I had a local installation set up and am now trying to change the location where the install stores imported and created files. ** ** I opened the config file universe_wsgi.ini and did the following: ** ** **- **Removed the ‘#’ mark from ‘#file_path = database/files’ **- **Changed value of file_path to the absolute path of an accessible, readable/writable location **- **Saved changes ** ** Despite this, the install still stores files in database/files. Is there a step I’m missing? Does the setup procedure (setup.sh run.sh) need to be run again? ** ** I also intend to change the database from the SQLite default to an existing database, so I assume the steps to change the file_path also apply to database_connection. ** ** Thank you for your time. ** ** --- Ravi Sanka ICS - Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 --- ** ** ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] changing file storage location
In order for Galaxy to acknowledge the changes, you'll have to restart Galaxy. I'll double check in the wiki to make sure this is stated somewhere but simply stopping the Galaxy process and starting it back up should reload the config file. If your Galaxy is running the background run: sh run.sh --stop-daemon sh run.sh --daemon If the process is not running in the background, simply stop it (i.e., ctrl + C) Enis On Wed, Jun 29, 2011 at 11:52 AM, Sanka, Ravi rsa...@jcvi.org wrote: Hi Enis, ** ** No I didn’t restart Galaxy after the changes were saved. Is that the process explained in the README.txt (setup.sh run.sh)? That process was for the first time, so I didn’t repeat it after saving the changes. ** ** --- Ravi Sanka ICS - Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 --- ** ** *From:* Enis Afgan [mailto:eaf...@emory.edu] *Sent:* Wednesday, June 29, 2011 11:32 AM *To:* Sanka, Ravi *Cc:* galaxy-dev@lists.bx.psu.edu *Subject:* Re: [galaxy-dev] changing file storage location ** ** Hi Ravi, Just a quick check - did you restart Galaxy after saving the changes? What you did should direct any new files to the specified location. ** ** Enis On Wed, Jun 29, 2011 at 10:43 AM, Sanka, Ravi rsa...@jcvi.org wrote: Greetings, My name is Ravi Sanka. I have a question regarding Galaxy. Recently, I had a local installation set up and am now trying to change the location where the install stores imported and created files. I opened the config file universe_wsgi.ini and did the following: - Removed the ‘#’ mark from ‘#file_path = database/files’ - Changed value of file_path to the absolute path of an accessible, readable/writable location - Saved changes Despite this, the install still stores files in database/files. Is there a step I’m missing? Does the setup procedure (setup.sh run.sh) need to be run again? I also intend to change the database from the SQLite default to an existing database, so I assume the steps to change the file_path also apply to database_connection. Thank you for your time. --- Ravi Sanka ICS - Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 --- ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ** ** ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] changing file storage location
Almost, mysql requires two forward slashes, like so: database_connection = mysql://galaxy:password_here@ host/galaxy?unix_socket=/var/run/mysqld/mysqld.sock NTW, unless you are not absolutely set on MySQL, the Galaxy team prefers PostrgeSQL as the backend database. Enis On Wed, Jun 29, 2011 at 4:03 PM, Sanka, Ravi rsa...@jcvi.org wrote: Hello Enis, ** ** Thank you. Galaxy is now directing the files to the new storage location.* *** ** ** On a related note, I am also planning to switch Galaxy from the default SQLite to an existing MYSQL database. The documentation says to do this, the database_connection field in the config file must be given the proper URL. This is the example it gave: ** ** database_connection = postgres:///db_name?user=user_namepassword=your_pass ** ** Would the same value work if postgres was switched out with mysql? ** ** --- Ravi Sanka ICS - Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 --- ** ** *From:* Enis Afgan [mailto:eaf...@emory.edu] *Sent:* Wednesday, June 29, 2011 12:02 PM *To:* Sanka, Ravi *Cc:* galaxy-dev@lists.bx.psu.edu *Subject:* Re: [galaxy-dev] changing file storage location ** ** In order for Galaxy to acknowledge the changes, you'll have to restart Galaxy. I'll double check in the wiki to make sure this is stated somewhere but simply stopping the Galaxy process and starting it back up should reload the config file. ** ** If your Galaxy is running the background run: sh run.sh --stop-daemon sh run.sh --daemon ** ** If the process is not running in the background, simply stop it (i.e., ctrl + C) ** ** Enis On Wed, Jun 29, 2011 at 11:52 AM, Sanka, Ravi rsa...@jcvi.org wrote: Hi Enis, No I didn’t restart Galaxy after the changes were saved. Is that the process explained in the README.txt (setup.sh run.sh)? That process was for the first time, so I didn’t repeat it after saving the changes. --- Ravi Sanka ICS - Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 --- *From:* Enis Afgan [mailto:eaf...@emory.edu] *Sent:* Wednesday, June 29, 2011 11:32 AM *To:* Sanka, Ravi *Cc:* galaxy-dev@lists.bx.psu.edu *Subject:* Re: [galaxy-dev] changing file storage location Hi Ravi, Just a quick check - did you restart Galaxy after saving the changes? What you did should direct any new files to the specified location. Enis On Wed, Jun 29, 2011 at 10:43 AM, Sanka, Ravi rsa...@jcvi.org wrote: Greetings, My name is Ravi Sanka. I have a question regarding Galaxy. Recently, I had a local installation set up and am now trying to change the location where the install stores imported and created files. I opened the config file universe_wsgi.ini and did the following: - Removed the ‘#’ mark from ‘#file_path = database/files’ - Changed value of file_path to the absolute path of an accessible, readable/writable location - Saved changes Despite this, the install still stores files in database/files. Is there a step I’m missing? Does the setup procedure (setup.sh run.sh) need to be run again? I also intend to change the database from the SQLite default to an existing database, so I assume the steps to change the file_path also apply to database_connection. Thank you for your time. --- Ravi Sanka ICS - Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 --- ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ** ** ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] thanks for the conference; presentations availbale?
A wiki page is available here: http://wiki.g2.bx.psu.edu/GCC2011 On Mon, May 30, 2011 at 4:20 AM, Martin Senger martin.sen...@gmail.comwrote: I really enjoyed the recent conference. Many thanks for the splendid logistics and, of course, the content. I wonder if the slides presented there will be available for downloading perhaps? Cheers, Martin -- Martin Senger email: martin.sen...@gmail.com,martin.sen...@kaust.edu.sa skype: martinsenger ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Using a Galaxy image on a local system
Hi Clare, Once you have a VM setup and accessible via ssh, you can also use our scripts for automated configuration and deployment of dependencies and tools. These scripts are used to setup Galaxy Cloud and they're targeted for Ubuntu 10.04 but should be applicable to other distributions as well. The scripts are available here: https://bitbucket.org/afgane/mi-deployment/overview Good luck, Enis On Thu, Apr 28, 2011 at 3:16 AM, Clare Sloggett s...@unimelb.edu.au wrote: Hi all, I would like to set up Galaxy locally. At the moment, I'm just trying to use it on my desktop (a Mac, OSX 10.6.7), but later we will want a local server to play with. Rather than install galaxy and then install all the tools it can use (and deal with OSX issues for some of them), it seems simpler just to use a virtual machine, since there are images which get regularly updated and come with pretty much everything. Is there anything wrong with this approach? I know there are Amazon EC2 images for Galaxy. So far as I know there are not other kinds of images? So for using it on my desktop, I think my options are either to run an EC2-compatible system locally, or to try to convert the AMI to a VMWare or VirtualBox image. I was just wondering if anyone has already tried either of these approaches? Also, is it possible to get hold of the galaxy AMI files themselves? Any advice welcome! Thanks, Clare ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] setup.sh missing
Hi Curtis, setup.sh has been removed from the installation process, as discussed in this thread: http://lists.bx.psu.edu/pipermail/galaxy-dev/2011-March/004588.html On Wed, Mar 9, 2011 at 2:12 PM, Curt Palm cp...@stanford.edu wrote: I just cloned the current release version of galaxy: hg clone http://bitbucket.org/galaxy/galaxy-dist and can not run setup.sh because it is not present in the /galaxy-dist/ directory. setup.sh is also not listed on the https://bitbucket.org/galaxy/galaxy-dist/src page, is this an error or has the installation process changed? thanks *** Curtis J. Palm cp...@stanford.edu Stanford Genome Technology Center MC: 8307 office: 650-812-1994cell: 408 858-7849 *** ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Amazon cloud formation for galaxy
Yeah, also that is is pretty much how CloudMan handles the infrastructure management now - it has it's own descriptor (i.e., template) and from there it handles all the provisioning and coordination of the underlying resources. Granted, having Amazon do it would add to the robustness of the overall system (especially because they seem to have also implemented a rollback option) but it would also tie Galaxy CloudMan to AWS even more. However, the intent is really to support other cloud infrastructure providers as more centers roll out their own cloud deployments... Enis On Fri, Feb 25, 2011 at 2:07 PM, Ry4an Brase ry4an+gal...@msi.umn.eduwrote: Today's release of amazon cloud formation has to make the Galaxy Cloud stuff a bit easier: http://aws.typepad.com/aws/2011/02/cloudformation-create-your-aws-stack-from-a-recipe.html In theory with a description file like this: https://s3.amazonaws.com/cloudformation-templates/CloudFormationSample_WordPress.template Once can define an entirely cluster to bring up from custom .amis with a single click. Exciting. -- Ry4an Brase 612-626-6575 Software Developer Application Development University of Minnesota Supercomputing Institutehttp://www.msi.umn.edu ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
