Re: [galaxy-dev] Galaxy on a Cluster -- Active Directory LDAP configuration
David, Thank you for the valuable feedback. I will look into the galaxy-pulsar server app and determine how we can implement it to connect our galaxy instance to the working nodes. Perhaps I’ll direct a few relevant questions while we get on board with the implementation, if you don’t mind.If you have any notes or lessons learned that you could share that would be very helpful to us. However, this idea seems already like a good solution for us.I’ll keep all my notes and setup a step by step instructions after we are done. Carlos. From: David Trudgian mailto:david.trudg...@utsouthwestern.edu>> Date: Tuesday, September 22, 2015 at 11:53 AM To: Nicola Soranzo mailto:nsora...@tiscali.it>>, Carlos Lijeron mailto:clije...@hunter.cuny.edu>>, "galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>" mailto:galaxy-dev@lists.galaxyproject.org>> Subject: RE: [galaxy-dev] Galaxy on a Cluster -- Active Directory LDAP configuration Hi Carlos, We aren’t using Active Directory here – but we do use an OpenLDAP directory which our cluster authenticates against, as does Galaxy. In order to track cluster usage for users running Galaxy jobs we use galaxy-pulsar between Galaxy itself and our SLURM cluster. Pulsar is configured to submit all jobs as a real user on the cluster, via SLURM-DRMAA. This means that all of the Galaxy usage on the cluster appears in our SLURM accounting database just like any other job would. The complexity here is file ownership. Pulsar has to copy all input files into a staging directory, and change ownership to the real user for the job to run on the cluster. It is/was(?) a little complex to setup, as there are more parts involved then a typical Galaxy install, but it works great for us here. When I was getting this setup John Chilton mentioned adding a PulsarEmbedded runner into Galaxy at some point. Not sure whether this has/is happening, but it’d make these situations easier: https://trello.com/c/4YwVZBtq/1865-embedded-pulsar-job-runner DT -- David Trudgian Ph.D. Computational Scientist, BioHPC UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 From: galaxy-dev [mailto:galaxy-dev-boun...@lists.galaxyproject.org] On Behalf Of Nicola Soranzo Sent: Tuesday, September 22, 2015 5:59 AM To: Carlos Lijeron mailto:clije...@hunter.cuny.edu>>; galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] Galaxy on a Cluster -- Active Directory LDAP configuration Hi Carlos, we are using Active Directory LDAP for user authentication, which works pretty well, but only the Galaxy user to submit jobs to the LSF cluster queue, so I can't help with the resource tracking. Cheers, Nicola On 21/09/15 20:19, Carlos Lijeron wrote: Everyone, We are setting up Galaxy to work with our cluster and SLURM as the work manager. The cluster itself authenticates to our local Active Directory, so I’m wondering if the best way to track resource utilization of Galaxy users on the cluster is to also have Galaxy authenticate to the same Active Directory LDAP. Is anyone on this list using the same configuration and tracking resource utilization from Galaxy users submitting jobs to the cluster nodes? Please advise. Thank you all ! Carlos. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy on a Cluster -- Active Directory LDAP configuration
Good day Nicola, Thank you for the advise on LDAP authentication. Did you need to run Galaxy on Apache to enable this, or did you simply change some configuration files to enable this. I think we’ll be able to write some scripts using SLURM commands to pull data about resource utilization, but my main concern is whether we need Apache to enable LDAP authentication of galaxy users. Thanks again ! Carlos From: Nicola Soranzo mailto:nicola.sora...@gmail.com>> on behalf of Nicola Soranzo mailto:nsora...@tiscali.it>> Date: Tuesday, September 22, 2015 at 6:59 AM To: Carlos Lijeron mailto:clije...@hunter.cuny.edu>>, "galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>" mailto:galaxy-dev@lists.galaxyproject.org>> Subject: Re: [galaxy-dev] Galaxy on a Cluster -- Active Directory LDAP configuration Hi Carlos, we are using Active Directory LDAP for user authentication, which works pretty well, but only the Galaxy user to submit jobs to the LSF cluster queue, so I can't help with the resource tracking. Cheers, Nicola On 21/09/15 20:19, Carlos Lijeron wrote: Everyone, We are setting up Galaxy to work with our cluster and SLURM as the work manager. The cluster itself authenticates to our local Active Directory, so I’m wondering if the best way to track resource utilization of Galaxy users on the cluster is to also have Galaxy authenticate to the same Active Directory LDAP. Is anyone on this list using the same configuration and tracking resource utilization from Galaxy users submitting jobs to the cluster nodes? Please advise. Thank you all ! Carlos. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Galaxy on a Cluster -- Active Directory LDAP configuration
Everyone, We are setting up Galaxy to work with our cluster and SLURM as the work manager. The cluster itself authenticates to our local Active Directory, so I’m wondering if the best way to track resource utilization of Galaxy users on the cluster is to also have Galaxy authenticate to the same Active Directory LDAP. Is anyone on this list using the same configuration and tracking resource utilization from Galaxy users submitting jobs to the cluster nodes? Please advise. Thank you all ! Carlos. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Running Galaxy on a cluster with SLURM ?
Hello everyone, I¹m setting up Galaxy to run on our cluster which uses SLURM as the job manager. I am wondering if any of you have any expertise in this area, ideas to share or lessons learned. I carried out the following steps listed below, but I can¹t make the final connection just yet. Any pointers will be greatly appreciated. 1. Setup Galaxy to run on the head node 2. Setup MySQL and it¹s running well with Galaxy 3. Setup the drmaa-python library inside our Galaxy folder Main questions: What would be the next step to take? Where to setup Galaxy to use an exclusive pool of resources (called partition in SLURM). Thank you all ! Carlos. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy on CentOS?
Thank you Peter, I am now getting a clear idea on how our deployment will go. Yes, I followed up with David and he provided additional insights into his deployment. What a great virtual team !!! Thank you. Carlos On 5/5/15, 4:12 AM, "Peter Cock" wrote: >Hi Carlos, > >We're running Galaxy on CentOS 6.6, so that in itself shouldn't be a >problem. Most of the effort was sorting out shared storage with the >cluster, which in our case is managed with SGE (fairly commonly >used with Galaxy). > >In reply to your thread last month David Trudgian said he was using >Bright Cluster Manager: > >http://dev.list.galaxyproject.org/Galaxy-on-HPC-and-Bright-Cluster-Manager >-tc4667015.html > >Regards, > >Peter > > >On Tue, May 5, 2015 at 2:51 AM, Carlos Lijeron >wrote: >> Hello everyone, >> >> We are soon acquiring our cluster which has CentOS on the master node >>along >> with Bright Cluster Manager and workload manager. We are wondering if >> there is a guideline on how to install and support Galaxy on CentOS. >>Does >> anyone have any info on this? >> >> Thanks ! >> >> Carlos. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy on CentOS?
Thank you for the feedback Zuzanna, Would you have documentation on scripts your updated or any lessons learned during your deployment? I¹m trying to get as much feedback as possible to hit the ground running when the cluster is delivered in our facility. One more question, do you use SGI or SLURM for the workload manager? Thanks again ! Carlos. On 5/5/15, 9:41 AM, "Zuzanna K. Filutowska" wrote: >>wondering if there is a guideline on how to install and support Galaxy >>on CentOS. Does anyone have any info on this? > >I am running Galaxy on Centos 6.6. But we are considering to move to >Fedora Server. Of course it is possible to run Galaxy on Centos, however >I had to compile python and several other packets to make all tools to >work. > > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Galaxy on CentOS?
Hello everyone, We are soon acquiring our cluster which has CentOS on the master node along with Bright Cluster Manager and workload manager. We are wondering if there is a guideline on how to install and support Galaxy on CentOS. Does anyone have any info on this? Thanks ! Carlos. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Galaxy on HPC and Bright Cluster Manager?
Good day everyone, Has anyone of you been able to implement Galaxy on a HPC using Bright Cluster Manager as the main DRM? I noticed that only a few have been known to work with Galaxy, but the list does not include Bright. Any advice/ideas will be greatly appreciated. TORQUE Resource Manager PBS Professional Open Grid Engine Univa Grid Engine (previously known as Sun Grid Engine and Oracle Grid Engine) Platform LSF HTCondor Slurm Galaxy Pulsar (formerly LWR) Thanks. Carlo ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Cloning state when installing tools.
Everyone, I've been encountering problems when installing particular tools in a local Galaxy instance. One such tool is Cummerbund as you can see in link below: https://www.dropbox.com/s/bmg4qcci97ayq2k/Cloning%20State.JPG?dl=0 Would you have a suggestion on how to troubleshoot failed tool installations that do not give an output error message? Thank you all ! Carlos. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Difference between hg pull -u (and) hg update stable?
Hello all, Is there any major difference between the commands hg pull -u (and) hg update stable when trying to update an existing local Galaxy installation? I tried running hg pull -u as suggested in the IRC channel, but nothing was updated. Then I ran the hg update stable and got this result: 324 files updated, 0 files merged, 86 files removed, 0 files unresolved I'm not clear about which one should be performed every once in a while. Any inputs or suggestions will be appreciated. Thanks. Carlos. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
