Re: [galaxy-dev] Tool shed tools, manual dep installation
Nicola, Thanks! I completely missed that. That’s very useful to know. Martin – yes a PR is on the to-do list, but unfortunately it’s a low-ish priority as the wish is to be using system / module deps. Have an impending vacation, so am having to get through a lot of stuff right now. Hopefully eventually we can get back to tool shed deps. DT -- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 From: Nicola Soranzo [mailto:nicola.sora...@gmail.com] On Behalf Of Nicola Soranzo Sent: Wednesday, April 13, 2016 11:03 AM To: Martin Čech <mar...@bx.psu.edu>; David Trudgian <david.trudg...@utsouthwestern.edu>; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Tool shed tools, manual dep installation Hi David, in the server I am administering I don't use the Tool Shed dependency resolver and is not much work if you are used to environment modules. For more information, see: https://docs.galaxyproject.org/en/master/admin/dependency_resolvers.html Cheers, Nicola On 13/04/16 16:42, Martin Čech wrote: David, You can have multiple versions of tool 'installed' manually (i.e. without shed). That said, there is an option of working with the IUC to make the wanted packages to be up-to-date and secure. i am pretty sure any PRs this direction will get merged. M. On Wed, Apr 13, 2016 at 11:39 AM David Trudgian <david.trudg...@utsouthwestern.edu<mailto:david.trudg...@utsouthwestern.edu>> wrote: Hi Martin, Thanks for the input - yes there is going to be maintenance headache either way, private tool-shed or not. If we don't go with our own tool-shed though, am I right in thinking we lose versioning for tools? I.E. installing manually into 'tools' and tool_conf.xml I can't use tags like the tool shed stuff does in shed_tool.conf. The infosec concern is mainly with libraries which are provided by tool-shed dependency package installs, not the main software package the tool wrapper is calling. With a tool shed it's feasible then we could keep older versions of tools/wrappers for reproducibility, rebuilding them against updated (but compatible) libraries when there's a security issue, perhaps? Glad to know that my idea of the workflow isn't totally wrong though. Thanks, DT -- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 From: Martin Čech [mailto:mar...@bx.psu.edu<mailto:mar...@bx.psu.edu>] Sent: Wednesday, April 13, 2016 8:47 AM To: David Trudgian <david.trudg...@utsouthwestern.edu><mailto:david.trudg...@utsouthwestern.edu>; galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] Tool shed tools, manual dep installation Hi David, one thing that you might be missing is that the wrapper for the infosec-approved version might not exist yet (you would have to adapt the older shed-wrapped version if there were any relevant changes). But this largely depends on what software you want to use. Besides that your flow seems quite accurate. I would probably argue against own tool shed, as the maintenance overhead will probably not be worth it for you. Let us know if we can help in any way, thank you for using Galaxy. Martin On Tue, Apr 12, 2016 at 5:42 PM David Trudgian <david.trudg...@utsouthwestern.edu<mailto:david.trudg...@utsouthwestern.edu>> wrote: Hi All, I’m currently caught between the requests of our Galaxy users to install things from main tool-shed, and our information security dept’s concerns r.e. the automated installation of tool-deps on our systems. Due to restrictive web access policies for servers here our galaxy server can’t access SourceForge, where many tool-dep downloads are. A request to unblock this for a particular tool-dep package led to our infosec justifiably raising concerns r.e. a tool-dep package that is quite out of date (details sent off list previously). We’re now currently unable to install tool-shed tools that users have requested. The current proposal from our infosec dept is to get all our deps from system repos etc. However the way I’m aware of implementing this for tool-shed tools, which need to run across our cluster, would be something pretty arduous like: * Clone the tool from the upstream toolshed repo * Edit the tool code to remove the package requirements * Identify and install all the requirements on the cluster as system pkgs / environment modules – with attention to versions so things work as expected * Edit the tool code so it knows to load the right environment modules / set right PATH when it runs * Install the tool into our galaxy ‘tools’ dir , not the ‘shed_tools’ * Manually add the tool to galaxy’s tool_conf.xml.main * Schedule downtime to restart galaxy * Test things out …
Re: [galaxy-dev] Tool shed tools, manual dep installation
Hi Martin, Thanks for the input - yes there is going to be maintenance headache either way, private tool-shed or not. If we don't go with our own tool-shed though, am I right in thinking we lose versioning for tools? I.E. installing manually into 'tools' and tool_conf.xml I can't use tags like the tool shed stuff does in shed_tool.conf. The infosec concern is mainly with libraries which are provided by tool-shed dependency package installs, not the main software package the tool wrapper is calling. With a tool shed it's feasible then we could keep older versions of tools/wrappers for reproducibility, rebuilding them against updated (but compatible) libraries when there's a security issue, perhaps? Glad to know that my idea of the workflow isn't totally wrong though. Thanks, DT -- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 From: Martin Čech [mailto:mar...@bx.psu.edu] Sent: Wednesday, April 13, 2016 8:47 AM To: David Trudgian <david.trudg...@utsouthwestern.edu>; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Tool shed tools, manual dep installation Hi David, one thing that you might be missing is that the wrapper for the infosec-approved version might not exist yet (you would have to adapt the older shed-wrapped version if there were any relevant changes). But this largely depends on what software you want to use. Besides that your flow seems quite accurate. I would probably argue against own tool shed, as the maintenance overhead will probably not be worth it for you. Let us know if we can help in any way, thank you for using Galaxy. Martin On Tue, Apr 12, 2016 at 5:42 PM David Trudgian <david.trudg...@utsouthwestern.edu> wrote: Hi All, I’m currently caught between the requests of our Galaxy users to install things from main tool-shed, and our information security dept’s concerns r.e. the automated installation of tool-deps on our systems. Due to restrictive web access policies for servers here our galaxy server can’t access SourceForge, where many tool-dep downloads are. A request to unblock this for a particular tool-dep package led to our infosec justifiably raising concerns r.e. a tool-dep package that is quite out of date (details sent off list previously). We’re now currently unable to install tool-shed tools that users have requested. The current proposal from our infosec dept is to get all our deps from system repos etc. However the way I’m aware of implementing this for tool-shed tools, which need to run across our cluster, would be something pretty arduous like: * Clone the tool from the upstream toolshed repo * Edit the tool code to remove the package requirements * Identify and install all the requirements on the cluster as system pkgs / environment modules – with attention to versions so things work as expected * Edit the tool code so it knows to load the right environment modules / set right PATH when it runs * Install the tool into our galaxy ‘tools’ dir , not the ‘shed_tools’ * Manually add the tool to galaxy’s tool_conf.xml.main * Schedule downtime to restart galaxy * Test things out …. or we have to host our own tool shed, import tools we want from upstream, edit out the package requirements, provide the deps ourselves. These have all the headaches of merging things in when upstream shed-tools change. Just wondering if I’m missing anything? I know you can turn off ‘handle repository dependencies’ when installing a tool, but the tool still defines ‘requirements’ in its XML file and shows ‘Missing repository/tool dependencies’ in the Admin. Has anyone had any experience of dealing with this kind of situation? Many thanks! -- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool shed tools, manual dep installation
Hi Bjoern, Yes, we can reach stuff in the cargo-port. Unfortunately doing that isn’t a solution as the package that concerns our infosec is the same older version there in cargo-port as it has been trying to pull from SourceForge. Galaxy might be able to get it then, but I’m not permitted to install that package. Thanks, DT -- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 From: Bjoern Gruening [mailto:bjoern.gruen...@gmail.com] Sent: Wednesday, April 13, 2016 8:46 AM To: David Trudgian <david.trudg...@utsouthwestern.edu>; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Tool shed tools, manual dep installation Hi David, which sites are blocked? Can you access tools that take it's binaries from https://github.com/galaxyproject/cargo-port? If so Eric has ported all URLs over to cargo-port sometime ago. So we just need to update the packages for you. This only holds true for IUC packages and we probably need to update the toolshed repos for a few packages. An other solution would be to get a similar setup as your cluster running elsewhere install all tools on this computer and just move your installations over. This should work as well. Cheers, Bjoern On 12.04.2016 23:42, David Trudgian wrote: Hi All, I'm currently caught between the requests of our Galaxy users to install things from main tool-shed, and our information security dept's concerns r.e. the automated installation of tool-deps on our systems. Due to restrictive web access policies for servers here our galaxy server can't access SourceForge, where many tool-dep downloads are. A request to unblock this for a particular tool-dep package led to our infosec justifiably raising concerns r.e. a tool-dep package that is quite out of date (details sent off list previously). We're now currently unable to install tool-shed tools that users have requested. The current proposal from our infosec dept is to get all our deps from system repos etc. However the way I'm aware of implementing this for tool-shed tools, which need to run across our cluster, would be something pretty arduous like: * Clone the tool from the upstream toolshed repo * Edit the tool code to remove the package requirements * Identify and install all the requirements on the cluster as system pkgs / environment modules - with attention to versions so things work as expected * Edit the tool code so it knows to load the right environment modules / set right PATH when it runs * Install the tool into our galaxy 'tools' dir , not the 'shed_tools' * Manually add the tool to galaxy's tool_conf.xml.main * Schedule downtime to restart galaxy * Test things out or we have to host our own tool shed, import tools we want from upstream, edit out the package requirements, provide the deps ourselves. These have all the headaches of merging things in when upstream shed-tools change. Just wondering if I'm missing anything? I know you can turn off 'handle repository dependencies' when installing a tool, but the tool still defines 'requirements' in its XML file and shows 'Missing repository/tool dependencies' in the Admin. Has anyone had any experience of dealing with this kind of situation? Many thanks! -- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Tool shed tools, manual dep installation
Hi All, I'm currently caught between the requests of our Galaxy users to install things from main tool-shed, and our information security dept's concerns r.e. the automated installation of tool-deps on our systems. Due to restrictive web access policies for servers here our galaxy server can't access SourceForge, where many tool-dep downloads are. A request to unblock this for a particular tool-dep package led to our infosec justifiably raising concerns r.e. a tool-dep package that is quite out of date (details sent off list previously). We're now currently unable to install tool-shed tools that users have requested. The current proposal from our infosec dept is to get all our deps from system repos etc. However the way I'm aware of implementing this for tool-shed tools, which need to run across our cluster, would be something pretty arduous like: * Clone the tool from the upstream toolshed repo * Edit the tool code to remove the package requirements * Identify and install all the requirements on the cluster as system pkgs / environment modules - with attention to versions so things work as expected * Edit the tool code so it knows to load the right environment modules / set right PATH when it runs * Install the tool into our galaxy 'tools' dir , not the 'shed_tools' * Manually add the tool to galaxy's tool_conf.xml.main * Schedule downtime to restart galaxy * Test things out or we have to host our own tool shed, import tools we want from upstream, edit out the package requirements, provide the deps ourselves. These have all the headaches of merging things in when upstream shed-tools change. Just wondering if I'm missing anything? I know you can turn off 'handle repository dependencies' when installing a tool, but the tool still defines 'requirements' in its XML file and shows 'Missing repository/tool dependencies' in the Admin. Has anyone had any experience of dealing with this kind of situation? Many thanks! -- David Trudgian Ph.D. Computational Scientist, BioHPC Lyda Hill Department of Bioinformatics UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] galaxy folder tree permissions
Hi John, No need to apologise! This isn't really a priority for me as I can fix it with ACLs. Using a lot of those lately - so another few won't hurt :-) DT -Original Message- From: John Chilton [mailto:jmchil...@gmail.com] Sent: Tuesday, November 17, 2015 2:10 PM To: David Trudgian <david.trudg...@utsouthwestern.edu> Cc: lejeczek <pelj...@yahoo.co.uk>; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] galaxy folder tree permissions Sorry David for the long delay. Unfortunately I'm not aware of a setting to do this, I think each install process handles this on their own. This is a problem, perhaps the tool shed code should go through and ensure directory permissions are set correctly - maybe all user permissions should be applied to group and other? If this is still a priority I would create an issue for this - https://github.com/galaxyproject/galaxy/issues/new. -John On Tue, Jul 21, 2015 at 5:05 PM, David Trudgian <david.trudg...@utsouthwestern.edu> wrote: > Apologies John - hit reply instead of reply-all first time around... > > Maybe this is the right place to ask about the shed_tools and tool > deps directory permissions. Installing tools from the web I get a > mixed bag of folder permission, some at 775, some at 777 > > drwxrwxr-x 43 galaxy galaxy 4.0K Apr 9 16:44 devteam drwxrwxr-x 27 galaxy > galaxy 4.0K Jul 10 14:15 iuc > drwxrwxr-x 4 galaxy galaxy 60 Mar 27 11:24 lparsons > drwxrwxrwx 3 galaxy galaxy 28 Jun 19 09:42 ngsplot > drwxrwxrwx 3 galaxy galaxy 32 Apr 9 16:40 pjbriggs > > drwxrwxrwx 3 galaxy galaxy 24 Jun 23 16:41 readline > drwxrwxrwx 3 galaxy galaxy 27 Jul 10 14:10 rnastar > drwxrwxr-x 6 galaxy galaxy 72 Jun 23 16:40 samtools > drwxrwxrwx 3 galaxy galaxy 26 Jun 23 16:36 sqlite > > On a 'run as real user' setup all users need read/execute access so you can't > lock down the upper-level directory holding the tools and deps. Having write > open to anyone when a tool is installed is then pretty nasty as in theory > someone could maliciously modify something. > > Wondering if I'm missing some setting in Galaxy somewhere that would result > in 775 all the time for newly installed tools and their deps? > > -- > David Trudgian Ph.D. > Computational Scientist, BioHPC > UT Southwestern Medical Center > Dallas, TX 75390-9039 > Tel: (214) 648-4833 > > Please contact biohpc-help@utsouthwestern with general BioHPC inquries. > > -Original Message- > From: galaxy-dev [mailto:galaxy-dev-boun...@lists.galaxyproject.org] > On Behalf Of John Chilton > Sent: Monday, July 20, 2015 7:59 AM > To: lejeczek > Cc: galaxy-dev@lists.galaxyproject.org > Subject: Re: [galaxy-dev] galaxy folder tree permissions > > It would be best practice to do this. Nate is working on packaging > (.deb) and our Anisble setup to accomplish this - getting these permissions > exactly correct I think will be a big part of that effort. > > All of that said - if you were really going to pursue this but just install > and use the tool shed normally from the Galaxy webapp it seems kind of a > wasted effort. These dependencies would be installed as the Galaxy user and > run arbitrary code (from a sort of sys admin perspective). So if I were going > to go through this effort I would probably try to setup a separate > configuration and user for installing things from the tool shed and disable > the main Galaxy instance and user from doing this. That would add > considerably to this effort. > > Anyway - it is a best practice so I don't mean to discourage it - but > realistically I don't think many Galaxy deployments have gone through this > effort. > > -John > > > > > On Mon, Jul 20, 2015 at 1:37 PM, lejeczek <pelj...@yahoo.co.uk> wrote: >> hi everybody >> >> I'd like to ask if you think it's worthwhile is pursuing finely >> grained tree permissions? Would this improve security to leave out >> everything but only files/folders necessary for writing - to galaxy >> user what needs to write everything else root? >> Or just full perms to galaxy user on whole tree is the only way? >> >> many thanks. >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this and other >> Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > ___ > Please keep all replies on the list by using "reply all" > in y
Re: [galaxy-dev] FW: Galaxy on Centos via Apache - connection refused
SELinux policies are very strict on CentOS by default. Apache isn't allowed to access files outside of its standard directories, nor access network resources. Your local Galaxy apps server is a network resource - even though it's local. If you want to keep SELinux on then use audit2allow to see what policies will enable access: cat /var/log/audit/audit.log | audit2allow -v Then you can use setsebool (temporary) and setsebool -P (permanent) to enable. -- David Trudgian Ph.D. Computational Scientist, BioHPC UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 From: galaxy-dev [mailto:galaxy-dev-boun...@lists.galaxyproject.org] On Behalf Of Makis Ladoukakis Sent: Friday, October 23, 2015 10:30 AM To: Juan Carlos <jcsanch...@gmail.com> Cc: galaxy-...@lists.bx.psu.edu Subject: Re: [galaxy-dev] FW: Galaxy on Centos via Apache - connection refused Hello, That didn't work. The apache restart failed with the following error: SELinux is preventing /usr/sbin/httpd from name_bind access on the tcp_socket port 8081. Any idea why? Kind regards, Makis Subject: Re: [galaxy-dev] FW: Galaxy on Centos via Apache - connection refused From: jcsanch...@gmail.com<mailto:jcsanch...@gmail.com> Date: Tue, 20 Oct 2015 22:25:00 +1030 CC: galaxy-...@lists.bx.psu.edu<mailto:galaxy-...@lists.bx.psu.edu> To: makis4e...@hotmail.com<mailto:makis4e...@hotmail.com> Hi, If you have a line in your Apache conf like "Listen 80" change to "Listen 8081" On 20 Oct 2015, at 21:00, Makis Ladoukakis <makis4e...@hotmail.com<mailto:makis4e...@hotmail.com>> wrote: Hello, I am sorry but I have really no experience with setting up the Apache web server so I am not really sure how to do that. Can you please help me out with it? My apache configuration file is in /etc/httpd/conf/ directory and there are no directories such as /sites-available/ or /sites-enabled/ (as I would find in an ubuntu installation). What I did already (after some advice from the server admin) is open up the 8081 port like that: firewall-cmd --permanent --add-port=8081/tcp firewall-cmd --reload and then I got another error: [cgi:error] [pid 29603] [client 115.230.124.164:4559] script not found or unable to stat: /var/www/cgi-bin/common [autoindex:error] [pid 29716] [client 218.76.28.36:4468] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.php) found, and server-generated directory index forbidden by Options directive which I tried to solve by adding welcome.html as a recognizable filename in the apache configuration: DirectoryIndex index.html welcome.html but nothing worked and now the error_log shows the following: [Tue Oct 20 13:15:23.719295 2015] [mpm_prefork:notice] [pid 29598] AH00170: caught SIGWINCH, shutting down gracefully [Tue Oct 20 13:15:24.810684 2015] [core:notice] [pid 46896] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0 [Tue Oct 20 13:15:24.811647 2015] [suexec:notice] [pid 46896] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Tue Oct 20 13:15:24.846399 2015] [so:warn] [pid 46896] AH01574: module wsgi_module is already loaded, skipping [Tue Oct 20 13:15:24.847316 2015] [auth_digest:notice] [pid 46896] AH01757: generating secret for digest authentication ... [Tue Oct 20 13:15:24.848294 2015] [lbmethod_heartbeat:notice] [pid 46896] AH02282: No slotmem from mod_heartmonitor [Tue Oct 20 13:15:24.870033 2015] [mpm_prefork:notice] [pid 46896] AH00163: Apache/2.4.6 (CentOS) PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations [Tue Oct 20 13:15:24.870075 2015] [core:notice] [pid 46896] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' And the webpage that galaxy is supposed to appear is still blank. Any ideas? Thank you, Makis Date: Tue, 20 Oct 2015 11:01:44 +1030 Subject: Re: [galaxy-dev] FW: Galaxy on Centos via Apache - connection refused From: jcsanch...@gmail.com<mailto:jcsanch...@gmail.com> To: makis4e...@hotmail.com<mailto:makis4e...@hotmail.com> CC: galaxy-...@lists.bx.psu.edu<mailto:galaxy-...@lists.bx.psu.edu> hi, Maybe sounds silly, but have you tried to put the apache configuration in a virtual host within the sites-enable site? cheers jc On Tue, Oct 20, 2015 at 12:36 AM, Makis Ladoukakis <makis4e...@hotmail.com<mailto:makis4e...@hotmail.com>> wrote: Forwading to this list too. I am not sure if they are two separate lists. Makis From: makis4e...@hotmail.com<mailto:makis4e...@hotmail.com> To: galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org> Date: Mon, 19 Oct 2015 17:04:13 +0300 Subject: [galaxy-dev] Galaxy on Centos via Apache - connection refused Dear all, I've been trying to set up a Galaxy instance on my CentOS server but even when
Re: [galaxy-dev] Galaxy on a Cluster -- Active Directory LDAP configuration
Hi Carlos, We aren’t using Active Directory here – but we do use an OpenLDAP directory which our cluster authenticates against, as does Galaxy. In order to track cluster usage for users running Galaxy jobs we use galaxy-pulsar between Galaxy itself and our SLURM cluster. Pulsar is configured to submit all jobs as a real user on the cluster, via SLURM-DRMAA. This means that all of the Galaxy usage on the cluster appears in our SLURM accounting database just like any other job would. The complexity here is file ownership. Pulsar has to copy all input files into a staging directory, and change ownership to the real user for the job to run on the cluster. It is/was(?) a little complex to setup, as there are more parts involved then a typical Galaxy install, but it works great for us here. When I was getting this setup John Chilton mentioned adding a PulsarEmbedded runner into Galaxy at some point. Not sure whether this has/is happening, but it’d make these situations easier: https://trello.com/c/4YwVZBtq/1865-embedded-pulsar-job-runner DT -- David Trudgian Ph.D. Computational Scientist, BioHPC UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 From: galaxy-dev [mailto:galaxy-dev-boun...@lists.galaxyproject.org] On Behalf Of Nicola Soranzo Sent: Tuesday, September 22, 2015 5:59 AM To: Carlos Lijeron <clije...@hunter.cuny.edu>; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Galaxy on a Cluster -- Active Directory LDAP configuration Hi Carlos, we are using Active Directory LDAP for user authentication, which works pretty well, but only the Galaxy user to submit jobs to the LSF cluster queue, so I can't help with the resource tracking. Cheers, Nicola On 21/09/15 20:19, Carlos Lijeron wrote: Everyone, We are setting up Galaxy to work with our cluster and SLURM as the work manager. The cluster itself authenticates to our local Active Directory, so I’m wondering if the best way to track resource utilization of Galaxy users on the cluster is to also have Galaxy authenticate to the same Active Directory LDAP. Is anyone on this list using the same configuration and tracking resource utilization from Galaxy users submitting jobs to the cluster nodes? Please advise. Thank you all ! Carlos. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy Resource
https://wiki.galaxyproject.org/ToolShedApi From: galaxy-dev [mailto:galaxy-dev-boun...@lists.galaxyproject.org] On Behalf Of CHELSEA JU Sent: Thursday, July 30, 2015 1:32 PM To: galaxy-dev Subject: [galaxy-dev] Galaxy Resource Dear Galaxy Developer, As one of our data mining projects, we are trying to learn the metadata structure of different tool repositories. Since Galaxy provides a platform with a rich resource, we are wondering if there is an API service to retrieve all the tool information. So far, we have discovered that the list of tools can be found at https://toolshed.g2.bx.psu.edu/, and would like to know if there is an easier way to retrieve the information than crawling through the links in toolshed. Thank you, Chelsea UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] galaxy folder tree permissions
Apologies John - hit reply instead of reply-all first time around... Maybe this is the right place to ask about the shed_tools and tool deps directory permissions. Installing tools from the web I get a mixed bag of folder permission, some at 775, some at 777 drwxrwxr-x 43 galaxy galaxy 4.0K Apr 9 16:44 devteam drwxrwxr-x 27 galaxy galaxy 4.0K Jul 10 14:15 iuc drwxrwxr-x 4 galaxy galaxy 60 Mar 27 11:24 lparsons drwxrwxrwx 3 galaxy galaxy 28 Jun 19 09:42 ngsplot drwxrwxrwx 3 galaxy galaxy 32 Apr 9 16:40 pjbriggs drwxrwxrwx 3 galaxy galaxy 24 Jun 23 16:41 readline drwxrwxrwx 3 galaxy galaxy 27 Jul 10 14:10 rnastar drwxrwxr-x 6 galaxy galaxy 72 Jun 23 16:40 samtools drwxrwxrwx 3 galaxy galaxy 26 Jun 23 16:36 sqlite On a 'run as real user' setup all users need read/execute access so you can't lock down the upper-level directory holding the tools and deps. Having write open to anyone when a tool is installed is then pretty nasty as in theory someone could maliciously modify something. Wondering if I'm missing some setting in Galaxy somewhere that would result in 775 all the time for newly installed tools and their deps? -- David Trudgian Ph.D. Computational Scientist, BioHPC UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 Please contact biohpc-help@utsouthwestern with general BioHPC inquries. -Original Message- From: galaxy-dev [mailto:galaxy-dev-boun...@lists.galaxyproject.org] On Behalf Of John Chilton Sent: Monday, July 20, 2015 7:59 AM To: lejeczek Cc: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] galaxy folder tree permissions It would be best practice to do this. Nate is working on packaging (.deb) and our Anisble setup to accomplish this - getting these permissions exactly correct I think will be a big part of that effort. All of that said - if you were really going to pursue this but just install and use the tool shed normally from the Galaxy webapp it seems kind of a wasted effort. These dependencies would be installed as the Galaxy user and run arbitrary code (from a sort of sys admin perspective). So if I were going to go through this effort I would probably try to setup a separate configuration and user for installing things from the tool shed and disable the main Galaxy instance and user from doing this. That would add considerably to this effort. Anyway - it is a best practice so I don't mean to discourage it - but realistically I don't think many Galaxy deployments have gone through this effort. -John On Mon, Jul 20, 2015 at 1:37 PM, lejeczek pelj...@yahoo.co.uk wrote: hi everybody I'd like to ask if you think it's worthwhile is pursuing finely grained tree permissions? Would this improve security to leave out everything but only files/folders necessary for writing - to galaxy user what needs to write everything else root? Or just full perms to galaxy user on whole tree is the only way? many thanks. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Using 'display_servers' even if not REMOTE_USER a good idea?
Hi, Spent a little while today trying to hook our Galaxy up to our local UCSC Browser. There was various stuff on the web but is a bit confusing since the 'display_servers' setting in galaxy.ini which comes up only applies when REMOTE_USER is in use. There's a separate UCSC_SERVERS list in lib/galaxy/web/framework/webapp.py and a sites Bunch in galaxy/security/__init__.py which define the hgwx.cse... URLs. Would it be reasonable if I came up with a little PR which makes it possible to override these from display_servers too? The one question I have is when are the security/__init.py__ checks used, and when is it framework/webapp.py? I couldn't work that out by looking. Thanks, DT -- David Trudgian Ph.D. Computational Scientist, BioHPC UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] package_bowtie_2_2_4 Executable Permissions - can't run bowtie as non-galaxy user
Installed bowtie2 from the toolshed, which brings in package_bowtie_2_2_4, which downloads bowtie binaries from depot.galaxyproject.org http://depot.galaxyproject.org/package/linux/x86_64/bowtie2/bowtie2-2.2.4-linux-x86_64.tgz Jobs submitted on our setup, running as a 'real user' (not galaxy user) via pulsar + SLURM fail with a permissions error. This is due to the permissions on some of the bowtie executables in the .tar.gz from depot being owner/group executable only. Other users cannot run bowtie2: -rwxrwx---. 1 galaxy galaxy 18K Oct 22 2014 bowtie2 If I download the tar.gz from depot manually and extract then I get: -rwxrwx--- 1 dtrudgian biohpc_admin 18K Oct 22 2014 bowtie2 Could the .tar.gz be fixed, so that the executables will work in a run as real user configuration? Is there any sensible way Galaxy could enforce permissions to guard against this issue? Cheers, -- David Trudgian Ph.D. Computational Scientist, BioHPC UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 Please contact biohpc-help@utsouthwestern with general BioHPC inquries. UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy on CentOS?
Hi Carlos, Our Galaxy is running on RedHat 6.6 on our cluster - which is basically identical to CentOS 6.6. Per Peter's email we haven't had any big issues with Galaxy on CentOS 6.x. You are likely to want to use the postgresql packages from www.postrgresql.org, not the out-of-date versions that come with the distribution. I also installed an up-to-date python 2.7.x for Galaxy using pyenv but this shouldn't be necessary - Galaxy should work fine on python 2.6. In my case I did it to have a standard setup with other apps that do need python 2.7. Which workload manager will your cluster manager be configured to use? DT -Original Message- From: galaxy-dev [mailto:galaxy-dev-boun...@lists.galaxyproject.org] On Behalf Of Peter Cock Sent: Tuesday, May 05, 2015 3:13 AM To: Carlos Lijeron Cc: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Galaxy on CentOS? Hi Carlos, We're running Galaxy on CentOS 6.6, so that in itself shouldn't be a problem. Most of the effort was sorting out shared storage with the cluster, which in our case is managed with SGE (fairly commonly used with Galaxy). In reply to your thread last month David Trudgian said he was using Bright Cluster Manager: http://dev.list.galaxyproject.org/Galaxy-on-HPC-and-Bright-Cluster-Manager-tc4667015.html Regards, Peter On Tue, May 5, 2015 at 2:51 AM, Carlos Lijeron clije...@hunter.cuny.edu wrote: Hello everyone, We are soon acquiring our cluster which has CentOS on the master node along with Bright Cluster Manager and workload manager. We are wondering if there is a guideline on how to install and support Galaxy on CentOS. Does anyone have any info on this? Thanks ! Carlos. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy on CentOS?
Hi Carlos, I previously used Galaxy with a GridEngine cluster. Having GridEngine is no more complex than having SLURM. R.E. you other email, recording cluster usage for labs' galaxy jobs, that requires the 'running jobs as real user' setup... https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#Submitting_Jobs_as_the_Real_User There are caveats with that setup r.e. file permissions / privacy of data between groups. In our situation everything has to be kept private, so to ensure jobs run as a real cluster user we have a complex setup using Galaxy pulsar as well as SLURM. Some notes on that in a GitHub issue here: https://github.com/galaxyproject/pulsar/issues/65 Cheers, DT -Original Message- From: Carlos Lijeron [mailto:clije...@hunter.cuny.edu] Sent: Tuesday, May 05, 2015 11:18 AM To: David Trudgian; Peter Cock Cc: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Galaxy on CentOS? David, We are in our final configuration stage and tomorrow we have a conversation with Silicon Mechanics business integration team to discuss the workload manager, and other details about the data center backbone speed. As per your suggestion I inquired about using SLURM instead of SGI, but I haven¹t been able to make a good case as to why this is. My only support is your email / suggestion and your existing experience with an HPC, Bright and SLURM. My inexperience with this type of setups and lack of personnel at our institution makes me rely 100% of everyone's feedback in this group Thank you David and anyone else in the group for your contributions and suggestions. Attached is the diagram of our future implementation. Another concern that I have is that our sequencer will be located 0.5 miles from the HPC, but there is a 10 Gpbs point to point connection between the 2 buildings. I wanted to find a way to calculate / predict the average amount of traffic that will go from the sequencer to the HPC so we can allocate specific bandwidth for our use and leave the rest to other users. Carlos. On 5/5/15, 10:24 AM, David Trudgian david.trudg...@utsouthwestern.edu wrote: Our Galaxy is running on RedHat 6.6 on our cluster - which is basically identical to CentOS 6.6. Per Peter's email we haven't had any big issues with Galaxy on CentOS 6.x. You are likely to want to use the postgresql packages from www.postrgresql.org, not the out-of-date versions that come with the distribution. I also installed an up-to-date python 2.7.x for Galaxy using pyenv but this shouldn't be necessary - Galaxy should work fine on python 2.6. In my case I did it to have a standard setup with other apps that do need python 2.7. Which workload manager will your cluster manager be configured to use? DT -Original Message- From: galaxy-dev [mailto:galaxy-dev-boun...@lists.galaxyproject.org] On Behalf Of Peter Cock Sent: Tuesday, May 05, 2015 3:13 AM To: Carlos Lijeron Cc: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Galaxy on CentOS? Hi Carlos, We're running Galaxy on CentOS 6.6, so that in itself shouldn't be a problem. Most of the effort was sorting out shared storage with the cluster, which in our case is managed with SGE (fairly commonly used with Galaxy). In reply to your thread last month David Trudgian said he was using Bright Cluster Manager: http://dev.list.galaxyproject.org/Galaxy-on-HPC-and-Bright-Cluster-Mana ger -tc4667015.html Regards, Peter UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy on HPC and Bright Cluster Manager?
Carlo, We have Bright Cluster Manager in use on our cluster for node provisioning etc. but the actual job scheduler in use in our case is SLURM, which we use directly. Are you using one of the integrated workload managers such as SLURM / SGE / TORQUE directly, or indirectly via cmsub? I guess the easiest way to come up with some kind of advice is if you can provide an example of generic job script you are using on your system. If you're using cmsub is it specifying a --wlmanager etc. DT -Original Message- From: galaxy-dev [mailto:galaxy-dev-boun...@lists.galaxyproject.org] On Behalf Of John Chilton Sent: Wednesday, April 22, 2015 8:26 AM To: Carlos Lijeron Cc: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Galaxy on HPC and Bright Cluster Manager? Hello Carlos, I have never heard of anyone running Galaxy with Bright Cluster Manager (though hopefully someone will chime in if they have). If you are interested in adding support it should be possible. One complication is that Bright Cluster Manager doesn't appear to have a DRMAA interface (http://www.drmaa.org/) which is the most direct way to utilize new DRMs. Without that my approach would be to build a new CLI runner: There are a few examples here that one can use as template: https://github.com/galaxyproject/galaxy/tree/dev/lib/galaxy/jobs/runners/util/cli/job I guess you would have to write a new one targeting cmsub I guess - you also need to be able to parse a job status somehow - I haven't figured out how to do that from the documentation - but I assume there is a way. I looks like Bright supports running SGE, SLURM, and Torque on the cluster - doing this and interfacing with one of those more common options directly might be a better approach for Galaxy (and other users if your cluster has them). -John On Wed, Apr 22, 2015 at 8:56 AM, Carlos Lijeron clije...@hunter.cuny.edu wrote: Good day everyone, Has anyone of you been able to implement Galaxy on a HPC using Bright Cluster Manager as the main DRM? I noticed that only a few have been known to work with Galaxy, but the list does not include Bright. Any advice/ideas will be greatly appreciated. TORQUE Resource Manager PBS Professional Open Grid Engine Univa Grid Engine (previously known as Sun Grid Engine and Oracle Grid Engine) Platform LSF HTCondor Slurm Galaxy Pulsar (formerly LWR) Thanks. Carlo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] MySQL InnoDB - Execution Order
InnoDB can be set as default: http://serverfault.com/questions/93559/how-do-i-set-the-default-table-type-as-innodb-in-my-cnf It should also be the default storage engine if you are using MySQL 5.5 or higher: http://dev.mysql.com/doc/refman/5.6/en/innodb-default-se.html Which version of MySQL are you using? 5.5 / 5.6 are generally a lot nicer than e.g. 5.1 Dave T From: galaxy-dev [mailto:galaxy-dev-boun...@lists.galaxyproject.org] On Behalf Of McCully, Dwayne (NIH/NIAMS) [C] Sent: Friday, April 10, 2015 3:54 PM To: 'galaxy-dev@lists.galaxyproject.org' Subject: [galaxy-dev] MySQL InnoDB - Execution Order Hello Everyone, The running Galaxy in a production environment documentation states: If you are using MySQLhttp://dev.mysql.com/ with MyISAMhttp://dev.mysql.com/doc/refman/5.1/en/myisam-storage-engine.html table engine when Galaxy is in multiprocess configuration, workflow steps will get executed out of orderhttp://dev.list.galaxyproject.org/Job-execution-order-mixed-up-tt4662488.html and fail. Use InnoDBhttp://dev.mysql.com/doc/refman/5.1/en/innodb-storage-engine.html engine instead or switch to PostgreSQLhttp://www.postgresql.org/. I've enabled InnoDB for my MySQL database via the my.cnf file but the default database and database to log all database transactions were created with MyISAM files. If this is normal, when will Galaxy create InnoDB tables? I don't believe you can create a database with InnoDB. Dwayne UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] DRMAA chmod to world-writable if to user chown fails should not be default?
John, Looks good. I’m afraid I can’t test this on our main install any more, but I do have a VM somewhere at home which will let me test in a reasonable amount of time. I could probably look at that by the end of the week. Sorry I can’t do it straight away. Cheers, DT From: John Chilton [mailto:jmchil...@gmail.com] Sent: Wednesday, April 08, 2015 8:12 AM To: David Trudgian Cc: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] DRMAA chmod to world-writable if to user chown fails should not be default? Yeah - I am with you on this. It is definitely less than ideal that any exception will cause that to happen. Here is a commit that I believe should fix the problem: https://github.com/galaxyproject/galaxy/commit/87e19aa4d708d6c719b396867f431d94c92077c5 Do you have a working run-as-real user setup to test this - it would take some time for me to test this but if you have something ready to go - let me know and I will open a PR :). Also - I noticed the documentation for the run-as-real-user stuff seems to match the code implementation - can you peak at https://github.com/galaxyproject/galaxy/pull/94 and let me know if it more accurately reflects how you got things up and running? Thanks for finding and reporting this issue! -John On Mon, Mar 30, 2015 at 3:33 PM, David Trudgian david.trudg...@utsouthwestern.edumailto:david.trudg...@utsouthwestern.edu wrote: Hi, Looking through the DRMAA job runner code I noticed that in lib/galaxy/jobs/__init__.py:1582 that if chown to a user in change_ownership_for_run fails then there is chmod 0777 instead. I’m thinking that this probably shouldn’t happen by default, or there needs to be a big warning on the wiki? I understand the world-readable security concerns posted there – but world-writable, for whatever reason, on a shared cluster seems like a disaster waiting to happen. Maybe either warn on the wiki, make the chmod 777 a non-default option, or an option to disable it? DT UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] NGINX uWSGI with require_login needs uwsgi_param SCRIPT_NAME '';
Dannon, thanks, This is on dev at commit 43a94ada37124dde5759b0c94bf25a97ff4da17a Am running nginx 1.6.2 on RedHat EL 6.6 Python 2.7.8 with uWSGI 2.0.10 I can send my galaxy.ini and nginx.conf to you separately off list. DT -- David Trudgian Ph.D., Computational Scientist UT Southwestern BioHPC From: Dannon Baker [dannon.ba...@gmail.com] Sent: Thursday, April 02, 2015 3:52 PM To: David Trudgian Cc: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] NGINX uWSGI with require_login needs uwsgi_param SCRIPT_NAME ''; Hey David, What version of galaxy are you using? I'm not able to reproduce this with the current release, but maybe I don't have all the information here. Previously, this SCRIPT_NAME issue was observed when static_enabled was true, but I've never seen it otherwise. Any information you might be able to share to help me reproduce this would be valuable -- we shouldn't have to inject meaningless parameters into the environment to make things work. -Dannon On Thu, Apr 2, 2015 at 3:23 PM, David Trudgian david.trudg...@utsouthwestern.edumailto:david.trudg...@utsouthwestern.edu wrote: Hi, When I enabled 'require_login = True' on a galaxy instance running with NGINX+uWSGI I receive internal server error pages due to a Key error for File 'lib/galaxy/web/framework/base.py', line 356 in path return self.environ['SCRIPT_NAME'] + self.environ['PATH_INFO'] KeyError: 'SCRIPT_NAME' Works fine with 'require_login = False' - I'm not sure why require_login makes it take a code path which needs SCRIPT_NAME. Anyway - the addition of... uwsgi_param SCRIPT_NAME ''; ... to the nginx.conf prevents this error. Should I try and add this to the uWSGI config snippets on the wiki? Thanks, -- David Trudgian Ph.D., Computational Scientist UT Southwestern BioHPC -- David Trudgian Ph.D., Computational Scientist UT Southwestern BioHPC UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] NGINX uWSGI with require_login needs uwsgi_param SCRIPT_NAME '';
Hi, When I enabled 'require_login = True' on a galaxy instance running with NGINX+uWSGI I receive internal server error pages due to a Key error for File 'lib/galaxy/web/framework/base.py', line 356 in path return self.environ['SCRIPT_NAME'] + self.environ['PATH_INFO'] KeyError: 'SCRIPT_NAME' Works fine with 'require_login = False' - I'm not sure why require_login makes it take a code path which needs SCRIPT_NAME. Anyway - the addition of... uwsgi_param SCRIPT_NAME ''; ... to the nginx.conf prevents this error. Should I try and add this to the uWSGI config snippets on the wiki? Thanks, -- David Trudgian Ph.D., Computational Scientist UT Southwestern BioHPC -- David Trudgian Ph.D., Computational Scientist UT Southwestern BioHPC UT Southwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Safe / possible to sync toolshed tools and deps from another machine?
Dave, Many thanks for your detailed reply. I like the sound of the second solution which should be no problem to setup on our systems. Still hoping it'll be possible to get a proxy exception for our system and avoid this altogether longer term, but good to know it should be possible for now. Thanks again, DT -Original Message- From: Dave Bouvier [mailto:d...@bx.psu.edu] Sent: Monday, March 30, 2015 8:03 AM To: David Trudgian; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Safe / possible to sync toolshed tools and deps from another machine? Dave, There are a number of things to consider when doing this, but it should be possible. Things to keep in mind are, in no particular order: The tool dependency path on the workstation should match what would be the path on the server. A substantial number of packages link their libraries with absolute paths, so any change in paths would of course break the package at runtime. The operating system, hardware, and installed libraries on the server should at worst be a superset of those on the workstation, to avoid compiling packages on the workstation that have dependencies not present on the server. I would recommend setting the install_database_connection option in your config/galaxy.ini file, so that you can export the database of installed repositories and tool dependencies from the workstation, and import this to the server without overwriting users, jobs, and so on. If those hurdles are cleared, it's likely that installing and copying dependencies from one system to another might work. Another solution, more complicated to set up but easier to use, would be to run a single install database that both systems access, and install the dependencies on the workstation, into a network filesystem that is mounted in the same path on both systems. You would still need to sync the shed_tool_conf.xml across machines, but a restart of the server would then show all the tools and their dependencies with whatever status they should have. --Dave B. On 03/27/2015 11:36 AM, David Trudgian wrote: Hi, I was wondering whether it might be possible/safe to install tools and deps from the toolshed on a VM galaxy installation, and then rsync the shed_tools and tool dependencies directories to a production galaxy server? Unfortunately, our institution has web access only through a proxy and blocks ‘personal web browsing’ from servers. This works on a white list – so everything is blocked unless there was considered to be a valid need when the list was originally setup, or an exception request is made and approved. We have an exception in place for the main toolshed, but various dependency downloads will fail, blocked by the proxy. Getting exceptions for each URL we find not to work is painful and slow. The suggestion I’ve been given is to download things on a workstation (not subject to the blocking) and transfer them manually to the server, hence the question about whether tool directories (and XML conf files) can be synced between installs? Is there anything in the SQL DB that would prevent this working, or being safe? Otherwise I’m installing a lot of deps manually L Thanks, Dave Trudgian -- -- UTSouthwestern Medical Center The future of medicine, today. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
