[galaxy-dev] Bug with multiple reverse proxies
Hi, We ran into a problem if galaxy is running behind multiple reverse proxies. Galaxy assumes that the X-Forwarded-Host header only contains a single host to which it redirects, but apache will append comma-separated components to it if it already exists, which is the case of multiple reverse proxies. The result is that galaxy sometimes (i.e. sometimes after user login, depending on browser specifics, and when installing toolshed tools) tries to redirect the browser to hostname, hostname. Apache docs http://httpd.apache.org/docs/2.4/mod/mod_proxy.html#x-headers I have a patch for galaxy in this pull request https://bitbucket.org/galaxy/galaxy-central/pull-request/560/use-first-component-of-x-forwarded-for/diff. That change fixes a file that makes the same assumption. The actual fix requires a change in Routes. The Routes PR is here https://github.com/bbangert/routes/pull/33, so actually fixing this requires an update of Routes after that pr is merged there. Jan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Fwd: Bug with multiple reverse proxies
Hi, We ran into a problem if galaxy is running behind multiple reverse proxies. Galaxy assumes that the X-Forwarded-Host header only contains a single host to which it redirects, but apache will append comma-separated components to it if it already exists, which is the case of multiple reverse proxies. The result is that galaxy sometimes (i.e. sometimes after user login, depending on browser specifics, and when installing toolshed tools) tries to redirect the browser to hostname, hostname. Apache docs http://httpd.apache.org/docs/2.4/mod/mod_proxy.html#x-headers I have a patch for galaxy in this pull request https://bitbucket.org/galaxy/galaxy-central/pull-request/560/use-first-component-of-x-forwarded-for/diff. That change fixes a file that makes the same assumption. The actual fix requires a change in Routes. The Routes PR is here https://github.com/bbangert/routes/pull/33, so actually fixing this requires an update of Routes after that pr is merged there. Jan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Fwd: Bug with multiple reverse proxies
Hi, We ran into a problem if galaxy is running behind multiple reverse proxies. Galaxy assumes that the X-Forwarded-Host header only contains a single host to which it redirects, but apache will append comma-separated components to it if it already exists, which is the case of multiple reverse proxies. The result is that galaxy sometimes (i.e. sometimes after user login, depending on browser specifics, and when installing toolshed tools) tries to redirect the browser to hostname, hostname. Apache docs http://httpd.apache.org/docs/2.4/mod/mod_proxy.html#x-headers I have a patch for galaxy in this pull request https://bitbucket.org/galaxy/galaxy-central/pull-request/560/use-first-component-of-x-forwarded-for/diff. That change fixes a file that makes the same assumption. The actual fix requires a change in Routes. The Routes PR is here https://github.com/bbangert/routes/pull/33, so actually fixing this requires an update of Routes after that pr is merged there. Jan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Patch for libxslt and libxml packages
Hi all, I was told that updates to packages owned by devteam should just be sent here, since there is no public repo. I have patches for the libxml2 and libxslt tool_dependency.xml's, that set some more environment variables that my tool needs (specifically, the python lxml package needs some of these). Can they be applied to the official packages? Jan diff -r 3af7ad13a6fb tool_dependencies.xml --- a/tool_dependencies.xml Wed Dec 11 15:03:25 2013 -0500 +++ b/tool_dependencies.xml Mon Aug 04 12:01:37 2014 +0200 @@ -1,3 +1,4 @@ +?xml version=1.0? tool_dependency package name=libxml2 version=2.9.1 install version=1.0 @@ -8,6 +9,10 @@ action type=set_environment environment_variable name=PKG_CONFIG_PATH action=prepend_to$INSTALL_DIR/lib/pkgconfig/environment_variable environment_variable name=PATH action=prepend_to$INSTALL_DIR/bin/environment_variable +environment_variable name=LD_LIBRARY_PATH action=prepend_to$INSTALL_DIR/lib/environment_variable +environment_variable name=LIBRARY_PATH action=prepend_to$INSTALL_DIR/lib/environment_variable +environment_variable name=C_INCLUDE_PATH action=prepend_to$INSTALL_DIR/include/environment_variable +environment_variable name=CPLUS_INCLUDE_PATH action=prepend_to$INSTALL_DIR/include/environment_variable /action /actions /install diff -r 039b04adcfee tool_dependencies.xml --- a/tool_dependencies.xml Tue Feb 25 14:26:13 2014 -0500 +++ b/tool_dependencies.xml Mon Aug 04 12:22:15 2014 +0200 @@ -17,6 +17,10 @@ action type=set_environment environment_variable action=prepend_to name=PKG_CONFIG_PATH$INSTALL_DIR/lib/pkgconfig/environment_variable environment_variable action=prepend_to name=PATH$INSTALL_DIR/bin/environment_variable +environment_variable name=LD_LIBRARY_PATH action=prepend_to$INSTALL_DIR/lib/environment_variable +environment_variable name=LIBRARY_PATH action=prepend_to$INSTALL_DIR/lib/environment_variable +environment_variable name=C_INCLUDE_PATH action=prepend_to$INSTALL_DIR/include/environment_variable +environment_variable name=CPLUS_INCLUDE_PATH action=prepend_to$INSTALL_DIR/include/environment_variable /action /actions /install ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] setup_venv doesn't honour set_environment_for_install / installing lxml
That indeed looks like a better solution. I'll use the workaround for now and update once setup_python_environment is implemented. On 25 June 2014 13:33, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Janis, that is not working and it is currently not clear if we will change it. Please see the following Trello Card: https://trello.com/c/NsLJv9la/61-clean-up-tool-shed-setup-actions One of my project during the upcoming GCC hackathon is to implement a setup_python_environment, like the R, perl or ruby ones. With that in place your use case will be easy to implement. If you get your python package up and running at that time, this would be great! Cheers, Bjoern Am 24.06.2014 18:57, schrieb Jan Kanis: Ok, here is the tool_dependencies.xml. This https://gist.github.com/JanKanis/650c88001c03ac4320fe# file-not_working_tool_dependencies (also attached, if that survives the list) is what I would like my tool_dependencies.xml to look like, but it doesn't work because the installation of lxml fails, as it can't find the libxml2 headers. I am now using this https://gist.github.com/JanKanis/42b9cace27b9693a0677# file-workaround_tool_dependencies-xml as a workaround. It works because lxml is installed from a shell_command which does include the variables set from the set_environment_for_install block. This is from the blast2html tool. Jan On 24 June 2014 15:30, Dave Bouvier d...@bx.psu.edu wrote: Jan, In order to help track down this issue, could you provide the tool_dependencies.xml you're using? --Dave B. On Tue 24 Jun 2014 05:40:30 AM EDT, Jan Kanis wrote: In a tool_dependency.xml file I want to install python package lxml in a virtual environment, as a tool I'm building needs it. The python lxml package requires the libxml2 tool dependency. I have added a set_environment_for_install action that refers to the libxml2 repository, but when python/pip tries to install lxml it fails, apparently because it can't find the required headers. This appears to be because the setup_virtualenv action does not include install environment variables. It seems to me that install environment variables should be sourced for every following action that can do nontrivial things, not just shell commands. Alternatively, am I trying to install lxml the wrong way, is there a better way? (I'm running on python 2.6) Jan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] How does Galaxy differentiate users who have not logged on?
Probably the same way by which it keeps track of you when you are logged in: setting a cookie in the browser. I didn't verify this, but that is how basically all web services do it. If you clear the galaxy cookies I expect you will end up in a new empty workspace. On 25 June 2014 01:18, Melissa Cline cl...@soe.ucsc.edu wrote: Hi folks, Quick question. When two different users use the same Galaxy instance without logging in, how does Galaxy keep their identities straight, such that when they come back to their computers the next day, they see only their own workspace and not the other user's workspace? Thanks! Melissa ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] setup_venv doesn't honour set_environment_for_install / installing lxml
In a tool_dependency.xml file I want to install python package lxml in a virtual environment, as a tool I'm building needs it. The python lxml package requires the libxml2 tool dependency. I have added a set_environment_for_install action that refers to the libxml2 repository, but when python/pip tries to install lxml it fails, apparently because it can't find the required headers. This appears to be because the setup_virtualenv action does not include install environment variables. It seems to me that install environment variables should be sourced for every following action that can do nontrivial things, not just shell commands. Alternatively, am I trying to install lxml the wrong way, is there a better way? (I'm running on python 2.6) Jan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] setup_venv doesn't honour set_environment_for_install / installing lxml
Ok, here is the tool_dependencies.xml. This https://gist.github.com/JanKanis/650c88001c03ac4320fe#file-not_working_tool_dependencies (also attached, if that survives the list) is what I would like my tool_dependencies.xml to look like, but it doesn't work because the installation of lxml fails, as it can't find the libxml2 headers. I am now using this https://gist.github.com/JanKanis/42b9cace27b9693a0677#file-workaround_tool_dependencies-xml as a workaround. It works because lxml is installed from a shell_command which does include the variables set from the set_environment_for_install block. This is from the blast2html tool. Jan On 24 June 2014 15:30, Dave Bouvier d...@bx.psu.edu wrote: Jan, In order to help track down this issue, could you provide the tool_dependencies.xml you're using? --Dave B. On Tue 24 Jun 2014 05:40:30 AM EDT, Jan Kanis wrote: In a tool_dependency.xml file I want to install python package lxml in a virtual environment, as a tool I'm building needs it. The python lxml package requires the libxml2 tool dependency. I have added a set_environment_for_install action that refers to the libxml2 repository, but when python/pip tries to install lxml it fails, apparently because it can't find the required headers. This appears to be because the setup_virtualenv action does not include install environment variables. It seems to me that install environment variables should be sourced for every following action that can do nontrivial things, not just shell commands. Alternatively, am I trying to install lxml the wrong way, is there a better way? (I'm running on python 2.6) Jan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ?xml version=1.0? tool_dependency package name=libxml2 version=2.9.1 repository name=package_libxml2_2_9_1 owner=jankanis prior_installation_required=True toolshed=http://toolshed.g2.bx.psu.edu; changeset_revision=98807f0505f0 / /package package name=libxslt version=1.1.28 repository name=package_libxslt_1_1_28 owner=devteam prior_installation_required=True toolshed=http://toolshed.g2.bx.psu.edu; changeset_revision=039b04adcfee / /package package name=zlib version=1.2.8 repository name=package_zlib_1_2_8 owner=iuc prior_installation_required=True toolshed=http://toolshed.g2.bx.psu.edu; changeset_revision=63a4a902cda2 / /package package name=blast2html_venv version=0.2 install version=1.0 actions action type=set_environment_for_install repository name=package_libxml2_2_9_1 owner=jankanis toolshed=http://toolshed.g2.bx.psu.edu; changeset_revision=98807f0505f0 package name=libxml2 version=2.9.1 / /repository repository name=package_libxslt_1_1_28 owner=devteam toolshed=http://toolshed.g2.bx.psu.edu; changeset_revision=039b04adcfee package name=libxslt version=1.1.28 / /repository repository name=package_zlib_1_2_8 owner=iuc toolshed=http://toolshed.g2.bx.psu.edu; changeset_revision=63a4a902cda2 package name=zlib version=1.2.8 / /repository /action action type=setup_virtualenv jinja2==2.7.3 six==1.7.2 argparse==1.2.1 lxml==3.3.5 /action action type=set_environment !-- add libxml2/zlib library paths so the runtime can find the shared libraries -- environment_variable name=LD_LIBRARY_PATH action=prepend_to$ENV[LD_LIBRARY_PATH]/environment_variable /action /actions /install readmeA Python 2 virtual environment that includes the python packages blast2html depends on, which are lxml and jinja2./readme /package /tool_dependency ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Per-tool configuration
I am not using job splitting, because I am implementing this for a client with a small (one machine) galaxy setup. Implementing a query limit feature in galaxy core would probably be the best idea, but that would also probably require an admin screen to edit those limits, and I don't think I can sell the required time to my boss under the contract we have with the client. I gave a quick try before on making the blast2html tool run in both python 2.6 and 3, but I gave up due to too many encoding issues. The client's machine has python 2.6. Maybe I should have another look. Jan On 17 June 2014 21:55, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Jun 17, 2014 at 4:57 PM, Jan Kanis jan.c...@jankanis.nl wrote: Too bad there aren't any really good options. I will use the environment variable approach for the query size limit. Are you using the optional job splitting (parallelism) feature in Galaxy? That seems to be me to be a good place to insert a Galaxy level job size limit. e.g. BLAST+ jobs are split into 1000 query chunks, so you might wish to impose a 25 chunk limit? Long term being able to set limits on the input file parameters of each tool would be nicer - e.g. Limit BLASTN to at most 20,000 queries, limit MIRA to at most 50GB FASTQ files, etc. For the gene bank links I guess modifying the .loc file is the least bad way. Maybe it can be merged into galaxy_blast, that would at least solve the interoperability problems. It would have to be sufficiently general, and backward compatible. FYI other people have also looked at extending the blast *.loc files (e.g. adding a category column for helping filter down a very large BLAST database list). @Peter: One potential problem in merging my blast2html tool could be that I have written it in python3, and the current tool wrapper therefore installs python3 and a host of its dependencies, making for a quite large download. Without seeing your code, it is hard to say, but actually writing Python code which works unmodified under Python 2.7 and Python 3 is quite doable (and under Python 2.6 with a few more provisos). Both NumPy and Biopython do this if you wanted some reassurance. On the other hand, Galaxy itself will need to more to Python 3 at some point, and certainly individual tools will too. This will probably mean (as with Linux Python packages) having double entries on the ToolSehd (one for Python 2, one for Python 3), e.g ToolShed package for NumPy under Python 2 (done) and under Python 3 (needed). Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Per-tool configuration
Too bad there aren't any really good options. I will use the environment
variable approach for the query size limit. For the gene bank links I guess
modifying the .loc file is the least bad way. Maybe it can be merged into
galaxy_blast, that would at least solve the interoperability problems.
@Peter: One potential problem in merging my blast2html tool could be that I
have written it in python3, and the current tool wrapper therefore installs
python3 and a host of its dependencies, making for a quite large download.
Jan
On 16 June 2014 09:08, Peter Cock p.j.a.c...@googlemail.com wrote:
On Mon, Jun 16, 2014 at 4:18 AM, John Chilton jmchil...@gmail.com wrote:
Hello Jan,
Thanks for the clarification. Not quite what I was expecting so I am
glad I asked - I don't have great answers for either case so hopefully
other people will have some ideas.
For the first use case - I would just specify some default input to
supply to the input wrapper - lets call this N - add a parameter to
the tool wrapper --limit-size=N - test that and then allow it to be
overridden via an environment variable - so in your command block use
--limit-size=\${BLAST_QUERY_LIMIT:N}. This will use N is not limit
is set, but deployers can set limits. There are a number of ways to
set such variables - DRM specific environment files, login rc files,
etc Just this last release I added the ability to define
environment variables right in job_conf.xml
(
https://bitbucket.org/galaxy/galaxy-central/pull-request/378/allow-specification-of-environment/diff
).
I thought the tool shed might have a way to collect such definitions
as well and insert them into package files - but Google failed to find
this for me.
Hmm. Jan emailed me off list earlier about this. We could insert
a pre-BLAST script to check the size of the query FASTA file,
and abort if it is too large (e.g. number of queries, total sequence
length, perhaps scaled according to the database size if we want
to get clever?).
I was hoping there was a more general mechanism in Galaxy -
after all, BLAST is by no means the only computationally
expensive tool ;)
We have had query files of 20,000 and more genes against NR
(both BLASTP and BLASTX), but our Galaxy has task-splitting
enabled so this becomes 20 (or more) individual cluster jobs
of 1000 queries each. This works fine apart from the occasional
glitch with the network drive when the data is merged afterwards.
(We know this failed once shortly after the underlying storage
had been expanded, and would have been under heavy load
rebalancing the data across the new disks.)
Not sure about how to proceed with the second use case - extending the
.loc file should work locally - I am not sure it is feasible within
the context of the existing tool shed tools, data manager, etc You
could certainly duplicate this stuff with your modifications - this
how down sides in terms of interoperability though.
Currently the BLAST wrappers use the *.loc files directly, but
this is likely to switch to the newer Data Manager approach.
That may or may not complicate local modifications like adding
extra columns...
Sorry I don't have great answers for either question,
-John
Thanks John,
Peter
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Per-tool configuration
I have two use cases: the first is for a modification of the ncbi blast wrapper to limit the query input size (for a publically accessible galaxy instance), so this needs a configuration option for the query size limit. I was thinking about a separate config file in tool-data for this. The second is is for a tool I have written to convert a blast xml output into a html report. The report contains links for each match to a gene bank (e.g. the ncbi database). These links should be configurable per database that was searched, and preferrably have an option of linking to the location of the match within the gene if the gene bank supports such links. One option is to add an extra column to the blast .loc files (if that doesn't break blast), where the databases are already configured. Jan Op 13 jun. 2014 18:02 schreef John Chilton jmchil...@gmail.com het volgende: I would have different answers for your depending on what options are available to the server admin. What exactly about the tool is configurable - can you be more specific? -John On Fri, Jun 13, 2014 at 10:59 AM, Jan Kanis jan.c...@jankanis.nl wrote: I am writing a tool that should be configurable by the server admin. I am considering adding a configuration file, but where should such a file be placed? Is the tool-data directory the right place? Is there another standard way for per-tool configuration? Jan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Per-tool configuration
I am writing a tool that should be configurable by the server admin. I am considering adding a configuration file, but where should such a file be placed? Is the tool-data directory the right place? Is there another standard way for per-tool configuration? Jan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] devteam packages
Hi all, Where are the toolshed packages owned by devteam maintained? The galaxy sources don't seem to include those tool shed packages and I was unable to find a related repository on github or a similar place. Jan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
