Re: [galaxy-dev] Downgrade tool upload tool
Thanks Peter. I think you are right and that its a change in Galaxy which causes this routing issue. I have prepared a fix for it: https://github.com/galaxyproject/galaxy/compare/dev...guerler:fix_upload_link_000?expand=1. Ill double check tomorrow with those working on the routing and then issue a PR. Thanks a lot for reporting this Makis. On Mon, Feb 29, 2016 at 8:17 AM, Peter Cockwrote: > Thanks for clarifying Makis, > > I think this change is likely due to changes in Galaxy rather than > in the upload tool itself. On our (slightly out of date) Galaxy or the > public server, that URL no longer loads the original tool form - > you get the new drag-and-drop upload interface. > > I would guess some of the special case code like this matters: > > https://github.com/galaxyproject/galaxy/blob/fe6d22e1a7e4e6d38b15f4128554783765bc69a1/client/galaxy/scripts/mvc/tool/tools.js#L525 > > Probably Sam (Aysam) can give you more practical advice (CC'd)? > > Peter > > On Mon, Feb 29, 2016 at 12:29 PM, Makis Ladoukakis > wrote: > > I am making some clickable buttons in the home page of my instance > through a > > java script that is called from the welcome.html file. These buttons > > correspond to my most used tools. For example I have a section like this: > > > > > > > > > > > > > > > src="feimg/GetData.gif"/> > > > > > href="http://myserver/galaxy/tool_runner?tool_id=upload1; > > target="_parent"> > height=300%/> > > > href="http://myserver/galaxy/library/browse_libraries; > target="_parent"> > src="feimg/Libraries.gif" width=250% height=300%/> > > > > > > > > > > > > > > indicating two buttons one for the upload tool and one for the libraries. > > The libraries button work but when I click on the button for the upload > tool > > the webpage just refreshes instead of opening the tool. This worked > > perfectly with the previous version of the upload tool and I was thinking > > how to get it back. > > > > Thank you, > > Makis > > > > > >> Date: Mon, 29 Feb 2016 11:39:26 + > >> Subject: Re: [galaxy-dev] Downgrade tool upload tool > >> From: p.j.a.c...@googlemail.com > >> To: makis4e...@hotmail.com > >> CC: j...@galaxyproject.org; galaxy-...@lists.bx.psu.edu; > >> galaxy-dev@lists.galaxyproject.org > > > >> > >> Hi Makis, > >> > >> The upload tool is a little special in that it is used internally by > >> Galaxy and doesn't entirely behave like a normal tool. > >> > >> What are you trying to achieve by "downgrading" it? > >> > >> Peter > >> > >> On Mon, Feb 29, 2016 at 11:29 AM, Makis Ladoukakis > >> wrote: > >> > What about if I used a second "tool upload" tool? I took the python > >> > script > >> > from an earlier version and named it upload_old.py and did the same > for > >> > the > >> > xml file (upload.xml -> upload_old.xml) while changing the latter to > >> > call > >> > the new python script. Nevertheless when I did that and tried to run > it > >> > I > >> > got a popup window with the following error: > >> > > >> > Tool request failed > >> > > >> > Uncaught exception in exposed API method: > >> > > >> > > >> > Is there a way to make this work? Or maybe another workaround? > >> > > >> > Thank you in advance, > >> > Makis > >> > > >> > > >> > > >> > > >> > From: j...@galaxyproject.org > >> > Date: Fri, 26 Feb 2016 10:22:44 -0800 > >> > Subject: Re: [galaxy-dev] Downgrade tool upload tool > >> > To: makis4e...@hotmail.com > >> > CC: galaxy-...@lists.bx.psu.edu; galaxy-dev@lists.galaxyproject.org > >> > > >> > > >> > Hello, > >> > > >> > The Upload tool is a direct component of Galaxy. To use an earlier > >> > version > >> > of the tool, I believe that you would have to run an earlier version > of > >> > Galaxy. > >> > > >> > Others here can amend this, if they know of an alternative. > >> > > >> > Thanks, > >> > > >> > Jen > >> > > >> > -- > >> > Jennifer Hillman-Jackson > >> > Galaxy Application Support > >> > http://usegalaxy.org > >> > http://galaxyproject.org > >> > http://biostar.usegalaxy.org > >> > > >> > > >> > On Fri, Feb 26, 2016 at 4:01 AM, Makis Ladoukakis > >> > > >> > wrote: > >> > > >> > Hello everyone, > >> > > >> > I was wondering whether it's possible to downgrade the upload tool to > >> > the > >> > previous version(1.1.4). > >> > > >> > I copied the upload.py and upload.xml scripts from a previous galaxy > >> > version > >> > to the tools/data_source/ directory and reloaded the tool as admin but > >> > it > >> > hasn't changed even when I restarted the server. > >> > > >> > Do I have to change something else in addition? > >> > > >> > Thank you, > >> > Makis > >> > > >> > ___ > >> > Please keep all replies on the list by using
Re: [galaxy-dev] Galaxy on Azure
Hi Alex, Assuming you'd want to set it up as a dedicated server on a VM running on Azure, I'd suggest taking a look at the production instance setup notes: https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer Other admin topics are also available on the wiki: https://wiki.galaxyproject.org/Admin Hope this helps. On Mon, Feb 29, 2016 at 1:40 PM, Alex Lenailwrote: > Hello, > > My name is Alex LeNail and I'm a researcher at MIT. I'm looking to set up > a Galaxy server, and our lab has a great deal of Microsoft Azure credits, > so we were hoping to deploy Galaxy on there. > > Where should I start if I wanted to set that up? I haven't really found > any resources online with any description of how that might be > accomplished. > > Thanks, > > --Alex > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Galaxy on Azure
Hello, My name is Alex LeNail and I'm a researcher at MIT. I'm looking to set up a Galaxy server, and our lab has a great deal of Microsoft Azure credits, so we were hoping to deploy Galaxy on there. Where should I start if I wanted to set that up? I haven't really found any resources online with any description of how that might be accomplished. Thanks, --Alex ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Dependencies for sudo scripts/drmaa_external_runner.py ?
Hello all, QUESTION: When submitting jobs to the cluster as the real user, how should sudo scripts/drmaa_external_runner.py be told which Python to use, and how would it activate the venv if needed for the DRMAA dependency? BACKGROUND: We're currently trying Galaxy out on a new CentOS 6 VM, with matching CentOS 6 cluster, where jobs are submitted to SGE via DRMAA and run as the Linux user rather than a generic Galaxy Linux user account. This is documented on the wiki here: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster This all seemed to be working under Galaxy v15.10 (using eggs), but we're now targeting the recently released Galaxy v16.10 (using wheels) instead and have run into problems. https://github.com/galaxyproject/galaxy/issues/1596 Because Galaxy is deprecating support for Python 2.6 (the default bundled with CentOS 6), we're now using a local copy of Python 2.7 (compiled from source) on a shared mount. This mismatch seems to be the root cause of the problem I will now describe. During job submission to SGE, Galaxy will attempt to run a command like this: $ sudo scripts/drmaa_external_runner.py 1005 /mnt/shared/galaxy/galaxy-dist/database/sge/132.jt_json From the terminal output from ./run.sh we'd see: RuntimeError: External_runjob failed (exit code 1) Child process reported error: Traceback (most recent call last): File "/mnt/shared/galaxy/galaxy-dist/scripts/drmaa_external_runner.py", line 15, in import drmaa ImportError: No module named drmaa Although a drmaa wheel was installed within the Python 2.7 virtual environment under ~/galaxy-dist/.venv Galaxy makes no attempt to activate the venv for scripts/drmaa_external_runner.py We then installed DRMAA under our local copy of Python 2.7, and realised sudo scripts/drmaa_external_runner.py was not even using this copy of Python. Changing the hash bang line was a crude way to solve that (see below). This in turn lead to finding $DRMAA_LIBRARY_PATH and $SGE_ROOT were not set in the sudo environment. Again, you can hack around this by modifying scripts/drmaa_external_runner.py (see below). In our case, I suspect the least invasive change would be to install the DRMAA libraries under the system provided Python 2.6, and let sudo scripts/drmaa_external_runner.py execute that way. We still need to solve why sudo scripts/drmaa_external_runner.py does not see $DRMAA_LIBRARY_PATH and $SGE_ROOT but we have some clues to follow up on that: http://stackoverflow.com/questions/257616/sudo-changes-path-why Peter P.S. See also https://twitter.com/pjacock/status/704335582651162624 -- Here's our workaround diff - lots of hard coded strings, not portable at all but it worked for testing/debugging: $ git diff scripts/drmaa_external_runner.py diff --git a/scripts/drmaa_external_runner.py b/scripts/drmaa_external_runner.py index a1474fe..61d2383 100755 --- a/scripts/drmaa_external_runner.py +++ b/scripts/drmaa_external_runner.py @@ -1,5 +1,6 @@ +#!/mnt/shared/galaxy/apps/python/2.7.11/bin/python +#Was #!/usr/bin/env python - """ Submit a DRMAA job given a user id and a job template file (in JSON format) defining any or all of the following: args, remoteCommand, outputPath, @@ -12,8 +13,15 @@ import os import pwd import sys +# Hack +# print "$DRMAA_LIBRARY_PATH is %s" % os.environ.get('DRMAA_LIBRARY_PATH') +# print "$SGE_ROOT is %s" % os.environ.get('SGE_ROOT') +os.environ['DRMAA_LIBRARY_PATH'] = '/mnt/sge/lib/lx-amd64/libdrmaa.so' +os.environ['SGE_ROOT'] = '/mnt/sge' + import drmaa DRMAA_jobTemplate_attributes = [ 'args', 'remoteCommand', 'outputPath', 'errorPath', 'nativeSpecification', 'workingDirectory', 'jobName', 'email', 'project' ] ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] tmp directory
Hi All, Where are the temporary files stored if the line below is commented out? # Temporary files are stored in this directory. new_file_path = database/tmp Thank you. Amos Raphenya Bioinformatics Software Developer Department of Biochemistry & Biomedical Sciences McMaster University, MDCL 2317 p: (905) 525-9140 ext: 22787 a: 1280 Main St W.,Hamilton, Ontario, Canada L8S 4K1 e: raphe...@mcmaster.ca skype: amos_raphenya ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Downgrade tool upload tool
Thanks for clarifying Makis, I think this change is likely due to changes in Galaxy rather than in the upload tool itself. On our (slightly out of date) Galaxy or the public server, that URL no longer loads the original tool form - you get the new drag-and-drop upload interface. I would guess some of the special case code like this matters: https://github.com/galaxyproject/galaxy/blob/fe6d22e1a7e4e6d38b15f4128554783765bc69a1/client/galaxy/scripts/mvc/tool/tools.js#L525 Probably Sam (Aysam) can give you more practical advice (CC'd)? Peter On Mon, Feb 29, 2016 at 12:29 PM, Makis Ladoukakiswrote: > I am making some clickable buttons in the home page of my instance through a > java script that is called from the welcome.html file. These buttons > correspond to my most used tools. For example I have a section like this: > > > > > > > src="feimg/GetData.gif"/> > > href="http://myserver/galaxy/tool_runner?tool_id=upload1; > target="_parent"> height=300%/> > href="http://myserver/galaxy/library/browse_libraries; target="_parent"> src="feimg/Libraries.gif" width=250% height=300%/> > > > > > > > indicating two buttons one for the upload tool and one for the libraries. > The libraries button work but when I click on the button for the upload tool > the webpage just refreshes instead of opening the tool. This worked > perfectly with the previous version of the upload tool and I was thinking > how to get it back. > > Thank you, > Makis > > >> Date: Mon, 29 Feb 2016 11:39:26 + >> Subject: Re: [galaxy-dev] Downgrade tool upload tool >> From: p.j.a.c...@googlemail.com >> To: makis4e...@hotmail.com >> CC: j...@galaxyproject.org; galaxy-...@lists.bx.psu.edu; >> galaxy-dev@lists.galaxyproject.org > >> >> Hi Makis, >> >> The upload tool is a little special in that it is used internally by >> Galaxy and doesn't entirely behave like a normal tool. >> >> What are you trying to achieve by "downgrading" it? >> >> Peter >> >> On Mon, Feb 29, 2016 at 11:29 AM, Makis Ladoukakis >> wrote: >> > What about if I used a second "tool upload" tool? I took the python >> > script >> > from an earlier version and named it upload_old.py and did the same for >> > the >> > xml file (upload.xml -> upload_old.xml) while changing the latter to >> > call >> > the new python script. Nevertheless when I did that and tried to run it >> > I >> > got a popup window with the following error: >> > >> > Tool request failed >> > >> > Uncaught exception in exposed API method: >> > >> > >> > Is there a way to make this work? Or maybe another workaround? >> > >> > Thank you in advance, >> > Makis >> > >> > >> > >> > >> > From: j...@galaxyproject.org >> > Date: Fri, 26 Feb 2016 10:22:44 -0800 >> > Subject: Re: [galaxy-dev] Downgrade tool upload tool >> > To: makis4e...@hotmail.com >> > CC: galaxy-...@lists.bx.psu.edu; galaxy-dev@lists.galaxyproject.org >> > >> > >> > Hello, >> > >> > The Upload tool is a direct component of Galaxy. To use an earlier >> > version >> > of the tool, I believe that you would have to run an earlier version of >> > Galaxy. >> > >> > Others here can amend this, if they know of an alternative. >> > >> > Thanks, >> > >> > Jen >> > >> > -- >> > Jennifer Hillman-Jackson >> > Galaxy Application Support >> > http://usegalaxy.org >> > http://galaxyproject.org >> > http://biostar.usegalaxy.org >> > >> > >> > On Fri, Feb 26, 2016 at 4:01 AM, Makis Ladoukakis >> > >> > wrote: >> > >> > Hello everyone, >> > >> > I was wondering whether it's possible to downgrade the upload tool to >> > the >> > previous version(1.1.4). >> > >> > I copied the upload.py and upload.xml scripts from a previous galaxy >> > version >> > to the tools/data_source/ directory and reloaded the tool as admin but >> > it >> > hasn't changed even when I restarted the server. >> > >> > Do I have to change something else in addition? >> > >> > Thank you, >> > Makis >> > >> > ___ >> > Please keep all replies on the list by using "reply all" >> > in your mail client. To manage your subscriptions to this >> > and other Galaxy lists, please use the interface at: >> > https://lists.galaxyproject.org/ >> > >> > To search Galaxy mailing lists use the unified search at: >> > http://galaxyproject.org/search/mailinglists/ >> > >> > >> > >> > ___ >> > Please keep all replies on the list by using "reply all" >> > in your mail client. To manage your subscriptions to this >> > and other Galaxy lists, please use the interface at: >> > https://lists.galaxyproject.org/ >> > >> > To search Galaxy mailing lists use the unified search at: >> > http://galaxyproject.org/search/mailinglists/
[galaxy-dev] nginx data upload request never reaches /api/tools
Hi all, I set up the data upload using nginx as described in the wiki (https://wiki.galaxyproject.org/Admin/Config/nginxProxy, using the subdirectory /galaxy). The uploaded file also appears inside the uploade_store folder on the galaxy server, but galaxy does not add the file. Investigating the log files, I found the corresponding lines in the nginx.error.log: # Post request is received 2016/02/29 12:55:37 [notice] 13156#0: *4 "^(GET|HEAD|POST)$" matches "POST", client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: http://172.16.12.122/galaxy/ # Upload starts 2016/02/29 12:55:37 [info] 13156#0: *4 started uploading file "boxes.txt" to "/mnt/idrive1/database/tmp/upload_store/0020132707" (field "files_0|file_data", content type "text/plain"), client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: http://172.16.12.122/galaxy/ # Upload done, the file is present in the given path 2016/02/29 12:55:37 [info] 13156#0: *4 finished uploading file "boxes.txt" to "/mnt/idrive1/database/tmp/upload_store/0020132707", client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: "http://172.16.12.122/galaxy/; # Redirection starts? 2016/02/29 12:55:37 [notice] 13156#0: *4 "^(GET|HEAD|POST)$" matches "POST", client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: "http://172.16.12.122/galaxy/; 2016/02/29 12:55:37 [notice] 13156#0: *4 "nginx_redir=([^&]+)" does not match "", client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: "http://172.16.12.122/galaxy/; 2016/02/29 12:55:37 [notice] 13156#0: *4 "" matches "/_upload_done", client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: "http://172.16.12.122/galaxy/; # Redirection should be done here 2016/02/29 12:55:37 [notice] 13156#0: *4 rewritten data: "/galaxy/api/tools", args: "", client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: "http://172.16.12.122/galaxy/; # ModSecurity seems to block the redirect? 2016/02/29 12:55:37 [info] 13156#0: *4 ModSecurity: invalid "Content-Length" header, client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: "http://172.16.12.122/galaxy/; 2016/02/29 12:55:37 [notice] 13156#0: *4 "^(GET|HEAD|POST)$" matches "GET", client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: "http://172.16.12.122/galaxy/; A ModSecurityEnabled off; within the /_upload and /_upload_done of the nginx config did not help. Any idea why the Content-Length header causes problems and is blocked by modSec? Any workaround for ModSec to accept everything that goes from internal to internal? Thanks for your help! Best, Jens ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Downgrade tool upload tool
Hi Makis, The upload tool is a little special in that it is used internally by Galaxy and doesn't entirely behave like a normal tool. What are you trying to achieve by "downgrading" it? Peter On Mon, Feb 29, 2016 at 11:29 AM, Makis Ladoukakiswrote: > What about if I used a second "tool upload" tool? I took the python script > from an earlier version and named it upload_old.py and did the same for the > xml file (upload.xml -> upload_old.xml) while changing the latter to call > the new python script. Nevertheless when I did that and tried to run it I > got a popup window with the following error: > > Tool request failed > > Uncaught exception in exposed API method: > > > Is there a way to make this work? Or maybe another workaround? > > Thank you in advance, > Makis > > > > > From: j...@galaxyproject.org > Date: Fri, 26 Feb 2016 10:22:44 -0800 > Subject: Re: [galaxy-dev] Downgrade tool upload tool > To: makis4e...@hotmail.com > CC: galaxy-...@lists.bx.psu.edu; galaxy-dev@lists.galaxyproject.org > > > Hello, > > The Upload tool is a direct component of Galaxy. To use an earlier version > of the tool, I believe that you would have to run an earlier version of > Galaxy. > > Others here can amend this, if they know of an alternative. > > Thanks, > > Jen > > -- > Jennifer Hillman-Jackson > Galaxy Application Support > http://usegalaxy.org > http://galaxyproject.org > http://biostar.usegalaxy.org > > > On Fri, Feb 26, 2016 at 4:01 AM, Makis Ladoukakis > wrote: > > Hello everyone, > > I was wondering whether it's possible to downgrade the upload tool to the > previous version(1.1.4). > > I copied the upload.py and upload.xml scripts from a previous galaxy version > to the tools/data_source/ directory and reloaded the tool as admin but it > hasn't changed even when I restarted the server. > > Do I have to change something else in addition? > > Thank you, > Makis > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Downgrade tool upload tool
What about if I used a second "tool upload" tool? I took the python script from an earlier version and named it upload_old.py and did the same for the xml file (upload.xml -> upload_old.xml) while changing the latter to call the new python script. Nevertheless when I did that and tried to run it I got a popup window with the following error: Tool request failedUncaught exception in exposed API method: Is there a way to make this work? Or maybe another workaround? Thank you in advance, Makis From: j...@galaxyproject.org Date: Fri, 26 Feb 2016 10:22:44 -0800 Subject: Re: [galaxy-dev] Downgrade tool upload tool To: makis4e...@hotmail.com CC: galaxy-...@lists.bx.psu.edu; galaxy-dev@lists.galaxyproject.org Hello, The Upload tool is a direct component of Galaxy. To use an earlier version of the tool, I believe that you would have to run an earlier version of Galaxy. Others here can amend this, if they know of an alternative. Thanks, Jen --Jennifer Hillman-JacksonGalaxy Application Supporthttp://usegalaxy.org http://galaxyproject.org http://biostar.usegalaxy.org On Fri, Feb 26, 2016 at 4:01 AM, Makis Ladoukakiswrote: Hello everyone, I was wondering whether it's possible to downgrade the upload tool to the previous version(1.1.4). I copied the upload.py and upload.xml scripts from a previous galaxy version to the tools/data_source/ directory and reloaded the tool as admin but it hasn't changed even when I restarted the server. Do I have to change something else in addition? Thank you, Makis ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/