Re: [galaxy-dev] Tool shed environment dependency - HOW TO do this?
Hi Pieter, you should probably add: action type=set_environment_for_install repository name=prims_metabolomics_r_dependencies owner=pieterlukasse package name=R_bioc_metams version=3.1.1 / /repository /action before using the R_ROOT_DIR environment variable. Best, Nicola Il 2014-11-06 11:28 Lukasse, Pieter ha scritto: Hi Bjoern, I tried your suggestion, but it didn't work. So now I have split up the R part from the Bioconductor part (blue and yellow below, respectively). But the problem is that the install part executes before the package part is finished and the $R_ROOT_DIR reference also doesn't seem to work as I get the following error : /bin/sh: 1: /bin/Rscript: not found . This R_ROOT_DIR is set by the package script in my other tool_dependency.xml but is not visible in the tool_dependency.xml below. But maybe I misunderstood what you meant by You can now access R_ROOT_DIR from any other tool_dependency file ...? tool_dependency !-- see also http://wiki.galaxyproject.org/ToolShedToolFeatures for syntax help -- package name=R_bioc_metams version=3.1.1 repository changeset_revision=0af753942e7b name=prims_metabolomics_dependencies owner=pieterlukasse prior_installation_required=True toolshed=http://testtoolshed.g2.bx.psu.edu; / install version=1.0 actions_group !-- the Bioconductor and metaMS part -- action type=shell_commandwget -P $INSTALL_DIR http://testtoolshed.g2.bx.psu.edu/repos/pieterlukasse/prims_metabolomics/raw-file/tip/INSTALL.r/action action type=shell_command$R_ROOT_DIR/bin/Rscript $INSTALL_DIR/INSTALL.r/action /actions_group /install /package readme This dependency: Ensures R 3.1.1 installation is triggered (via dependency). Ensures Bioconductor 3.0 and package metaMS, multtest and snow are installed. /readme /tool_dependency Thanks, Pieter -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Lukasse, Pieter Sent: donderdag 6 november 2014 10:02 To: 'Björn Grüning'; galaxy-dev@lists.bx.psu.edu; Dave Bouvier Subject: Re: [galaxy-dev] Tool shed environment dependency - HOW TO do this? Hi Bjoern, Thanks for this reply. I will have to try referring to the R_ROOT_DIR from another tool_dependency. If it doesn't work I'll come back ;) Regards, Pieter -Original Message- From: Björn Grüning [mailto:bjoern.gruen...@gmail.com [1]] Sent: donderdag 30 oktober 2014 11:25 To: galaxy-dev@lists.bx.psu.edu [2]; Lukasse, Pieter; Dave Bouvier Subject: Re: [galaxy-dev] Tool shed environment dependency - HOW TO do this? Hi Pieter, Am 30.10.2014 um 11:15 schrieb Lukasse, Pieter: Hi, I have a Bioconductor package which depends on R 3.1.1 , so I think I cannot use the setup_r_environment trick. Why? Do you need a new release? The latest current release in the Tool Shed is R 3.1.0. If this is not enough we need to poke Dave to build a new package. But this way would be the preferred way. I already got a tool_dependency.xml working that installs R 3.1.1 and the necessary Bioconductor packages using bioclite method (see attachment). Now my question is: * I would like to split up this into two steps as I don't want to trigger the compilation of new R environment every time I when I need to just update the Bioconductor packagethe question is: how to do such things in general? How can I access the INSTALL_DIR of another tool from within another tool_dependency.xml? If I can do this, then my problem is solved. If you really want to build your R packages by your own. Have a look at https://github.com/bgruening/galaxytools/blob/master/packages/package_r_3_0_3/tool_dependencies.xml [3] You will find an exhaustive installation instruction how to build R properly. Also note that we export a variable called R_ROOT_DIR that is set to $INSTALL_DIR. You can now access R_ROOT_DIR from any other tool_dependency file ... Hope this helps, Bjoern P.S. Do you mind asking this question on biostar again, I think this is a very nice question that can be of interest to a lot more people. Thanks! Pieter Lukasse Wageningen UR, Plant Research International Department of Bioinformatics (Bioscience) Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB, Wageningen, the Netherlands T: +31-317481122; M: +31-628189540; skype: pieter.lukasse.wur http://www.pri.wur.nlhttp://www.pri.wur.nl/ [4] ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ [5] To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ [6] ___ Please keep all replies on the list by using reply all
Re: [galaxy-dev] Data uploaded with new upload tool doesn't get added to history
Hi Dannon, so the very last line (set $dst /galaxy/tool_runner/index;) on the wiki page https://wiki.galaxyproject.org/Admin/Config/nginxProxy is still wrong? Ciao, Nicola Il giorno ven, 10/10/2014 alle 08.50 -0400, Dannon Baker ha scritto: Just to follow up on this thread -- we resolved this over IRC. It was a proxy configuration issue with nginx's upload_config settings. Correct settings are now on the wiki, but the previous rule for: location _upload_done : set $dst /tool_runner/index; is was causing a redirect and preventing communication with the tools API, which the new upload form works. -Dannon On Thu, Oct 9, 2014 at 3:00 PM, Peter van Heusden p...@sanbi.ac.za wrote: Hey there Using the new upload tool my data uploads but never appears in the history. All I can see in paster.log when I do an upload is: 10.8.0.2 - - [09/Oct/2014:20:43:58 +0200] POST /tool_runner/index HTTP/1.0 200 - https://phillipl.sanbi.ac.za/root; Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:32.0) Gecko/20100101 Firefox/32.0 10.8.0.2 - - [09/Oct/2014:20:43:58 +0200] GET /api/histories/f597429621d6eb2b/contents HTTP/1.0 200 - https://phillipl.sanbi.ac.za/root; Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:32.0) Gecko/20100101 Firefox/32.0 10.8.0.2 - - [09/Oct/2014:20:43:58 +0200] GET /api/histories/f597429621d6eb2b HTTP/1.0 200 - https://phillipl.sanbi.ac.za/root; Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:32.0) Gecko/20100101 Firefox/32.0 No errors in the javascript console, and this happens with both Chrome and Firefox (on Linux), code is latest galaxy-central. Any ideas where to debug? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy Bioblend option for importing dataset into a library?
Il giorno gio, 02/10/2014 alle 16.50 -0700, Dooley, Damion ha scritto: I see Galaxy API has a feature to import a history dataset into the library (in copy_hda_to_ldda() fn from GCC2013 training day course). Is this available as well via Bioblend? Latest docs don't seem to include this feature. It would be the opposite of Bioblend's upload_dataset_from_library(history_id, lib_dataset_id) ) Hi Damion, indeed this feature is not in BioBlend yet, but I have it in my TODO list because we also need it. Hopefully I'll soon have some time to implement the method, test it and open a PR. Best, Nicola Objective is to get customized blast indexes into library that way for shared use. Or have them actually exist outside galaxy, and linked in. Regards, Damion Hsiao lab, BC Public Health Microbiology Reference Laboratory, BC Centre for Disease Control 655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed Categories
Hi Khalid, I'd suggest you to open a Trello card for this: http://galaxyproject.org/trello/ Best, Nicola Il giorno ven, 03/10/2014 alle 17.30 +, Alam, Khalid K. (MU-Student) ha scritto: Hi; I’m getting ready to submit several (5) tools for processing data related to combinatorial selection technologies (aptamer and (deoxy)ribozyme selections, phage display, etc.) and would like to propose a new category for the Tool Shed entitled “Combinatorial Selections” where these repositories can be published. Khalid Kamal Alam University of Missouri Department of Biochemistry 415 Life Sciences Center 1201 Rollins St. Columbia, MO 65211 mailto:khalidka...@mail.missouri.eduKhalidKAlammailto:khalidka...@mail.missouri.edu@mail.missouri.edumailto:kka...@mail.missouri.edu @BiochemPhD ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Updated Freebayes Wrapper - Bugs
Il 2014-06-30 18:40 Anton Nekrutenko ha scritto: Lance: Here is github URL: https://github.com/nekrut/freebayes [3] On Wed, Jun 25, 2014 at 2:40 PM, Lance Parsons lpars...@princeton.edu [4] wrote: Thanks for the major update of the Freebayes wrapper, excellent! I've run into two issues, however. 1) When using set allelic scope I get the following error: Fatal error: Exit code 1 () freebayes: unrecognized option `--min-repeat-length' did you mean --min-repeat-size ? This bug (and other 2) would be fixed by this pull request: https://github.com/nekrut/freebayes/pull/1 Best, Nicola 2) When using a vcf file as input I get the following error: Fatal error: Exit code 1 () open: No such file or directory [bgzf_check_bgzf] failed to open the file: input_variant_vcf.vcf.gz [tabix++] was bgzip used to compress this file? input_variant_vcf.vcf.gz I'd be happy to help resolve these. Is there a bitbucket repo somewhere to submit pull requests? -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ [1] To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ [2] -- Anton Nekrutenko Associate Professor Dept. of Biochemistry and Molecular Biology http://nekrut.bx.psu.edu [5] (814) 826-3051 Links: -- [1] http://lists.bx.psu.edu/ [2] http://galaxyproject.org/search/mailinglists/ [3] https://github.com/nekrut/freebayes [4] mailto:lpars...@princeton.edu [5] http://nekrut.bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] fix for GATK2 wrapper from IUC
Hi Geert, thanks for the bug report, I've fixed the bug upstream: https://github.com/bgruening/galaxytools/commit/cab6d1e84d440b7c73e99adc2bcb2c3c64c5b2c5 Björn, can you release an update on the Tool Shed? I think we have accumulated a lot of fixes for GATK2 wrappers already. Nicola Il 2014-06-27 10:15 Geert Vandeweyer ha scritto: Hi, There is a missing parameter in the unified genotyer config from the iuc gatk2 wrapper: The following line should be added around line 204 (just before /expand/inputs): param name=sample_ploidy type=integer value=2 label=Sample Ploidy (number of allles). For tumour, set to 2x the number of expected subclones help=-ploidy,--sample_ploidy / Without this line, activating the advanced analysis options causes the job preparation to fail because there is a reference to it in the command section gatk2 wrapper used: version : 8bcc13094767 hg tip: changeset: 3:2553f84b8174 tag: tip user:iuc date:Wed Feb 19 04:39:38 2014 -0500 summary: Uploaded Best, Geert -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandewe...@ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/in/geertvandeweyer ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] monitoring jobs start time
Il 2014-05-19 04:47 neil.burd...@csiro.au ha scritto: Hi, I am currently using the reports tool to see what jobs have been executed etc ... When you look at the information for each job it states the time the job was put on the queue (creation time) not the actual start time of the job. Do you know if the start time of the job is stored somewhere? Hi Neil, take a look at this pull request, which was merged recently in galaxy-central and will be available in the next Galaxy release: https://bitbucket.org/galaxy/galaxy-central/pull-request/352/implement-plugin-framework-and-plugins-for/diff Best, Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Jobs deleted staying in 'dr' status
Il 2013-10-11 17:21 Sytchev, Ilya ha scritto: On 9/12/13 10:35 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Sep 12, 2013 at 2:01 PM, Mathieu Bahin mathieu.ba...@irisa.fr wrote: Hi all, We have been developing our own Galaxy instance for a while now. We have a cluster on which the job are sent to be executed, it is managed through SGE. Usually, communication between SGE and DRMAA is ok and we don't have any problem with that. When a job is deleted by the user, most of the times, the job disappears but sometimes, we don't know why, the job stays and has the status 'dr' within SGE. If we don't kill it 'manually', it stays forever. It is not always the same tools which produces this error. Have you any idea why how manage it ? I have noticed problem with our DRMMA/SGE setup where a user can cancel a large job (using the job splitter in at least some cases), but Galaxy does not seem to cancel the jobs on the cluster. I've not tried to diagnose this yet - it could be a similar issue though. Also, in our DRMAA/LSF setup (using a fork of the latest galaxy-dist) jobs generated by the current workflow step continue running on the cluster after history is deleted. Ilya Hi Ilya, I also see this behaviour with DRMAA/GridEngine. I think this has been already reported: https://trello.com/c/1whC9did/245-currently-running-jobs-in-deleted-histories-should-be-killed Please upvote it! Best, Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Launching a workflow via API, multiple params for one tool
Il 2014-04-09 22:18 John Marmaduke Eppley ha scritto: As far as I can figure out, the parameter map for running a workflow via the API is supposed to look like this: {'blastn': {'param': 'evalue', 'value': '1e-06’}} This doesn’t seem to allow for multiple parameters to a single tool. Is there a way to submit multiple parameters that I’m missing? For example: {'blastn': {'param': 'evalue', 'value': '1e-06’, 'param': ‘identity_cutoff’, ‘value’: ’70’}} …will not work. The dict() container doesn’t allow duplicate keys, it just overwrites the values. Hi John, you are using the old and deprecated syntax, you can use instead: {'blastn': {'evalue': '1e-06', 'identity_cutoff': 70}} For more details look at _update_step_parameters() function documentation: https://bitbucket.org/galaxy/galaxy-dist/src/tip/lib/galaxy/webapps/galaxy/api/workflows.py Or look at this complete example: https://github.com/crs4/bioblend/blob/objects/docs/examples/w5_galaxy_api.py Moreover, you may consider using the more high-level bioblend API: http://bioblend.readthedocs.org/en/latest/ Best, Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] enforcing minimum length on text params
Hi Ravi, for that you need a regex validator: https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/parameters/validation.py#cl-27 e.g. validator type=regex message=This field should contain some non-whitespace character.*\S/validator Best, Nicola Il giorno gio, 10/04/2014 alle 10.20 -0400, Sanka, Ravi ha scritto: Hi Nicola, Perchance, do you know what type of validator I can put to ensure that the string input is does not consist of only whitespace? Such a string is still invalid for my tool, but counts as a string. -- Ravi Sanka ICS Sr. Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 -- On 4/9/14 2:35 PM, Sanka, Ravi rsa...@jcvi.org wrote: Hi Nicola, Thank you. I tried the validator as you illustrated, and it worked perfectly. -- Ravi Sanka ICS Sr. Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 -- On 4/9/14 11:06 AM, Nicola Soranzo sora...@crs4.it wrote: Il 2014-04-09 17:00 Sanka, Ravi ha scritto: Greetings, I am adding a new tool to our in-house Galaxy, which has a few string inputs. I have made each one a text param in the XML, but want to ensure that the user does not leave any empty when he executes the tool. How can I do this? I tried setting the optional field of each text param to false, but that doesn't work since, technically, an empty string still counts as a string. Is there a minimum length setting I can enforce? Hi Ravi, you should use a validator, as in the following example: param name=rgid type=text label=Read group identifier (ID) validator type=empty_field / /param Best, Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Running a different command based on selection in the tool
Hi Saba and Jens, : are not needed, they are optional, see http://www.cheetahtemplate.org/docs/users_guide_html/users_guide.html#SECTION00068 Maybe it's a problem with the left quotes around Use_Snapshot in the #if line? Nicola Il 2014-03-20 06:26 Keilwagen, Jens ha scritto: Hi Saba, there is a syntax error in the line of the #if. Please add : at the end and report whether the error still occurs. All the best, Jens VON: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] IM AUFTRAG VON Saba Sehrish GESENDET: Mittwoch, 19. März 2014 23:07 AN: galaxy-dev@lists.bx.psu.edu BETREFF: [galaxy-dev] Running a different command based on selection in the tool Hi all, I am trying to write a tool that based on user criteria will run a different script. In the code below, If I select Use_Snapshot, I see the right interface but somehow galaxy tries to execute the #else part and look for m input parameter. The example I have seen in documentation is using the script with same name in both if and else and the number of input arguments is the same as well. Any suggestions/ideas why always #else part gets executed although right interface appears based on user selection. Thanks, Saba command #if $source.source_select==Use_Snapshot cM-snapshot.sh $input $output #else cM.sh $m $b $ns $w $s $z $output #end if /command inputs conditional name=source param name=source_select type=select label=Specify the input option value=Use_SnapshotUse Snapshot/option option value=Set_ValuesSet Values/option /param when value=Use_Snapshot param name=input type=data format=dbm size=100 label=Input Snapshot/ sanitizer sanitize=False/ /when when value=Set_Values param name=m label=Omega_m h^2 type=float value=0.1279 min=0.120 max=0.155/ param name=b label=Omega_b h^2 type=float value=0.0232 min=0.0215 max=0.0235/ param name=ns label=n_s type=float value=0.8629 min=0.85 max=1.05/ param name=w label=w type=float value=-1.184 min=-1.30 max=-0.70/ param name=s label=sigma_8 type=float value=0.6159 min=0.61 max=0.9/ param name=z label=z type=float value=1.0 min=0.0 max=1.0/ sanitizer sanitize=False/ /when /conditional /inputs ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] GATK 3.1
Il 2014-03-18 14:41 Berner, Thomas ha scritto: Hi, I've seen there is a new version (3.1-1) of GATK available at http://www.broadinstitute.org/gatk/download [1] . Are there any plans of getting this version into Galaxy in the nearer future? Hi Thomas, as you probably know there is a wrapper for GATK v.2 on the Tool Shed: http://toolshed.g2.bx.psu.edu/view/iuc/gatk2 which is developed and maintained by Jim Johnson, Björn Grüning and me. GATK v.3 has some new tools and also some changes to logging which prevent the use of gatk2 wrapper with v.3 , so we are planning to create a new gatk3 Tool Shed repository sooner or later, no time frame decided yet. You can follow development, and contribute if you want, at this git repository: https://github.com/bgruening/galaxytools Best, Nicola -- Nicola Soranzo, Ph.D. Bioinformatics Program, CRS4 Loc. Piscina Manna, 09010 Pula (CA), Italy http://www.bioinformatica.crs4.it/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] filter tag in output, multiple criteria
Il 2014-01-31 11:52 Geert Vandeweyer ha scritto: Hi, I'm looking for the correct syntax to achieve the following: data type=data format=txt name=loh label=${tool.name} result on ${on_string} (loh) filterouttype == 0 || outttype == 2 /filter /data data type=data format=txt name=cnv label=${tool.name} result on ${on_string} (loh) filterouttype == 1 || outttype == 2 /filter /data I tried the above, with and without parentheses, but it creates the second dataset also when outtype == 0. Is this type of dual filtering possible? I don't know if this is the reason, but you seem to have a typo: outttype == 2 should be outtype == 2 Best, Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Import workflows via API
Hi Neil, can you retry after applying this commit from my new pull request? https://bitbucket.org/nsoranzo/galaxy-central/commits/11597a078e6505087064315567ad3357c2d40c56 -- Nicola Soranzo Bioinformatics Program, CRS4 Loc. Piscina Manna, 09010 Pula (CA), Italy http://www.bioinformatica.crs4.it/ neil.burd...@csiro.au ha scritto: Hi Nicola, I've merged your changes into my version of the galaxy code (slightly older than galaxy-central) I believe, however, I get the following error when I try and execute your script: Traceback (most recent call last): File get_wfs.py, line 13, in module common.post(api_key, 'http://barium-rbh:9100/extras/api/workflows/import', data) File /home/galaxy/milxcloud/scripts/api/common.py, line 48, in post return simplejson.loads( urllib2.urlopen( req ).read() ) File /usr/lib/python2.7/urllib2.py, line 126, in urlopen return _opener.open(url, data, timeout) File /usr/lib/python2.7/urllib2.py, line 406, in open response = meth(req, response) File /usr/lib/python2.7/urllib2.py, line 519, in http_response 'http', request, response, code, msg, hdrs) File /usr/lib/python2.7/urllib2.py, line 444, in error return self._call_chain(*args) File /usr/lib/python2.7/urllib2.py, line 378, in _call_chain result = func(*args) File /usr/lib/python2.7/urllib2.py, line 527, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 500: Internal Server Error so I printed out the some lines in common.py def post( api_key, url, data ): # Do the actual POST. url = make_url( api_key, url ) print URL is %s % url URL is http://barium-rbh:9100/extras/api/workflows/import?key=34ee80757f0d03de36e33a1676d245e4 the script is: import sys #sys.path.insert(1, 'milxcloud/scripts/api') import common api_key = '34ee80757f0d03de36e33a1676d245e4' workflows = common.get(api_key, 'http://barium-rbh:9100/api/workflows?show_published=True') print workflows published_workflow_ids = [str(workflow[u'id']) for workflow in workflows if bool(workflow[u'published'])] print published_workflow_ids for pw_id in published_workflow_ids: data = {} data['workflow_id'] = pw_id common.post(api_key, 'http://barium-rbh:9100/extras/api/workflows/import', data) and I run it from ~/scripts/api The output from the script is : python get_wfs.py [{'name': 'SUVR', 'tags': [], 'url': '/extras/api/workflows/f2db41e1fa331b3e', 'published': True, 'model_class': 'StoredWorkflow', 'id': 'f2db41e1fa331b3e'}, {'name': 'processSUVRData', 'tags': [], 'url': '/extras/api/workflows/f597429621d6eb2b', 'published': True, 'model_class': 'StoredWorkflow', 'id': 'f597429621d6eb2b'}] ['f2db41e1fa331b3e', 'f597429621d6eb2b'] URL is http://barium-rbh:9100/extras/api/workflows/import?key=34ee80757f0d03de36e33a1676d245e4 Any ideas why the import maybe failing? Thanks Neil From: Nicola Soranzo [sora...@crs4.it] Sent: Monday, January 20, 2014 9:03 PM To: Burdett, Neil (CCI, Herston - RBWH) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Import workflows via API Hi Neil, to import all published workflows you can use a script like this: import sys sys.path.insert(1, 'galaxy-central/scripts/api') import common api_key = 'YOUR_USER_API_KEY' workflows = common.get(api_key, 'http://YOUR_SERVER/api/workflows?show_published=True') published_workflow_ids = [str(workflow[u'id']) for workflow in workflows if bool(workflow[u'published'])] for pw_id in published_workflow_ids: data['workflow_id'] = pw_id common.post(api_key, 'http://YOUR_SERVER/api/workflows/import', data) Nicola Il 2014-01-19 03:12 neil.burd...@csiro.au ha scritto: Hi Nicola, that is exactly what I'm looking for, however, how do I execute the script/tool? I would like to import all published workflows. What is the name of the script to run and the arguments ? Can you give an example please? Thanks again Neil Date: Fri, 17 Jan 2014 11:36:03 +0100 From: Nicola Soranzo sora...@crs4.it To: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Import workflows via API Message-ID: d834d247437269704a26e8bbf8f67...@crs4.it Content-Type: text/plain; charset=UTF-8; format=flowed Il 2014-01-17 06:45 neil.burd...@csiro.au ha scritto: Hi, I execute workflows via the API. However, if I want another user to use my workflows, I can publish my workflows, but the new user then has to go on to the web browser and import this workflow. Is there a method/script which I can call via the API which can import all available (published) workflows so the user doesn't have to click a button on the web browser import workflow ? Thanks for any help Hi Neil, you are very lucky, just yesterday my pull request implementing exactly this has been merged in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/pull-request/298/api-display-and-import-workflows-shared
Re: [galaxy-dev] Import workflows via API
Hi Neil, to import all published workflows you can use a script like this: import sys sys.path.insert(1, 'galaxy-central/scripts/api') import common api_key = 'YOUR_USER_API_KEY' workflows = common.get(api_key, 'http://YOUR_SERVER/api/workflows?show_published=True') published_workflow_ids = [str(workflow[u'id']) for workflow in workflows if bool(workflow[u'published'])] for pw_id in published_workflow_ids: data['workflow_id'] = pw_id common.post(api_key, 'http://YOUR_SERVER/api/workflows/import', data) Nicola Il 2014-01-19 03:12 neil.burd...@csiro.au ha scritto: Hi Nicola, that is exactly what I'm looking for, however, how do I execute the script/tool? I would like to import all published workflows. What is the name of the script to run and the arguments ? Can you give an example please? Thanks again Neil Date: Fri, 17 Jan 2014 11:36:03 +0100 From: Nicola Soranzo sora...@crs4.it To: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Import workflows via API Message-ID: d834d247437269704a26e8bbf8f67...@crs4.it Content-Type: text/plain; charset=UTF-8; format=flowed Il 2014-01-17 06:45 neil.burd...@csiro.au ha scritto: Hi, I execute workflows via the API. However, if I want another user to use my workflows, I can publish my workflows, but the new user then has to go on to the web browser and import this workflow. Is there a method/script which I can call via the API which can import all available (published) workflows so the user doesn't have to click a button on the web browser import workflow ? Thanks for any help Hi Neil, you are very lucky, just yesterday my pull request implementing exactly this has been merged in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/pull-request/298/api-display-and-import-workflows-shared-by/diff and is also available in BioBlend thanks to my colleague Simone Leo: https://github.com/afgane/bioblend/pull/51 Best, Nicola -- Nicola Soranzo, Ph.D. Bioinformatics Program, CRS4 Loc. Piscina Manna, 09010 Pula (CA), Italy http://www.bioinformatica.crs4.it/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Import workflows via API
Il 2014-01-17 06:45 neil.burd...@csiro.au ha scritto: Hi, I execute workflows via the API. However, if I want another user to use my workflows, I can publish my workflows, but the new user then has to go on to the web browser and import this workflow. Is there a method/script which I can call via the API which can import all available (published) workflows so the user doesn't have to click a button on the web browser import workflow ? Thanks for any help Hi Neil, you are very lucky, just yesterday my pull request implementing exactly this has been merged in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/pull-request/298/api-display-and-import-workflows-shared-by/diff and is also available in BioBlend thanks to my colleague Simone Leo: https://github.com/afgane/bioblend/pull/51 Best, Nicola -- Nicola Soranzo, Ph.D. Bioinformatics Program, CRS4 Loc. Piscina Manna, 09010 Pula (CA), Italy http://www.bioinformatica.crs4.it/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] A new tool_data_table_conf.xml created under tool-data directory
Greg Von Kuster greg@... writes: On Nov 9, 2012, at 12:56 AM, Derrick Lin wrote: Also how does this change affect the existing shed tools like BWA that is using tool_data_table_conf.xml? The existing shed tools will contiue to function as usual, but entries for them will be in the tool_data_table_conf.xml file. ANy new tools installed from the tool shed that use entries like this will be added to the new shed_tool_data_table_conf.xml file. Entries from both of these files are loaded into the ToolDataTableManager. The intent is for the Galaxy admin to be able to manuall make changes to the tool_data_table_conf.xml file and not have them munged by the tool shed's installation process. This is the same approach used for the tool_conf.xml file, where the Galaxy tool shed repository installation process uses the shed_tool_conf.xml file. Reviving an old thread... I noticed that when I install a repository from the Tool Shed which contains a tool_data_table_conf.xml.sample , the sample files mentioned therein are copied to: - tool-data/ (with AND without .sample extension) - tool-data/toolshed.g2.bx.psu.edu/repos/OWNER/REPO/REVISION/ (without .sample extension) The tool_data_table_conf.xml file in tool-data/toolshed.g2.bx.psu.edu/repos/OWNER/REPO/REVISION/ references the files in the same directory, so I find the duplication confusing. Moreover, there is an increasing number of repositories which install the same .loc files, e.g.: - the 4 cufflinks tools plus sam_pileup, sam_to_bam and samtools_mpileup install fasta_indexes.loc ; - blat, lastz and lastz_paired_reads install lastz_seqs.loc . I suppose it was made to prevent repositories from getting in each other's way, but I see 2 problems: - system administrators have to manually hard-link these files; - duplications in shed_tool_data_table_conf.xml generate errors like: galaxy.tools.data ERROR 2014-01-17 15:29:41,409 Attempted to add fields (['hg19full', 'hg19', 'Human Feb. 2009 (GRCh37/hg19) (hg19) Full', '/mnt/genome/hg19/sam_index/hg19full/hg19full.fa']) to data table 'fasta_indexes', but this entry already exists and allow_duplicates is False. Suggestions? Thanks, Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Access Control
Hi Eric, I've updated again the Python code in the wiki page since the previous version was not correctly retrieving 'required_role' and 'final_destination' attributes for tools installed from a Tool Shed if you was using the short version of the tool ID in job_conf.xml . Best, Nicola Il 2013-12-09 12:38 Nicola Soranzo ha scritto: Thanks Eric, I'm sure this will be useful to many Galaxy admins! I've edited a bit the wiki page, in particular I've updated the code to use roles instead of groups, which is more appropriate in the context of access control. Best, Nicola Il 2013-12-07 03:07 Rasche Eric ha scritto: Hi Nicola, Just thought I'd write back since I added a Wiki page in Admin/Config [11] area for implementing access control. Thank you for sharing the snippets you posted in the email thread you linked me! http://wiki.galaxyproject.org/Admin/Config/Access%20Control [12] Cheers, Eric Rasche 06.11.2013, 18:57, Eric Rasche rasche.e...@yandex.ru: Hi Nicola, Oh, excellent. I must've skipped over that, given the strange title of the thread. Your solution at the end of that thread is very promising, and certainly handles failure MUCH better than mine does (i.e. raising exceptions and not breaking a workflow if the user isn't permitted access.) (Did you put it on the galaxy wiki anywhere? If it weren't for you linking that, I never would've known about it and that's very useful info!) In my organisation's case; if a user isn't allowed access to a given tool, we believe that - - galaxy has no reason to admit it exists - - galaxy should not share default information about a tool Which is a bit different from the case of having a license to use a tool. For licensing issues, naturally it would be fine to say yes this exists and if you can't run it, obtain a license. For my org's case, we might want to store administrative tools (for other services) in galaxy. It's a very convenient platform for more than just bioinformatics and we have some non-technical people on staff who occasionally need to pull various data sets from various services/make database backups/etc. Students and clients who use our galaxy instance don't need to know that these tools are available. Thoughts? On 11/06/2013 12:12 PM, Nicola Soranzo wrote: Hi Eric, please also take a look at this mailing list thread: http://dev.list.galaxyproject.org/pass-user-groups-to-dynamic-job-runner-td4661753.html [7] If you are interested in the is_user_in_group solution, I have a slightly updated version which also uses roles instead of groups. Nicola Il giorno mer, 06/11/2013 alle 11.38 -0600, Eric Rasche ha scritto: Howdy devs, I've implemented some rather basic tool access control and am looking for feedback on my implementation. # Why Our organisation wanted the ability to restrict tools to different users/roles. As such I've implemented as an execute tag which can be applied to either section or tools in the tool configuration file. # Example galaxy-admin changes For example: section execute=a...@b.co [1],b...@b.co [2] id=EncodeTools name=ENCODE Tools tool file=encode/gencode_partition.xml / tool execute=b...@b.co [3] file=encode/random_intervals.xml / /section which would allow A and B to access gencode_parition, but only B would be able to access random_intervals. To put it explicity - by default, everyone can access all tools - if section level permissions are set, then those are set as defaults for all tools in that section - if tool permissions are set, they will override the defaults. # Pros and Cons There are some good features - non-accessible tools won't show up in the left hand panel, based on user - non-accessible tools cannot be run or accessed. There are some caveats however. - existence of tools is not completely hidden. - Labels are not hidden at all. - workflows break completely if a tool is unavailable to a shared user and the user copies+edits. They can be copied, and viewed (says tool not found), but cannot be edited. Tool names/id/version info can be found in the javascript object due to the call to app.toolbox.tool_panel.items() in templates/webapps/galaxy/workflow/editor.mako, as that returns the raw tool list, rather than one that's filtered on whether or not the user has access. I'm yet to figure out a clean fix for this. Additionally, empty sections are still shown even if there aren't tools listed in them. For a brief overview of my changes, please see the attached diff. (It's missing one change because I wasn't being careful and started work on multiple different features) # Changeset overview In brief, most of the changes consist of - new method in model.User to check if an array of roles overlaps at all with a user's roles - modifications to appropriate files for reading in the new tool_config.xml's options - modification to get_tool to pass user information, as whether or not a tool exists
Re: [galaxy-dev] Skip automated testing of tools in this revision setting?
Thanks Greg for the heads-up! These are great news, I can surely wait until mid-January (the wait will be mostly vacation time :). Just let us know when we can start checking again the test results on the main Tool Shed. Nicola Il 2013-12-17 21:15 Greg Von Kuster ha scritto: Hi Nicola, There's some bad news and good new regarding this issue. The bad news is that the old version of the tool shed's install and test framework required complete reengineering, and this resulted in it not being functional in the main tool shed's environment which runs the Galaxy stable branch. I've tried several approaches to committing and grafting changes to the stable branch in an attempt to piece together a functional environment for the main tool shed until the next Galaxy release, but the reengineering effort was so extensive that it made this extremely difficult. So the main tool shed cannot be tested until the next-stable branch is created in preparation for the next Galaxy release. The next-stable branch is currently scheduled for mid January, 2014, with the Galaxy release currently scheduled for the first week of February, 2014. The good news is that the reengineerig effort has resulted in a much more stable and functional install and test framework that will provide many more benefits than its previous incarnation. I am now working on enhancing the framework to include installing and testing repositories of type tool_dependency_definition. This enhancement will be included in the next-stable branch when it is created in mid-January. Sorry for the inconvenience this may cause, but after mid-January things should be in good shape on the main tool shed. Greg Von Kuster On Dec 12, 2013, at 5:00 PM, Nicola Soranzo sora...@crs4.it wrote: Hi Greg, thanks for looking at this! Unfortunately still no test results displayed for http://toolshed.g2.bx.psu.edu/view/crs4/blat Best, Nicola Il giorno mar, 10/12/2013 alle 10.43 -0500, Greg Von Kuster ha scritto: Hello Nicola, The main tool shed is running a bit behind the test tool shed since it is on the stable branch. I've commited a fix for the main tool shed in 11664:3bf805dfa14e that should correct the problem resulting in no test results being displayed. Results should begin being displayed with the test run tomorrow morning. I'll make sure to inspect the test runs for the main tool shed more closely from here on (I've been focusing more on the test tool shed) to make sure the different issues that arise on the stable branch are handled more quickly. Thanks for letting me know about this. Greg Von Kuster On Dec 10, 2013, at 10:09 AM, Nicola Soranzo sora...@crs4.it wrote: Hi Greg! I don't know if it's the same problem, but I've not been able to see test results for this repository for a while: http://toolshed.g2.bx.psu.edu/view/crs4/blat Best, Nicola Il giorno mar, 10/12/2013 alle 09.46 -0500, Greg Von Kuster ha scritto: There are several flags for each repository revision that determine if and how the revision is tested. If tested, flags also detemine which category the latest revision falls into when clicking on the links in the Tool Shed's menu. One of these flags is tools_functionally_correct, which was incorrectly set on http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp/ee10017fcd80 due to a bug introduced during the reengineering of the Topll Shed's install and test framework that I recently did. This bug was corrected in the following change set. https://bitbucket.org/galaxy/galaxy-central/commits/ca25579a90e97157e32b527e7471cde90e9beed0 I've corrected the setting for the tools_functionally_correct flag for your repository so it should no longer be displayed in the Latest revision: failing tool tests category. Thanks for letting us know about this, and please let us know if you discover other reposiory revisions that require the same correction. Greg Von Kuster On Dec 9, 2013, at 5:06 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Dave Greg, Could you double check how the Skip automated testing of tools in this revision setting is being used on the Test Tool Shed? e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp/ee10017fcd80 Test runs Automated test environment Not tested Licensing issues restrict automated tool dependency installation. 2013-11-29 17:19:34 Automated test environment Not tested Licensing issues restrict automated tool dependency installation. (Ticked) Skip automated testing of tools in this revision That all looks fine, and it is listed under Latest revision: skip tool tests. However, this tool is also listed under Latest revision: failing tool tests. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu
Re: [galaxy-dev] Skip automated testing of tools in this revision setting?
Hi Greg, thanks for looking at this! Unfortunately still no test results displayed for http://toolshed.g2.bx.psu.edu/view/crs4/blat Best, Nicola Il giorno mar, 10/12/2013 alle 10.43 -0500, Greg Von Kuster ha scritto: Hello Nicola, The main tool shed is running a bit behind the test tool shed since it is on the stable branch. I've commited a fix for the main tool shed in 11664:3bf805dfa14e that should correct the problem resulting in no test results being displayed. Results should begin being displayed with the test run tomorrow morning. I'll make sure to inspect the test runs for the main tool shed more closely from here on (I've been focusing more on the test tool shed) to make sure the different issues that arise on the stable branch are handled more quickly. Thanks for letting me know about this. Greg Von Kuster On Dec 10, 2013, at 10:09 AM, Nicola Soranzo sora...@crs4.it wrote: Hi Greg! I don't know if it's the same problem, but I've not been able to see test results for this repository for a while: http://toolshed.g2.bx.psu.edu/view/crs4/blat Best, Nicola Il giorno mar, 10/12/2013 alle 09.46 -0500, Greg Von Kuster ha scritto: There are several flags for each repository revision that determine if and how the revision is tested. If tested, flags also detemine which category the latest revision falls into when clicking on the links in the Tool Shed's menu. One of these flags is tools_functionally_correct, which was incorrectly set on http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp/ee10017fcd80 due to a bug introduced during the reengineering of the Topll Shed's install and test framework that I recently did. This bug was corrected in the following change set. https://bitbucket.org/galaxy/galaxy-central/commits/ca25579a90e97157e32b527e7471cde90e9beed0 I've corrected the setting for the tools_functionally_correct flag for your repository so it should no longer be displayed in the Latest revision: failing tool tests category. Thanks for letting us know about this, and please let us know if you discover other reposiory revisions that require the same correction. Greg Von Kuster On Dec 9, 2013, at 5:06 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Dave Greg, Could you double check how the Skip automated testing of tools in this revision setting is being used on the Test Tool Shed? e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp/ee10017fcd80 Test runs Automated test environment Not tested Licensing issues restrict automated tool dependency installation. 2013-11-29 17:19:34 Automated test environment Not tested Licensing issues restrict automated tool dependency installation. (Ticked) Skip automated testing of tools in this revision That all looks fine, and it is listed under Latest revision: skip tool tests. However, this tool is also listed under Latest revision: failing tool tests. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Skip automated testing of tools in this revision setting?
Hi Greg! I don't know if it's the same problem, but I've not been able to see test results for this repository for a while: http://toolshed.g2.bx.psu.edu/view/crs4/blat Best, Nicola Il giorno mar, 10/12/2013 alle 09.46 -0500, Greg Von Kuster ha scritto: There are several flags for each repository revision that determine if and how the revision is tested. If tested, flags also detemine which category the latest revision falls into when clicking on the links in the Tool Shed's menu. One of these flags is tools_functionally_correct, which was incorrectly set on http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp/ee10017fcd80 due to a bug introduced during the reengineering of the Topll Shed's install and test framework that I recently did. This bug was corrected in the following change set. https://bitbucket.org/galaxy/galaxy-central/commits/ca25579a90e97157e32b527e7471cde90e9beed0 I've corrected the setting for the tools_functionally_correct flag for your repository so it should no longer be displayed in the Latest revision: failing tool tests category. Thanks for letting us know about this, and please let us know if you discover other reposiory revisions that require the same correction. Greg Von Kuster On Dec 9, 2013, at 5:06 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Dave Greg, Could you double check how the Skip automated testing of tools in this revision setting is being used on the Test Tool Shed? e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp/ee10017fcd80 Test runs Automated test environment Not tested Licensing issues restrict automated tool dependency installation. 2013-11-29 17:19:34 Automated test environment Not tested Licensing issues restrict automated tool dependency installation. (Ticked) Skip automated testing of tools in this revision That all looks fine, and it is listed under Latest revision: skip tool tests. However, this tool is also listed under Latest revision: failing tool tests. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Access Control
Thanks Eric, I'm sure this will be useful to many Galaxy admins! I've edited a bit the wiki page, in particular I've updated the code to use roles instead of groups, which is more appropriate in the context of access control. Best, Nicola Il 2013-12-07 03:07 Rasche Eric ha scritto: Hi Nicola, Just thought I'd write back since I added a Wiki page in Admin/Config [11] area for implementing access control. Thank you for sharing the snippets you posted in the email thread you linked me! http://wiki.galaxyproject.org/Admin/Config/Access%20Control [12] Cheers, Eric Rasche 06.11.2013, 18:57, Eric Rasche rasche.e...@yandex.ru: Hi Nicola, Oh, excellent. I must've skipped over that, given the strange title of the thread. Your solution at the end of that thread is very promising, and certainly handles failure MUCH better than mine does (i.e. raising exceptions and not breaking a workflow if the user isn't permitted access.) (Did you put it on the galaxy wiki anywhere? If it weren't for you linking that, I never would've known about it and that's very useful info!) In my organisation's case; if a user isn't allowed access to a given tool, we believe that - - galaxy has no reason to admit it exists - - galaxy should not share default information about a tool Which is a bit different from the case of having a license to use a tool. For licensing issues, naturally it would be fine to say yes this exists and if you can't run it, obtain a license. For my org's case, we might want to store administrative tools (for other services) in galaxy. It's a very convenient platform for more than just bioinformatics and we have some non-technical people on staff who occasionally need to pull various data sets from various services/make database backups/etc. Students and clients who use our galaxy instance don't need to know that these tools are available. Thoughts? On 11/06/2013 12:12 PM, Nicola Soranzo wrote: Hi Eric, please also take a look at this mailing list thread: http://dev.list.galaxyproject.org/pass-user-groups-to-dynamic-job-runner-td4661753.html [7] If you are interested in the is_user_in_group solution, I have a slightly updated version which also uses roles instead of groups. Nicola Il giorno mer, 06/11/2013 alle 11.38 -0600, Eric Rasche ha scritto: Howdy devs, I've implemented some rather basic tool access control and am looking for feedback on my implementation. # Why Our organisation wanted the ability to restrict tools to different users/roles. As such I've implemented as an execute tag which can be applied to either section or tools in the tool configuration file. # Example galaxy-admin changes For example: section execute=a...@b.co [1],b...@b.co [2] id=EncodeTools name=ENCODE Tools tool file=encode/gencode_partition.xml / tool execute=b...@b.co [3] file=encode/random_intervals.xml / /section which would allow A and B to access gencode_parition, but only B would be able to access random_intervals. To put it explicity - by default, everyone can access all tools - if section level permissions are set, then those are set as defaults for all tools in that section - if tool permissions are set, they will override the defaults. # Pros and Cons There are some good features - non-accessible tools won't show up in the left hand panel, based on user - non-accessible tools cannot be run or accessed. There are some caveats however. - existence of tools is not completely hidden. - Labels are not hidden at all. - workflows break completely if a tool is unavailable to a shared user and the user copies+edits. They can be copied, and viewed (says tool not found), but cannot be edited. Tool names/id/version info can be found in the javascript object due to the call to app.toolbox.tool_panel.items() in templates/webapps/galaxy/workflow/editor.mako, as that returns the raw tool list, rather than one that's filtered on whether or not the user has access. I'm yet to figure out a clean fix for this. Additionally, empty sections are still shown even if there aren't tools listed in them. For a brief overview of my changes, please see the attached diff. (It's missing one change because I wasn't being careful and started work on multiple different features) # Changeset overview In brief, most of the changes consist of - new method in model.User to check if an array of roles overlaps at all with a user's roles - modifications to appropriate files for reading in the new tool_config.xml's options - modification to get_tool to pass user information, as whether or not a tool exists is now dependent on who is asking. Please let me know if you have input on this before I create a pull request on this feature. # Fixes I believe this will fix a number of previously brought up issues (at least to my understanding of the issues listed) + https://trello.com/c/Zo7FAXlM/286-24-add-ability-to-password-secure
Re: [galaxy-dev] Including descriptions etc in extended BLAST+ tabular output
Il 2013-12-04 20:48 Jim Johnson ha scritto: On 12/4/13, 10:21 AM, Peter Cock wrote: On Mon, Dec 2, 2013 at 3:37 PM, Peter Cock p.j.a.c...@googlemail.com [1] wrote: On Sun, Dec 1, 2013 at 7:10 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de [2] wrote: Great. I'll probably merge the c25 branch and push that to the Test Tool Shed next week (adding the descriptions as a new 25th column to the default extended tabular output). +1 Done, see this and the preceding commits: https://github.com/peterjc/galaxy_blast/commit/eb1e522a864e5274a2d274a49fcb15c313e93d69 [3] And the Test Tool Shed repository has been updated: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/8f9023b30384 [4] We'll also be running this update locally for a little more testing, but I'd appreciate some of you also testing this, e.g. via the Test Tool Shed on your own local (development) Galaxy instances. If there are no problems, that can probably go out to the main Tool Shed by the end of the week. I pushed another update to the Test Tool Shed addressing a slight change in the autogenerated IDs used in the BLAST XML output, and an overlooked mention of 24 columns: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/72170c3f515a [5] The functional tests are passing on the Test Tool Shed, so before I push this to the main Tool Shed, any other comments? I'll look at the pick-your-own column output for the next revision of the NCBI BLAST+ wrappers. I've been working on this using JJ's work on column picking in the BLAST XML to tabular tool - viewable here: https://github.com/jmchilton/galaxy_blast/commit/d79afc03522768323494818a40aac10513a6fa09 [6] See this branch updating and extending that work: https://github.com/peterjc/galaxy_blast/tree/blastxml_jj [7] I'm considering splitting the column picker into three, the standard 12 columns, the further 13 used in our extended 25 column output, and a third category for any other columns offered by BLAST+ (like the taxonomy columns added in BLAST+ 2.2.28). Here's a screenshot showing the column picker offering the 25 columns, split in two: Note that (following JJ's earlier work) these all include the column numbers as used in the standard 12 column BLAST tabular output, or our extended 25 column output. Note I would not intend to number the final set of extra columns. Do people think this is helpful, or potentially confusing (since the column numbering will change in a custom selection)? Peter Splitting into 3 sections seems like a good idea. That makes it easier to select groups of columns. I also think labeling the column options with the column number in the standard tabular outputs will be useful to users. I also like the splitting into 3 sections, but I think that the column numbering may be confusing and I would prefer it's not included. Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Including descriptions etc in extended BLAST+ tabular output
Il 2013-12-02 13:08 Peter Cock ha scritto: On Sun, Dec 1, 2013 at 7:10 PM, Björn Grüning wrote: Peter wrote: But, I would think we should offer the option to get all available result information. The multi-select checkboxes could serve that purpose. OK, so adding a pick-you-own columns tabular output would be useful :) +1 Any thoughts on if this should be the standard 12 columns plus a user selection, or a free selection of any columns (e.g. could omit most of the standard 12). I'd go with free selection. Best, Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Access Control
Hi Eric, please also take a look at this mailing list thread: http://dev.list.galaxyproject.org/pass-user-groups-to-dynamic-job-runner-td4661753.html If you are interested in the is_user_in_group solution, I have a slightly updated version which also uses roles instead of groups. Nicola Il giorno mer, 06/11/2013 alle 11.38 -0600, Eric Rasche ha scritto: Howdy devs, I've implemented some rather basic tool access control and am looking for feedback on my implementation. # Why Our organisation wanted the ability to restrict tools to different users/roles. As such I've implemented as an execute tag which can be applied to either section or tools in the tool configuration file. # Example galaxy-admin changes For example: section execute=a...@b.co,b...@b.co id=EncodeTools name=ENCODE Tools tool file=encode/gencode_partition.xml / tool execute=b...@b.co file=encode/random_intervals.xml / /section which would allow A and B to access gencode_parition, but only B would be able to access random_intervals. To put it explicity - by default, everyone can access all tools - if section level permissions are set, then those are set as defaults for all tools in that section - if tool permissions are set, they will override the defaults. # Pros and Cons There are some good features - non-accessible tools won't show up in the left hand panel, based on user - non-accessible tools cannot be run or accessed. There are some caveats however. - existence of tools is not completely hidden. - Labels are not hidden at all. - workflows break completely if a tool is unavailable to a shared user and the user copies+edits. They can be copied, and viewed (says tool not found), but cannot be edited. Tool names/id/version info can be found in the javascript object due to the call to app.toolbox.tool_panel.items() in templates/webapps/galaxy/workflow/editor.mako, as that returns the raw tool list, rather than one that's filtered on whether or not the user has access. I'm yet to figure out a clean fix for this. Additionally, empty sections are still shown even if there aren't tools listed in them. For a brief overview of my changes, please see the attached diff. (It's missing one change because I wasn't being careful and started work on multiple different features) # Changeset overview In brief, most of the changes consist of - new method in model.User to check if an array of roles overlaps at all with a user's roles - modifications to appropriate files for reading in the new tool_config.xml's options - modification to get_tool to pass user information, as whether or not a tool exists is now dependent on who is asking. Please let me know if you have input on this before I create a pull request on this feature. # Fixes I believe this will fix a number of previously brought up issues (at least to my understanding of the issues listed) + https://trello.com/c/Zo7FAXlM/286-24-add-ability-to-password-secure-tools + (I saw some solution where they were adding _beta to tool names which gave permissions to developers somewhere, but cannot find that now) Cheers, Eric Rasche ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] RFC: remove trailing semicolons from command line - Broken bowtie2_wrapper on some SGE systems
Il giorno ven, 11/10/2013 alle 01.10 +0200, Björn Grüning ha scritto: Dear all, some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed. The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper and omitting the trailing ';' but maybe its better to fix it once for all in the corresponding tools code, patched attached. The Bowtie2 bug may be fixed if my pull request 234 will be merged: https://bitbucket.org/galaxy/galaxy-central/pull-request/234/bug-fixes-for-3-tools-rebased-222 I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';' from the command line? That would be sensible too. Ciao, Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] RFC: remove trailing semicolons from command line - Broken bowtie2_wrapper on some SGE systems
I already fixed it one week ago: https://bitbucket.org/nsoranzo/galaxy-central-tools/commits/74b5fd2318d33b76382b052eae6a28b035dc075f but then Ross committed a similar fix to galaxy-central, I merged it in my pull request and then Ross reverted the fix! This is the reason for rebasing pull request #222 to #234. Ciao, Nicola Il 2013-10-11 16:05 Björn Grüning ha scritto: Hi Nicola, interesting you fixed bowtie in the same way than we did. But somehow I would like to strip a trailing ';' by default in Galaxy, when it does not break anything else. Thanks for sharing! Bjoern Il giorno ven, 11/10/2013 alle 01.10 +0200, Björn Grüning ha scritto: Dear all, some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed. The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper and omitting the trailing ';' but maybe its better to fix it once for all in the corresponding tools code, patched attached. The Bowtie2 bug may be fixed if my pull request 234 will be merged: https://bitbucket.org/galaxy/galaxy-central/pull-request/234/bug-fixes-for-3-tools-rebased-222 I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';' from the command line? That would be sensible too. Ciao, Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
actions_group is neither in stable nor in default branch of galaxy-dist repository, it's present only in galaxy-central. Nicola Il giorno lun, 07/10/2013 alle 11.24 -0400, Dave Bouvier ha scritto: Peter, The platform detection code is not yet in the stable branch, but is intended to be included in the upcoming release. The automated testing framework does run the default branch, though, so the nightly tests will make use of the new definitions. --Dave B. On 10/07/2013 11:22 AM, Peter Cock wrote: On Mon, Oct 7, 2013 at 3:44 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:37 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh. Does that mean the test system has switched between sh and bash and back to sh again? Is there anything written down about the expected shell(s) available for a standard Galaxy instance? For sh, the corresponding syntax would look like: if [ $string = value ] then echo string is equal to value fi However, it strikes me that the platform detection and download url determination could also be done using the recently introduced actions_group feature, which would bypass shell-dependent conditional syntax entirely. Yes, but is that in stable releases for galaxy-dist yet? In the meantime I will try that on the Test Tool Shed, based on the tool_dependencies.xml example you shared before: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/c83440a9bdfb Note there seems to me to be a lot of duplication - I would like a way to unambiguously combine multiple CPU architectures into a single action. For example, combine these: actions os=darwin architecture=x86_64 actions os=darwin architecture=i386 into: actions os=darwin architecture=x86_64,i386 or just: actions os=darwin and similarly, combine these: actions os=linux architecture=i386 actions os=linux architecture=i686 into something like: actions os=linux architecture=i386,i686 Assuming that works on the Test Tool Shed, and the code for this is already in the stable releases, I will update the main Tool Shed definition as well. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Download URL not accepted in tool_dependencies.xml
This problem is probably due to the 2 ampersands '', which are not allowed in XML. Try substituting them with 'amp;'. Nicola Il giorno lun, 30/09/2013 alle 13.51 +0200, Joachim Jacob | VIB | ha scritto: Hi all, The download URL seems not to be accepted in tool_dependencies.xml. It is an URL from sourceforge, the direct link to a package. The error upon uploading the tool to my toolshed: ** Metadata may have been defined for some items in revision '5665a799775d'. Correct the following problems if necessary and reset metadata. tool_dependencies.xml - Exception attempting to parse /mnt/toolsheddb/database/000/repo_26/tool_dependencies.xml: not well-formed (invalid token): line 6, column 149 ** The tool_dependencies.xml file: ** 1 ?xml version=1.0? 2 tool_dependency 3 package name=transpose version=2.0.0 4install version=1.0 5 actions 6 action type=download_by_urlhttp://downloads.sourceforge.net/project/transpose/transpose/transpose-2.0/2.0/transpose-2.0.zip?r=ts=1380535239use_mirror=surfnet/action 7action type=shell_commandmkdir bin/action 8 action type=shell_commandunzip transpose-2.0.zip/action action type=shell_commandcd transpose-2.0/src/action action type=shell_commandgcc transpose.c -o transpose/action action type=move_file source$INSTALL_DIR/transpose-2.0/src/transpose/source destination$INSTALL_DIR/bin/destination /action action type=shell_commandchmod +x $INSTALL_DIR/bin/transpose/action action type=set_environment environment_variable name=PATH action=prepend_to$INSTALL_DIR/bin/environment_variable /action /actions /install readme Compiling transpose and putting in the path. /readme /package /tool_dependency ** Cheers, Joachim ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] pass user groups to dynamic job runner
Il giorno ven, 20/09/2013 alle 10.25 -0500, John Chilton ha scritto: On Fri, Sep 20, 2013 at 9:08 AM, Nicola Soranzo sora...@crs4.it wrote: First step is to create a new dynamic job destination, this is demonstrated in job_conf.xml in the above changeset. This demonstrates the new style job_conf section - this will need to be adapted for the old style runners but is still doable. Then you will want to add a dynamic job runner rule that implements this logic. This is the file lib/galaxy/jobs/rules/license_checker.py. Is there a way to specify in job_conf.xml a per-tool final destination after the has_license filtering ? E.g.: tool id=cat1 destination=has_license final_destination=local / tool id=cat2 destination=has_license final_destination=remote_cluster / The 'final_destination' attribute would then be used in has_license function instead of returning DEFAULT_JOB_DESTINATION_ID. There is no way currently to pass information from the tool tag to the dynamic runner or to have the dynamic runner pass on making a decision in some way. It could potentially make sense to add these - but I would put forward a workaround that in my opinion is actually a little better: ... I have actually found a way to implement my idea of specifying a final destination! I also generalized your approach giving the possibility to also specify a per-tool required group in jobs_conf.xml . job_conf.xml : ?xml version=1.0? job_conf plugins workers=4 plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner / plugin id=drmaa type=runner load=galaxy.jobs.runners.drmaa:DRMAAJobRunner / /plugins handlers handler id=main/ /handlers destinations default=remote_cluster destination id=local runner=local / destination id=is_user_in_group runner=dynamic param id=functionis_user_in_group/param /destination destination id=remote_cluster runner=drmaa / /destinations tools tool id=tool1 destination=is_user_in_group required_group=other_group / tool id=tool2 destination=is_user_in_group final_destination=local / /tools /job_conf lib/galaxy/jobs/rules/is_user_in_group.py : from galaxy.jobs.mapper import JobMappingException DEFAULT_GROUP = 'have_license' def is_user_in_group(user, app, tool_id): # app is a galaxy.app.UniverseApplication object if user is None: raise JobMappingException('You must login to use this tool!') user_group_assocs = user.groups or [] default_destination_id = app.job_config.get_destination(None) for jtc in app.job_config.get_job_tool_configurations(tool_id): # app.job_config.get_job_tool_configurations(tool_id) returns a list of galaxy.jobs.JobToolConfiguration objects if not jtc.params: required_group = jtc.get('required_group', DEFAULT_GROUP) final_destination = jtc.get('final_destination', default_destination_id) break else: required_group = DEFAULT_GROUP final_destination = default_destination_id user_in_group = required_group in [user_group_assoc.group.name for user_group_assoc in user_group_assocs] if not user_in_group: raise JobMappingException('This tool is restricted to users in the %s group, please contact a site administrator to be authorized!' % required_group) else: return final_destination Hope that helps, Nicola -- Nicola Soranzo, Ph.D. CRS4 Bioinformatics Program Loc. Piscina Manna 09010 Pula (CA), Italy http://www.bioinformatica.crs4.it/ There is no longer any need to display such tools to the user thanks to dynamic toolbox filters. This is that last file: lib/galaxy/tools/filters/license_filter.py. This will prevent any tool in the list 'LICENSED_TOOLS' from even being seen by unlicensed users. You do need to specify this filter is being used by adding the line: tool_filters = license_filter:has_license in the [app:main] section of universe_wsgi.ini. I think that something like this: # Tool filtering. Multiple such filters can be specified by giving # module (relative to galaxy.tools.filters) and function names for each # as a comma separated list. # See lib/galaxy/tools/filters/examples.py.sample for some examples. #tool_filters = #tool_label_filters = #tool_section_filters = should be added to universe_wsgi.ini.sample , otherwise this wonderful feature is very difficult to discover. Do you mean to suggest that merely documenting my contributions to Galaxy on my Twitter feed is insufficient? Hmm I don't know about that... Seriously though, Björn has implemented some extensions to dynamic toolbox filters that include this: https://bitbucket.org/galaxy/galaxy-central/pull-request/179/implement-the-ability-to-change-the-tool/diff#chg-universe_wsgi.ini.sample I have added more options to Galaxy then I have documented in that file and I am always
Re: [galaxy-dev] error in 2013_08_12 DevNewsBriefs
Il 2013-09-23 20:43 Curtis Hendrickson (Campus) ha scritto: Dear Galaxy Team I was upgrading our servers with the 8/12 release, and going over the DevNotes [1]. I noted item Pull Requets Merged #5: SAMtools indexes #188 [2]. But the file created by the pull request didn't seem to exist. After looking in bitbucket at the history, it looks like the pull request was backed out [3] before the release. Have I correctly understood the situation? Dear Curtis, unfortunately you're right, my pull request has been backed out before the latest release. It seems that the Galaxy Team intention is to first move SAMtools and cufflinks wrapper to the Tool Shed and then re-apply my changes. There is a Trello card if you would like to monitor the progresses: https://trello.com/c/GvypU9T7/1021-tools-migrate-sam-fa-indices-tools-to-the-tool-shed-and-reimplement-nicola-soranzo-s-enhancements-to-this-data-table Nicola Regards, Curtis Links: -- [1] http://wiki.galaxyproject.org/DevNewsBriefs/2013_08_12 [2] http://wiki.galaxyproject.org/DevNewsBriefs/2013_08_12#Pull_Requests_Merged [3] https://bitbucket.org/galaxy/galaxy-central/commits/73d7a93e2f8d0bafbcb0b4dd9bf562d1fa6a88e0#general-comments ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] pass user groups to dynamic job runner
Il giorno gio, 19/09/2013 alle 12.42 -0500, John Chilton ha scritto: Feature request received. The following changeset demonstrates how one could implement this https://github.com/jmchilton/galaxy-central/commit/b500a24bc1aa94bef48db20a7cc1f3d68855b7fd Hi John, that's pretty interesting and we are probably going to use something similar for some problematic tools, thanks for sharing! First step is to create a new dynamic job destination, this is demonstrated in job_conf.xml in the above changeset. This demonstrates the new style job_conf section - this will need to be adapted for the old style runners but is still doable. Then you will want to add a dynamic job runner rule that implements this logic. This is the file lib/galaxy/jobs/rules/license_checker.py. Is there a way to specify in job_conf.xml a per-tool final destination after the has_license filtering ? E.g.: tool id=cat1 destination=has_license final_destination=local / tool id=cat2 destination=has_license final_destination=remote_cluster / The 'final_destination' attribute would then be used in has_license function instead of returning DEFAULT_JOB_DESTINATION_ID. There is no longer any need to display such tools to the user thanks to dynamic toolbox filters. This is that last file: lib/galaxy/tools/filters/license_filter.py. This will prevent any tool in the list 'LICENSED_TOOLS' from even being seen by unlicensed users. You do need to specify this filter is being used by adding the line: tool_filters = license_filter:has_license in the [app:main] section of universe_wsgi.ini. I think that something like this: # Tool filtering. Multiple such filters can be specified by giving # module (relative to galaxy.tools.filters) and function names for each # as a comma separated list. # See lib/galaxy/tools/filters/examples.py.sample for some examples. #tool_filters = #tool_label_filters = #tool_section_filters = should be added to universe_wsgi.ini.sample , otherwise this wonderful feature is very difficult to discover. Hope this helps! -John Thanks again! Nicola On Thu, Sep 19, 2013 at 2:45 AM, Vandeweyer Geert geert.vandewe...@uantwerpen.be wrote: Hi, Could somebody point me to the place where I can create a ticket for the following feature: - We want to have tools available only to users who provided a licence for this tool. - To prevent very long email lists (see example on dev-only tools), I'd like to have a group 'have_licence' - in dynamic job runner function : check if user is in usergroup : ok = run ; fail = give message. Or can I hide tools from the menu based on usergroup? Best, Geert ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Defining $GALAXY_SLOTS for use in tool wrappers
Il 2013-07-30 17:18 Peter Cock ha scritto: Hello all, Re: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-June/010153.html http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-October/011557.html Something I raised during the GCC2013, and we talked about via Twitter as well was a Galaxy environment variable for use within Tool Wrappers setting the number of threads/CPUs to use. The idea is that you can configure a default value, and then override this per runner or per tool etc. Thanks Peter for pushing this idea, I totally support this proposal. In the mean time, I've been using for my tools the solution by Jim Johnson for its CD-HIT wrapper: http://toolshed.g2.bx.psu.edu/view/jjohnson/cdhit But this requires the system administrator to modify both the tool env.sh and job_conf.xml to be in sync. Is there an open Trello card for this? A Trello card would be useful indeed. Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] blastn: Error: NCBI C++ Exception
Il giorno gio, 13/06/2013 alle 18.06 +0100, Peter Cock ha scritto: On Thu, Jun 13, 2013 at 6:02 PM, Nicola Soranzo sora...@crs4.it wrote: There are two issues here, one that we would like to be able to handle multi-threading better via Galaxy configuration on a per-tool level, but for now it must often be hard coded or done via tool-specific environment variables. Regarding the configuration of multi-threading for BLAST+ tools, instead of hardcoding -num_threads 8 in the XML files, I'd like to do implement something like this: http://toolshed.g2.bx.psu.edu/repos/jjohnson/cdhit/rev/cca0838c1597 What do you think? Should I prepare a pull request to discuss at GCC2013? That's a sensible idea, something like $BLASTTHREADS as an environment variable? I was thinking something like BLAST_SITE_OPTIONS='-num_threads 8', which could also include other parameters (e.g. memory limits) in the future. I'd still prefer something built into Galaxy like $THREADS or $GALAXYTHREADS or whatever which can be sets with a default value and adjusted in the per-tool job runner setup (e.g. send BLAST jobs to this cluster queue with 16 threads). This is definitely a good general topic for the tool authors and/or BLAST wrapping BoF sessions at the conference: http://wiki.galaxyproject.org/Events/GCC2013/BoF/ToolDevelopers http://wiki.galaxyproject.org/Events/GCC2013/BoF/GalaxyBlast Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] ASN.1 (text and binary) formats in Galaxy Tool Shed
Il giorno mer, 03/04/2013 alle 14.27 +0100, Peter Cock ha scritto: On Tue, Feb 19, 2013 at 2:00 PM, James Taylor ja...@jamestaylor.org wrote: On Tue, Feb 19, 2013 at 6:32 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Would a pull request implementing this be acceptable? Yes. My understanding is that ASN is a completely flexible metaformat, like XML, and so should be under either Text or Data, with appropriate subtypes defined for blast, et cetera. Nicola made the pull request we discussed a month ago: https://bitbucket.org/galaxy/galaxy-central/pull-request/135/add-generic-asn1-text-and-binary-datatypes/diff Could someone please review and commit this? Please note that this pull request contains also this commit: https://bitbucket.org/nsoranzo/galaxy-central/commits/62d0924f163b8731adaf582056587c0c4f415a7b which fixes a bug where uploading an unsniffable binary file whose filename has more than one dot (e.g. foo.bar.scf ) results in the error 'The uploaded binary file contains inappropriate content'. Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Problem running ncbi_makeblastdb as real user in cluster
Il giorno mar, 19/03/2013 alle 15.20 -0400, Raj Ayyampalayam ha scritto: One of the first few tools I tested was the ncbi makebkastdb tool. When I try to run it the history for that particular job is immediately shown as follows: 0: (unnamed dataset) Failed to retrieve dataset information. An error occurred with this dataset:/hasattr(): attribute name must be string/ Hi Raj, this error in the history should be fixed by this patch: https://bitbucket.org/nsoranzo/peterjc-galaxy-central-tools2/commits/35163cbc6c694f598ed1a1875353c58b275e2423 I have sent a pull request to Peter (the maintainer), hopefully he will release soon a new version of blast_datatypes with this fix. Best, Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Problem running ncbi_makeblastdb as real user in cluster
Il giorno mer, 20/03/2013 alle 12.41 +, Peter Cock ha scritto: On Wed, Mar 20, 2013 at 10:08 AM, Nicola Soranzo sora...@crs4.it wrote: Il giorno mar, 19/03/2013 alle 15.20 -0400, Raj Ayyampalayam ha scritto: One of the first few tools I tested was the ncbi makebkastdb tool. When I try to run it the history for that particular job is immediately shown as follows: 0: (unnamed dataset) Failed to retrieve dataset information. An error occurred with this dataset:/hasattr(): attribute name must be string/ Hi Raj, this error in the history should be fixed by this patch: https://bitbucket.org/nsoranzo/peterjc-galaxy-central-tools2/commits/35163cbc6c694f598ed1a1875353c58b275e2423 I have sent a pull request to Peter (the maintainer), hopefully he will release soon a new version of blast_datatypes with this fix. Best, Nicola The patch just removes the MetadataElement call - is that wise? Hi Peter, my question instead is, what are they for? With 'name' they have no meaning. Looking at the code, I don't see a recent API change regarding the name: https://bitbucket.org/galaxy/galaxy-central/history-node/default/lib/galaxy/datatypes/metadata.py?at=default Is there perhaps a universe_wsgi.ini setting which might be involved, since I've not seen this error locally? $ cd lib/galaxy/datatypes/ $ grep 'MetadataElement(' *.py|grep -v name returns nothing, so the 'name' parameter is really mandatory. Best, Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] ASN.1 (text and binary) formats in Galaxy Tool Shed
Il giorno mar, 19/02/2013 alle 20.38 +, Peter Cock ha scritto: On Tue, Feb 19, 2013 at 6:45 PM, Nicola Soranzo sora...@crs4.it wrote: Il giorno mar, 19/02/2013 alle 14.15 +, Peter Cock ha scritto: On Tue, Feb 19, 2013 at 2:00 PM, James Taylor ja...@jamestaylor.org wrote: On Tue, Feb 19, 2013 at 6:32 AM, Peter Cock wrote: I think it could make sense to define generic 'asn1' and 'asn1-binary' formats in the Galaxy core (name suggestions welcome) What about extension=asn type=galaxy.datatypes.data:GenericAsn1 and extension=asnb type=galaxy.datatypes.binary:GenericAsn1Binary like GenericXml class? Those seem sensible to me as the class names, although I'm not so sure about the format names (aka 'extensions' in Galaxy terms). I'd prefer to see the '1' in the name for clarity. My suggestions of 'asn1' and 'asn1-binary' were based on NCBI usage. Ok, no problem! Nicola - do you want to make a pull request to galaxy-central defining ASN.1 text and binary formats (which we can then subclass for the NCBI BLAST+ wrappers)? Or should I? If you mean a minimal implementation, I can surely do that. If something more elaborated is needed, then probably you are more qualified than me! The current minimal implementation you sent me for BLAST+ would be an excellent start. Things like data type sniffers etc would be a nice to have feature, but can be added later I think. And the sooner this gets into the Galaxy core, the sooner we can use it in the BLAST+ wrappers :) Before proceeding on this route, one question: we are going to introduce a dependency for blast_datatypes/ncbi_blast_plus on a future galaxy release, is it ok? I don't think it is possible to express such dependency in a repository_dependencies.xml file yet. Best, Nicola -- Nicola Soranzo, Ph.D. CRS4 Bioinformatics Program Loc. Piscina Manna 09010 Pula (CA), Italy http://www.bioinformatica.crs4.it/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] ASN.1 (text and binary) formats in Galaxy Tool Shed
Il giorno mar, 19/02/2013 alle 14.15 +, Peter Cock ha scritto: On Tue, Feb 19, 2013 at 2:00 PM, James Taylor ja...@jamestaylor.org wrote: On Tue, Feb 19, 2013 at 6:32 AM, Peter Cock p.j.a.c...@googlemail.com wrote: I think it could make sense to define generic 'asn1' and 'asn1-binary' formats in the Galaxy core (name suggestions welcome) What about extension=asn type=galaxy.datatypes.data:GenericAsn1 and extension=asnb type=galaxy.datatypes.binary:GenericAsn1Binary like GenericXml class? Would a pull request implementing this be acceptable? Yes. My understanding is that ASN is a completely flexible metaformat, like XML, and so should be under either Text or Data, with appropriate subtypes defined for blast, et cetera. Thank James, Yes - very like XML, but with the subtlety that ASN.1 comes in text and binary favours (which I presume applies to all the variants, although the binary versions may not be as commonly used for the smaller files). Nicola - do you want to make a pull request to galaxy-central defining ASN.1 text and binary formats (which we can then subclass for the NCBI BLAST+ wrappers)? Or should I? If you mean a minimal implementation, I can surely do that. If something more elaborated is needed, then probably you are more qualified than me! I think the mime-type for the base ASN.1 text format should probably be text/plain based on the NCBI usage patterns. Ok. I'm not sure what the mime-type for the base ASN.1 binary format should be (but I don't think it should be chemical/ncbi-asn1-binary). application/octet-stream ? Best, Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] DustMasker tool for ncbi_blast_plus
Il giorno lun, 11/02/2013 alle 13.19 +, Peter Cock ha scritto: On Fri, Feb 8, 2013 at 4:30 PM, Nicola Soranzo sora...@crs4.it wrote: Il giorno mer, 06/02/2013 alle 20.01 +0100, Nicola Soranzo ha scritto: Hi Peter, I added these file formats mostly as placeholders for a future implementation. Now I have changed a bit the tool by removing acclist and seqloc_xml formats since they are not recognized by the last versions of dustmasker (I also sent an email to blast-h...@ncbi.nlm.nih.gov to inform them of this bug). As before, you can find the new version at: https://bitbucket.org/nsoranzo/ncbi_blast_plus I stripped the old commit and did a new one, not a very good practice, sorry about that. It seems to have confused the bitbucket page a little, but I have checked in your initial wrapper to my development repository (I use the tools branch): https://bitbucket.org/peterjc/galaxy-central/commits/2284d485e36f74f19b0dbe78709b098d9eba4ef6 Note I'm not going to include this in the Tool Shed release yet, we need to sort out the file format definitions first. Hi Peter, I implemented minimal datatypes for maskinfo ASN.1 binary and text, plus some other improvements to ncbi_blast_plus, and I sent you a pull request through Bitbucket for your development repository. I think that would be easier for you, let me know if it is not. Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Datatype display_in_upload not respected in upload1
Hi all, I noticed that when I go in Get Data - Upload File from your computer (i.e. upload1 tool) the File Format list contains also datatypes which have display_in_upload=false in datatypes_conf.xml (e.g. afg, fli), while the list does not contain datatypes which have no display_in_upload attribute at all (e.g. bedstrict, bed6). This sounds like a bug to me, but I have not found any reference around. Thanks, Nicola -- Nicola Soranzo, Ph.D. CRS4 Bioinformatics Program Loc. Piscina Manna 09010 Pula (CA), Italy http://www.bioinformatica.crs4.it/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] DustMasker tool for ncbi_blast_plus
Il giorno mer, 06/02/2013 alle 20.01 +0100, Nicola Soranzo ha scritto: Hi Peter, I added these file formats mostly as placeholders for a future implementation. Now I have changed a bit the tool by removing acclist and seqloc_xml formats since they are not recognized by the last versions of dustmasker (I also sent an email to blast-h...@ncbi.nlm.nih.gov to inform them of this bug). As before, you can find the new version at: https://bitbucket.org/nsoranzo/ncbi_blast_plus I stripped the old commit and did a new one, not a very good practice, sorry about that. Hi Peter, I've added a new commit to this repo which updates the test output files to (recommended) BLAST 2.2.26+, since functional tests were returning errors. Hope you find it useful. Nicola -- Nicola Soranzo, Ph.D. CRS4 Bioinformatics Program Loc. Piscina Manna 09010 Pula (CA), Italy http://www.bioinformatica.crs4.it/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] DustMasker tool for ncbi_blast_plus
Adding galaxy-dev list in CC as suggested by Peter. Il giorno mer, 06/02/2013 alle 16.57 +, Peter Cock ha scritto: On Tue, Feb 5, 2013 at 11:45 AM, Nicola Soranzo sora...@crs4.it wrote: Dear Peter, I have created a simple Galaxy tool for DustMasker of the NCBI BLAST+ suite, which I think would be a useful addition to the ncbi_blast_plus repository you're maintaining in the Galaxy Tool Shed. You can find it and hopefully pull it from: https://bitbucket.org/nsoranzo/ncbi_blast_plus Kind regards, Nicola Hi Nicola, Thanks for getting involved - we can discuss this on the galaxy-dev mailing list if you prefer? For now I have CC'd Edward Kirton as he is/was working on masking in BLAST databases for Galaxy. I can see the new file tools/ncbi_blast_plus/ncbi_dustmasker_wrapper.xml however it refers to multiple new file formats - where are they defined? * acclist * maskinfo_asn1_bin * maskinfo_asn1_text * seqloc_asn1_bin * seqloc_asn1_text Hi Peter, I added these file formats mostly as placeholders for a future implementation. Now I have changed a bit the tool by removing acclist and seqloc_xml formats since they are not recognized by the last versions of dustmasker (I also sent an email to blast-h...@ncbi.nlm.nih.gov to inform them of this bug). As before, you can find the new version at: https://bitbucket.org/nsoranzo/ncbi_blast_plus I stripped the old commit and did a new one, not a very good practice, sorry about that. Have you looked at the (commented out) bits in the makeblastdb wrapper which would perhaps be relevant? This is something Edward Kirton wrote which I haven't integrated yet: !-- SEQUENCE MASKING OPTIONS -- !-- TODO repeat name=mask_data title=Provide one or more files containing masking data param name=file type=data format=asnb label=File containing masking data help=As produced by NCBI masking applications (e.g. dustmasker, segmasker, windowmasker) / /repeat repeat name=gi_mask title=Create GI indexed masking data param name=file type=data format=asnb label=Masking data output file / /repeat -- Perhaps all you need to offer in ncbi_dustmasker_wrapper.xml is 'fasta' and 'asnb' (binary ASN) formats? Edward - did you have an 'asnb' definition? 'fasta' and 'interval' are the ones I'm interested for my use case. 'maskinfo_asn1_bin' is probably the one referenced as 'asnb' in the cited code (ASN1 is a general data serialization format like XML). A file in this format can be given as input to makeblastdb -mask_data. Nicola -- Nicola Soranzo, Ph.D. CRS4 Bioinformatics Program Loc. Piscina Manna 09010 Pula (CA), Italy http://www.bioinformatica.crs4.it/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] File Upload doesn't generate any activity
Il giorno lun, 04/02/2013 alle 16.33 +, Hakeem Almabrazi ha scritto: Thank you Dannon. Hopefully things should be moving forward from now on. This issue with the LWR has been fixed in changeset 8731:3299529e0fe8 which will be in the next distribution release. That will be awesome. It's already in galaxy-dist since Saturday 2013/02/02, changeset 8531:3299529e0fe8 . Kind Regards, Nicola -- Nicola Soranzo, Ph.D. CRS4 Bioinformatics Program Loc. Piscina Manna 09010 Pula (CA), Italy http://www.bioinformatica.crs4.it/ -Original Message- From: Dannon Baker [mailto:dannonba...@me.com] Sent: Monday, February 04, 2013 9:40 AM To: Hakeem Almabrazi Cc: Galaxy Dev Subject: Re: [galaxy-dev] File Upload doesn't generate any activity On Feb 4, 2013, at 10:36 AM, Hakeem Almabrazi halmabr...@idtdna.com wrote: No there was nothing special about this installation. I installed it on CentOS and I tried to follow instructions in installing and configuring tools and data. The only thing that I did not was to test this feature before I integrate it with postgres, nginx, tools such as bwa, samtools, bowtie etc. I might have done something not right while doing so. But once I was done with all the configuration steps, I tried to load files via Get Data or Data Library. It kept spinning and nothing is actually being written to the system. I tailed the logs and it was writing some logs every seconds but no errors or warnings. The issue was resolved when I used URL/Text to go to this link as another trail test. After that everything works just fine. Now we can load files via either way. Very strange, but I'm glad it's working for you now. If this does crop up again, let me know and we can try to reproduce what's happening and fix it. The only error I keep seeing in the log is when we restart the server and from what I read, this is a bug... Traceback (most recent call last): File /Users/dtenenba/dev/galaxy-dist/lib/galaxy/jobs/handler.py, line 447, in _load_plugin module = __import__( module_name ) File /Users/dtenenba/dev/galaxy-dist/lib/galaxy/jobs/runners/lwr.py, line 232 worker = threading.Thread( ( name=LwrJobRunner.thread-%d % i ), target=self.run_next ) ^ SyntaxError: invalid syntax Please let me know if you want me to do anything else that might help debugging this issue. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] File Upload doesn't generate any activity
Il giorno lun, 04/02/2013 alle 17.41 +, Hakeem Almabrazi ha scritto: Thank you Nicola, Should I replace my current galaxy-dist with this version or is there different way to upgrade? Take a look at this wiki page: http://wiki.galaxyproject.org/Admin/Get% 20Galaxy#Keep_your_code_up_to_date Nicola -Original Message- From: Nicola Soranzo [mailto:sora...@crs4.it] Sent: Monday, February 04, 2013 10:50 AM To: Hakeem Almabrazi Cc: Dannon Baker; Galaxy Dev Subject: Re: [galaxy-dev] File Upload doesn't generate any activity Il giorno lun, 04/02/2013 alle 16.33 +, Hakeem Almabrazi ha scritto: Thank you Dannon. Hopefully things should be moving forward from now on. This issue with the LWR has been fixed in changeset 8731:3299529e0fe8 which will be in the next distribution release. That will be awesome. It's already in galaxy-dist since Saturday 2013/02/02, changeset 8531:3299529e0fe8 . Kind Regards, Nicola ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Pull request for bowtie_wrappers tool
Dear Galaxy DevTeam, I have made a few updates to the bowtie_wrappers tool from the Tool Shed, may someone please pull from: https://bitbucket.org/nsoranzo/bowtie_wrappers/ for the following changes: - Add fields for --nofw and --norc options also for single-end reads - Remove field for --cutoff option of bowtie-build - Do not show fields for --best and --strata options for paired-end reads - Add control to choose between -a and -k options - Show fields for --pairtries and --maxbts options only when noTryHard - Add control to choose between Maq- and SOAP-like alignment - Add range validators to all integer parameters I am not sure this mailing list is the correct place for such pull requests, if not please advise me! Thanks, Nicola -- Nicola Soranzo, Ph.D. CRS4 Bioinformatics Program Loc. Piscina Manna 09010 Pula (CA), Italy http://www.bioinformatica.crs4.it/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/