Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
On Thu, Mar 13, 2014 at 6:40 PM, Sanka, Ravi rsa...@jcvi.org wrote: I do not think so. Several individual datasets have been deleted (clicked the upper-right X on the history item box) but no History has been permanently deleted. Is there any indication in the database if target dataset or datasets were marked for permanent deletion? In the dataset table, I see fields deleted, purged, and purgable, but nothing that says permanently deleted. I would welcome clarification from the Galaxy Team, here and on the wiki page which might benefit from a flow diagram? https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20and%20Datasets My assumption is using permanently delete in the user interface marks an entry as purgable, and then it will be moved to purged (and the associated file on disk deleted) by the cleanup scripts - but I'm a bit hazy on this any why it takes a while for a user's usage figures to change. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Pick-you-own columns in BLAST+ tabular output
On Tue, Mar 4, 2014 at 12:19 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Mar 4, 2014 at 10:35 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Peter, Hello all, I've mentioned before that a forthcoming update to the BLAST+ Galaxy wrappers would be adding a most customisable output option with pick-your-own columns. This is available to test now from the Test Tool Shed, along with other enhancements like more masking tools and an update to BLAST+ 2.2.29: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus ... Some community testing/comments would be appreciated. I tested it briefly on my test instance and it looks all fine and seems to work :) Thanks, looking forward to see it in MainTS. Bjoern Great - thanks. I'm about to update our local production server, and barring any problems will push this BLAST+ update to the main Tool Shed later this week. This is now live, with a regression about output formats caught and fixed: http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Quota History not getting purged
Hi there, We have installed a local copy of galaxy but there seems to be a strange behavior with deleting data sets in the history panel. A few users complain that they have deleted their data sets (I can confirm that) but somehow the quota limit still doesn't show the appropriate freed space. Looking at their saved histories and adding up their data the quota limit should have changed. Any clues why this happens? I have also tried to run the clean up script but this didn't do anything on their quota limits. Best, Alex ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Quota History not getting purged
On Fri, Mar 14, 2014 at 12:28 PM, Alexander Kurze alexander.ku...@gmail.com wrote: Hi there, We have installed a local copy of galaxy but there seems to be a strange behavior with deleting data sets in the history panel. A few users complain that they have deleted their data sets (I can confirm that) but somehow the quota limit still doesn't show the appropriate freed space. Looking at their saved histories and adding up their data the quota limit should have changed. Any clues why this happens? I have also tried to run the clean up script but this didn't do anything on their quota limits. Best, Alex Does this help, or make no difference? $ cd scripts $ python set_user_disk_usage.py It will recalculate and update the user disk totals, and prints this to the terminal too. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Purging datasets as part of workflow?
I don't even think you can delete datasets in workflow presently - let alone permanently delete them - only hide them. https://trello.com/c/YfLGkJKe The core team just meet and a number of high priorities for the next 9 months were identified and reworking workflow scheduling was very high on this list. I would expect an number of key workflow modifications to be made by early fall that would make this much easier. https://trello.com/c/K2qLZCrg My own opinion on the particulars of this topic are that whether datasets are deleted or permanently deleted should be up to the workflow runner not the workflow author. In particular, the workflow model describes what datasets are to be deleted and the workflow runner could opt to permanently delete them if their Galaxy instance allows this. It is exactly because this is not a universally allowed option that I think it should not be part of the workflow description that can be shared between instances and on the tool shed. -John On Thu, Mar 13, 2014 at 1:38 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Mar 13, 2014 at 6:30 PM, Sveinung Gundersen sveinung.gunder...@medisin.uio.no wrote: Hi, Is there an automatic way to permanently delete and purge datasets that are the result of intermittent tools in a workflow? That is, how can one automatically keep only the first and the last steps of an workflow without having to manually delete all the intermittent datasets and running the purge scripts? Thanks, Sveinung Gundersen +1 In the workflows editor we can star particular outputs to be kept, and the rest are deleted. I'd much prefer they be more aggressively permanently deleted. Alternately, when editing a workflow there (used to be) a delete dataset action - how about a permanently delete action which would be great for some big-data workflows? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Quota History not getting purged
Perfect! This solved my problem. Alex On Fri, Mar 14, 2014 at 1:23 PM, Peter Cock p.j.a.c...@googlemail.comwrote: On Fri, Mar 14, 2014 at 12:28 PM, Alexander Kurze alexander.ku...@gmail.com wrote: Hi there, We have installed a local copy of galaxy but there seems to be a strange behavior with deleting data sets in the history panel. A few users complain that they have deleted their data sets (I can confirm that) but somehow the quota limit still doesn't show the appropriate freed space. Looking at their saved histories and adding up their data the quota limit should have changed. Any clues why this happens? I have also tried to run the clean up script but this didn't do anything on their quota limits. Best, Alex Does this help, or make no difference? $ cd scripts $ python set_user_disk_usage.py It will recalculate and update the user disk totals, and prints this to the terminal too. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Quota History not getting purged
On Fri, Mar 14, 2014 at 1:31 PM, Alexander Kurze alexander.ku...@gmail.com wrote: Perfect! This solved my problem. Alex That's progress. I suspect something is going wrong in the cleanup scripts failing to update the totals - but as a workaround you could include set_user_disk_usage.py as part of the cleanup cron job? Note set_user_disk_usage.py isn't on the wiki - so I assume it isn't intended to be needed this way: https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20and%20Datasets I've been looking at our data usage figures this week as part of a storage move - and also observed the totals can fail to drop after a user permanently deleted some large datasets I'd run the cleanup scripts. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] How To properly use GALAXY_SLOTS
I am not sure I understand what you mean, but if you are setting the
job to run with a ppn of greater than one but GALAXY_SLOTS is being
evaluated as 1 then you have likely uncovered a bug - maybe in the
runner, your job configuration, or the GALAXY_SLOTS logic for
pbs/torque.
I have created a Galaxy tool for debugging the problem -
https://gist.github.com/jmchilton/9548516. If you could run in your
environment and assign it multiple cores it should spit out some
interesting information relevant to debugging the problem. It will
create Galaxy datasets corresponding to the runtime-computed Galaxy
slots, the contents of PBS_NODEFILE which Galaxy attempts to use to
compute Galaxy slots, and the full contents of your worker node
environment. This last one may contain sensitive information - but if
you could post the first two or send them to me directly it would
hopefully help debug the problem.
-John
On Wed, Mar 12, 2014 at 12:37 PM, Geert Vandeweyer
geert.vandewey...@uantwerpen.be wrote:
How would those statements translate to pbs/torque (pbs_python)
I request resources using the nodes=1:ppn:4 syntax. The -pe argument is not
available it seems...
Best,
Geert
On 03/12/2014 04:43 PM, Björn Grüning wrote:
Hi Geert,
you can give every tool a different amount of SLOTS via the job_conf.xml
file.
https://wiki.galaxyproject.org/Admin/Config/Jobs
For example:
tool
id=toolshed.g2.bx.psu.edu/repos/iuc/gatk2/gatk2_variant_recalibrator
destination=12cores_24G /
where 12cores_24G is defined with a parallel SGE environment.
Cheers,
Bjoern
Am 12.03.2014 16:32, schrieb Geert Vandeweyer:
Hi,
Is there documentation for the proper setup and utilisation of the
\${GALAXY_SLOTS:-4} style options?
I noticed that tools with this setting, including BWA and GATK run
single threaded, with the following settings respectively:
BWA \${GALAXY_SLOTS:-4}
===
ps output: python
/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/b4427dbb6ced/bwa_wrappers/bwa_wrapper.py
--threads=1 --fileSource=indexed
--ref=/galaxy/galaxy_references/hg19/bwa-0.5.9/base/hg19.fasta
--do_not_build_index
--input1=/galaxy/galaxy-dist/database/files/093/dataset_93021.dat
--input2=/galaxy/galaxy-dist/database/files/093/dataset_93023.dat
--output=/galaxy/galaxy-dist/database/files/000/112/dataset_112301.dat
--genAlignType=paired --params=pre_set --suppressHeader=false
xml config: --threads=\${GALAXY_SLOTS:-4}
GATK Base Recalibrator \${GALAXY_SLOTS:-8}
==
python
/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/gatk2/8bcc13094767/gatk2/gatk2_wrapper.py
--max_jvm_heap 6g --stdout
/galaxy/galaxy-dist/database/files/000/112/dataset_112292.dat -d -I
/galaxy/galaxy-dist/database/files/000/112/dataset_112289.dat bam
gatk_input -d
/galaxy/galaxy-dist/database/files/_metadata_files/012/metadata_12686.dat
bam_index
gatk_input -p java -jar
$GATK2_PATH/GenomeAnalysisTK.jar-T BaseRecalibrator
$GATK2_SITE_OPTIONS --num_cpu_threads_per_data_thread
${GALAXY_SLOTS:-8} --no_standard_covs -R
/galaxy/galaxy_references/hg19/srma-0.1.15/hg19.fasta --out
/galaxy/galaxy-dist/database/files/000/112/dataset_112291.dat -cov
ContextCovariate -cov CycleCovariate -d
--knownSites:dbsnp,%(file_type)s
/galaxy/galaxy-dist/database/files/000/101/dataset_101086.dat vcf
input_dbsnp_0 -p --pedigreeValidationType STRICT -d --intervals
/galaxy/galaxy-dist/database/files/000/100/dataset_100129.dat bed
input_intervals_0 -p --interval_set_rule UNION -p --downsampling_type
NONE -p --baq OFF --baqGapOpenPenalty
40.0 --defaultBaseQualities -1
--validation_strictness STRICT --interval_merging ALL
java -Xmx6g -jar
/galaxy/galaxy-dist/tool-data/shared/jars/gatk2//GenomeAnalysisTK.jar -T
BaseRecalibrator --num_cpu_threads_per_data_thread 1 --no_standard_covs
-R /galaxy/galaxy_references/hg19/srma-0.1.15/hg19.fasta --out
/galaxy/galaxy-dist/database/files/000/112/dataset_112291.dat -cov
ContextCovariate -cov CycleCovariate --pedigreeValidationType STRICT
--interval_set_rule UNION --downsampling_type NONE --baq OFF
--baqGapOpenPenalty 40.0 --defaultBaseQualities -1
--validation_strictness STRICT --interval_merging ALL -I
/tmp/tmp-gatk-fIuncH/gatk_input.bam --knownSites:dbsnp,vcf
/tmp/tmp-gatk-fIuncH/input_dbsnp_0.vcf --intervals
/tmp/tmp-gatk-fIuncH/input_intervals_0.bed
Best,
Geert
--
Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: geert.vandewe...@ua.ac.be
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/in/geertvandeweyer
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
Re: [galaxy-dev] Write permission for database/files
I did not implement this - so this is just an educated guess but I believe the relevant line of code is this: https://bitbucket.org/galaxy/galaxy-central/src/3fb927653301a0c06a0bf94f2b6bd71b3595ec0d/lib/galaxy/objectstore/__init__.py?at=default#cl-295 Hopefully someone will correct me if I am wrong. -John On Wed, Mar 12, 2014 at 1:39 PM, Alper Kucukural al...@kucukural.com wrote: Hi All, Galaxy opens a directory under database/files when the #of data files increased. So, it goes 1,2,3 ... etc. I want to give a write permission for these specific directories when they are created without changing umask in the user profile. Is there anybody who can tell me where galaxy creates these directories. Thanks in advance. Best, Alper ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Empty history pane
Thanks Dannon, I set java to open the console (I assume that's what you are referring to) but when going to galaxy it doesn't open. The console does open when i go to java.com and verify my version. i've also just removed and reinstalled java and rebooted machine. still no history pane :( From: Dannon Baker dannon.ba...@gmail.com To: Liisa Koski liisa.ko...@basf.com Cc: Galaxy Dev galaxy-dev@lists.bx.psu.edu Date: 13/03/2014 06:39 PM Subject:Re: [galaxy-dev] Empty history pane If it's possible, can you check (or ask the user to check) if there are any javascript errors if you open the browser console when experiencing this failure? On Thu, Mar 13, 2014 at 2:07 PM, Liisa Koski liisa.ko...@basf.com wrote: Hello, Our site maintains a local Galaxy installation (Nov.4th distributuion). One of our usrers has lost the ability to view anything in his history pane. He is able to run tools and view the list of 'Saved Histories' in the middle pane but the History pane itself stays empty. When I impersonate him I can see his history pane. Very weird. Any insight would be much appreciated. Thanks, Liisa ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
On Fri, Mar 14, 2014 at 11:24 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Mar 13, 2014 at 6:40 PM, Sanka, Ravi rsa...@jcvi.org wrote: I do not think so. Several individual datasets have been deleted (clicked the upper-right X on the history item box) but no History has been permanently deleted. Is there any indication in the database if target dataset or datasets were marked for permanent deletion? In the dataset table, I see fields deleted, purged, and purgable, but nothing that says permanently deleted. I would welcome clarification from the Galaxy Team, here and on the wiki page which might benefit from a flow diagram? https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20and%20Datasets My assumption is using permanently delete in the user interface marks an entry as purgable, and then it will be moved to purged (and the associated file on disk deleted) by the cleanup scripts - but I'm a bit hazy on this any why it takes a while for a user's usage figures to change. Hmm. Right now I've unable (via the web interface) to permanently delete a history - it stays stuck as deleted, and thus (presumably) won't get purged by the clean up scripts. I've tried: 1. Load problem history 2. Rename the history DIE DIE to avoid confusion 3. Top right menu, Delete permanently 4. Prompted Really delete the current history permanently? This cannot be undone, OK 5. Told History deleted, a new history is active 6. Top right menu, Saved Histories 7. Click Advanced Search, status all 8. Observe DIE DIE history is only deleted (while other older histories are deleted permanently) (BAD) 9. Run the cleanup scripts, $ sh scripts/cleanup_datasets/delete_userless_histories.sh $ sh scripts/cleanup_datasets/purge_histories.sh $ sh scripts/cleanup_datasets/purge_libraries.sh $ sh scripts/cleanup_datasets/purge_folders.sh $ sh scripts/cleanup_datasets/purge_datasets.sh 10. Reload the saved history list, no change. 11. Using the drop down menu, select Delete Permanently 12. Prompted History contents will be removed from disk, this cannot be undone. Continue, OK 13. No change to history status (BAD) 14. Tick the check-box, and use the Delete Permanently button at the bottom of the page 15. Prompted History contents will be removed from disk, this cannot be undone. Continue, OK 16. No change to history status (BAD) 17. Run the cleanup scripts, no change. Note that in my universe_wsgi.ini I have not (yet) set: allow_user_dataset_purge = True If this setting is important, then the interface seems confused - and if quotas are enforced, very frustrating :( Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Problem with change_format and conditional inputs?
Grepping around the code I think this is the only way a ftype attribute on an output affects the evaluation of test data. if attributes.get( 'ftype', None ) == 'bam': local_fh, temp_name = self._bam_to_sam( local_name, temp_name ) local_name = local_fh.name I am not sure it was ever meant as a strict test. I worry about breaking backward compatibility but it is easy enough to implement this as an actual check when using newer API driven tests. I have opened a pull request for this functionality here: https://bitbucket.org/galaxy/galaxy-central/pull-request/347/check-ftype-attribute-if-defined-on-test/diff Thoughts? -John On Tue, Mar 11, 2014 at 10:30 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Mar 11, 2014 at 2:58 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Peter, I think you need to have: change_format when input=output.out_format value=0 format=txt/ when input=output.out_format value=0 -html format=html/ when input=output.out_format value=2 format=txt/ when input=output.out_format value=2 -html format=html/ when input=output.out_format value=4 format=txt/ when input=output.out_format value=4 -html format=html/ when input=output.out_format value=5 format=blastxml/ /change_format Sorry, can't test it right now. That seems to work - thanks Björn :) This seems to have exposed a bug in the test framework, e.g. test param name=query value=rhodopsin_nucs.fasta ftype=fasta / param name=db_opts_selector value=file / param name=subject value=three_human_mRNA.fasta ftype=fasta / param name=database value= / param name=evalue_cutoff value=1e-40 / param name=out_format value=5 / param name=adv_opts_selector value=basic / output name=output1 file=blastn_rhodopsin_vs_three_human.xml ftype=blastxml / /test This was saying the output should have been blastxml, but until I just fixed it the output was being tagged as tabular (although run_functional_tests.sh did check the content it didn't check the datatype matched). Dave - do think this is a reasonable enhancement? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Verifying test output datatypes, was: Problem with change_format and conditional inputs?
Thanks John, I suggest making this test framework perform this check by default (the twill and API based frameworks) and seeing what - if anything - breaks as a result on the Test Tool Shed. Note that one area of fuzziness is subclasses, e.g. if the tool output was labelled fastqsanger, but the test just said fastq, I would say the test was broken. On the other hand, if the test used a specific datatype like fastqsanger but the tool produced a dataset tagged with a more generic datatype like fastq I think that is a again a real failure. Peter On Fri, Mar 14, 2014 at 3:15 PM, John Chilton jmchil...@gmail.com wrote: Grepping around the code I think this is the only way a ftype attribute on an output affects the evaluation of test data. if attributes.get( 'ftype', None ) == 'bam': local_fh, temp_name = self._bam_to_sam( local_name, temp_name ) local_name = local_fh.name I am not sure it was ever meant as a strict test. I worry about breaking backward compatibility but it is easy enough to implement this as an actual check when using newer API driven tests. I have opened a pull request for this functionality here: https://bitbucket.org/galaxy/galaxy-central/pull-request/347/check-ftype-attribute-if-defined-on-test/diff Thoughts? -John On Tue, Mar 11, 2014 at 10:30 AM, Peter Cock p.j.a.c...@googlemail.com wrote: That seems to work - thanks Björn :) This seems to have exposed a bug in the test framework, e.g. test param name=query value=rhodopsin_nucs.fasta ftype=fasta / param name=db_opts_selector value=file / param name=subject value=three_human_mRNA.fasta ftype=fasta / param name=database value= / param name=evalue_cutoff value=1e-40 / param name=out_format value=5 / param name=adv_opts_selector value=basic / output name=output1 file=blastn_rhodopsin_vs_three_human.xml ftype=blastxml / /test This was saying the output should have been blastxml, but until I just fixed it the output was being tagged as tabular (although run_functional_tests.sh did check the content it didn't check the datatype matched). Dave - do think this is a reasonable enhancement? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Dataset's extra files
The directory can be obtained using $galaxyData.extra_files_path (be sure to check it exists before zipping it up). I would discourage re-using Galaxy components directly from inside of a tool or tool wrapper - but if you want to reference that code it is actually inside of the datatypes module - https://bitbucket.org/galaxy/galaxy-central/src/3fb927653301a0c06a0bf94f2b6bd71b3595ec0d/lib/galaxy/datatypes/data.py?at=default#cl-228 Hope this helps. -John On Tue, Mar 11, 2014 at 5:35 AM, Charles Girardot charles.girar...@embl.de wrote: Hi all, We have a local tool which role is to transfer (ie copy) a dataset file to a directory on our NFS. This is extremely convenient as it can be included within workflows and therefore save the time of clicking download button (we also have configurable renaming/compression as part of it). It is heavily used by our users. The problem is with datasets that have associated files like FASTQC as these extra files are simple not ignored... We'd like to improve our 'NFS_transfer' tool so it can deal with this in a similar fashion as the download button. Foreseen solution : * Check if a directory named 'dataset_id_files' exists within the dataset store * if so, 'cp -r' it into a tmp dir, cp the dataset itself into same tmp dir (with renaming on the fly) * zip/tar.gz the tmp dir * copy it to final NFS location Question is : is this the right way to do it ? As a non python specialist, it is a little tricky to find the right way to it (I can t locate the piece of code that does this in galaxy ie behind the download button). In particular, can I get the list of extra files using the '$galaxyFile' object given in the tool by : param type=data name=galaxyFile label=File to transfer/ i.e. in the same way we get the dataset name or file extension ($galaxyFile.dataset.name and $galaxyFile.ext) ? Any advise on how best to implement this, in a portable way, very appreciated. Thanks for your time, Charles = Charles Girardot Head of Genome Biology Computational Support (GBCS) European Molecular Biology Laboratory Tel: +49 6221 387 -8585 Fax: +49-(0)6221-387-8166 Email: charles.girar...@embl.de Room V205 Meyerhofstraße 1, 69117 Heidelberg, Germany = ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Job never starts
Hmm... unfortunately I have no particular guess about either of these issues. It sounds like you upgraded Galaxy and made large changes to your configuration at the same time. Can you try just one process and no load balancer so you can determine if the issue is with the Apache configuration or the latest version of Galaxy. If that works - can you just do one web process and one handler and see if the problems persist? The goal would be to scale up, but still figure out at what point the problems start. -John On Tue, Mar 11, 2014 at 1:30 PM, Alper Kucukural al...@kucukural.com wrote: Hi All, I have installed the recent version of galaxy and starting multiple web and job handlers(six each) on a Centos 5.1 machine. It is working almost perfectly. 1. The first problem is sometimes jobs never start and they are in grey stage. After I killed them and start again, they work fine. There are no any log about why they haven't started. So, if you direct me to find the reason, it would be great. 2. Second problem is about UCSC visualization issue using the links from history. I haven't found a solution yet, my XSendFile is all set and working in the previous version but not this one. The only difference is I am using the new version and balancer in Apache since I run multiple instances. So, balancer might not use XSendFile properly or there is a configuration problem. http://lists.bx.psu.edu/pipermail/galaxy-user/2012-November/005508.html https://wiki.galaxyproject.org/Admin/Config/ApacheProxy Let me know, if anybody has a similar settings and solved this problem with Apache. Best, Alper Kucukural, PhD Bioinformatics Core, University of Massachusetts Medical School 368 Plantation St.Room AS4.2067 Worcester, MA 01605-2324 Phone: 774-312-4493 E-mail: al...@kucukural.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Persistent jobs in cluster queue even after canceling job in galaxy
I believe this problem was fixed by Nate after the latest dist release and pushed to the stable branch of galaxy-central. https://bitbucket.org/galaxy/galaxy-central/commits/1298d3f6aca59825d0eb3d32afd5686c4b1b9294 If you are eager for this bug fix, you can track the latest stable branch of galaxy-central instead of the galaxy-dist tag mentioned in the dev news. Right now it has some other good bug fixes not in the latest release. -John On Tue, Mar 11, 2014 at 9:53 PM, Brian Claywell bclay...@fhcrc.org wrote: We've had the same problem since updating from the April 2013 stable to Feb 2014 stable. Our jobs are going off to a slurm cluster via the SlurmJobRunner plugin (though this was happening with the DRMAAJobRunner plugin too, if I remember right). Removing pending datasets occasionally removes the entry in Admin/Jobs, but not reliably, and regardless the jobs stay queued in slurm with no noise in galaxy.log at DEBUG or higher. We're not using parallelism either. I noticed this around the same time as I noticed the new in-progress animation in the history pane; perhaps there's an ajax-y callback that's not firing? Otherwise I'd expect to see something in the log about Galaxy at least trying to cancel the job. On Tue, Mar 11, 2014 at 10:57 AM, Ravi Alla ravi.a...@berkeley.edu wrote: Hi All, I've been able to submit jobs to the cluster through galaxy, it works great. But when the job is in queue to run (it is gray in the galaxy history pane) and I cancel the job, it still remains in queue on the cluster. Why does this happen? How can I delete the jobs in queue as well? I tried qdel job_id as galaxy user but it says I am not authenticated to delete the job. Any help would be greatly appreciated. Thanks Ravi. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Brian Claywell, Systems Analyst/Programmer Fred Hutchinson Cancer Research Center bclay...@fhcrc.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] How to install bowtie2 tool in galaxy
On Wed, Mar 5, 2014 at 8:18 PM, Ravi Alla ravi.a...@berkeley.edu wrote: Hi Jennifer, Thank you for this information. I was able to troubleshoot the bowtie2_indices. Like Bjoern said the bowtie2 tool needed to be reinstalled because the tool_table does not load the correct indices. I am still trying to wrap my head around the different toolsheds. The main tool shed resides in the galaxy-dist folder and uses tool_conf.xml to load tools into the side bar and tool_data_table_conf.xml to load indices (and other data) for the default tools that come with galaxy. Hi Ravi, The tools in the galaxy-dist directory and controlled via tool_conf.xml are not a part of the tool shed. The inclusion of tools directly in Galaxy predates the existence of the tool shed. Other than that, what you have above is correct. The shed tools reside in the ../shed_tools/ directory and use shed_tools_conf.xml to load tools into the side bar and the shed_tools_data_table_conf.xml to load indices. The shed_tool xml is setup in the universe.ini file. Right. The .loc files for default main tools reside in galaxy-dist/tool-data and the .loc files for the shed_tools reside in ../shed_tools/.../tool-name/tool_id/tool-name/ By default they'll all be under galaxy-dist/tool-data/. The versions in ../shed_tools are the sample location files provided by tool authors. And when I uninstall tools and reinstall I would have to update the metadata for that tool. I am not sure what you mean by this. If ../shed_tools dir is not present then shed-tools get installed under galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/ ../shed_tools should be created if it does not exist. Tools won't be installed in galaxy-dist/tool-data/ And it is always better to install tools as wrappers + dependencies when possible. I agree. --nate Am I on the right track with this tool organization? Thanks Ravi On Mar 5, 2014, at 11:06 AM, Jennifer Jackson j...@bx.psu.edu wrote: Hi Ravi, The directory structure for the installation of ToolShed tools changed, which is why you have three directories. You perhaps had bowtie2 installed once before, then reinstalled (without completely removing the older version and associated data)? Or updated without resetting the metadata? In either case, the ../shed_tools directory is the new one. Having this as the path in your .xml configuration files (as Bjoern suggested earlier) and moving all contents data to be under the same location will be the simplest global solution ongoing. Links near the end of my reply can help explain how-to. For the specific reason why bowtie2 indices are not working: I noticed that the reference genome .fa file is not linked from the directory containing the indexes. This is required. Adding it in, the same way that you did for the bowtie2 indexes, into this dir: /global/referenceData/databases/bowtie2/hg19 will probably solve that part of the problem. I didn't see this posted - but I apologize if I am duplicating advice already given. I also tend to advise keeping all data under the same master data directory - all indexes and sequence data - as symbolic links to additional file system paths that are unknown to the 'galaxy user' cause a different set of problems. However, that said, this doesn't seem to be an issue in your specific case: if the bowtie2 indexes are functioning correctly - then environment is set up so that the other dir hierarchy where the .fa files are kept must included in the 'galaxy' user's ENV. Symbolic links that go outside of the local dir structure are known to cause problems unless the ENV config is carefully set up, and to my knowledge are best avoided entirely for certain uses such as upload by file path into libraries. For reference: NGS data set-up is described in this wiki - including expected content for each type of index: https://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup For examples, you can rsync a genome or two and examine the contents. Or, our /location dir and have a look at the .loc files. https://wiki.galaxyproject.org/Admin/DataIntegration Tool Shed help (very detailed): https://wiki.galaxyproject.org/Tool%20Shed In particular, if you had previously installed repositories (this is not clear, just suspected from the duplications), updating the Metadata with certain distribution updates can be very important. This has been necessary for the last few releases to update to changes. The News Brief noted this, and included a link to this wiki page. Also see the Related Pages lower down on the wiki. https://wiki.galaxyproject.org/ResettingMetadataForInstalledRepositories This may also be useful: https://wiki.galaxyproject.org/RepairingInstalledRepositories Hopefully you have sorted most of this out by now, or this helps! Jen Galaxy team On 3/4/14 12:21 PM, Ravi Alla wrote: Bjoern, This is getting frustrating. There are three places bowtie2_indices.loc file is
Re: [galaxy-dev] Tool Shed install best practise : Precompiled binaries vs local compile
Hi Peter, I just wanted to add my $0.02 USD to say that I mostly agree with this - I have long used binaries precompiled by the tool author on Main, especially for cases where, as you say, the compile-time dependency list is large and painful. The only gotcha here is to make sure that binaries support the oldest possible version of glibc that might be installed on any of the Linux distributions and versions that we support, and that no non-standard preinstalled libraries have been pulled in during the build process. --nate On Thu, Mar 6, 2014 at 12:46 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Greg, I've retitled the thread, previously about a ToolShed nightly test failure. A brief recap, we're talking about the Galaxy ToolShed XML installation recipes for the NCBI BLAST+ packages and my MIRA4 wrapper in their tool_dependencies.xml files: http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_29 http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_29 http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler These use the pattern of having os/arch specific action tags (which download and install the tool author's precompiled binaries) and a fall back default action which is to report an error with the os/arch combination and that there are no ready made binaries available. Greg is instead advocating the fall back action be to download the source code, and do a local compile. My reply is below... On Thu, Mar 6, 2014 at 5:24 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Mar 6, 2014 at 4:53 PM, Greg Von Kuster g...@bx.psu.edu wrote: As we briefly discussed earlier, your mira4 recipe is not currently following best practices. Although you uncovered a problem in the framework which has now been corrected, your recipe's fall back actions tag set should be the recipe for installing mira4 from source ( http://sourceforge.net/projects/mira-assembler/ ) since there is no licensing issues for doing so. This would be a more ideal approach than echoing the error messages. Thanks very much for helping us discover this problem though! Greg Von Kuster Hi Greg, No problem - I'm good at discovering problems ;) If the download approach failed, it it most likely due to a transient error (e.g. network issues with download). Here I would much prefer Galaxy aborted and reported this as an error (and does not attempt the default action). Is that what you just fixed? As to best practice for the fall back action, I think that needs a new thread. Regards, Peter As to best practice, I do not agree that in cases like this (MIRA4, NCBI BLAST+) where there are provided binaries for the major platforms that the fall back should be compiling from source. The NCBI BLAST+ provide binaries for 32 bit and 64 bit Linux and Mac OS X (which I believe covers all the mainstream platforms Galaxy runs on). Similarly, MIRA4 provides binaries for 64 bit Linux and Mac OS X. Note that 32 bit binaries are not provided, but would be very restricted in terms of the datasets they could be used on anyway - and I doubt many of the systems Galaxy runs on these days are 32 bits. If the os/arch combination is exotic enough that precompiled binaries are not available, then it is likely compilation will be tricky anyway - or not supported for that tool, or Galaxy itself. Essentially I am arguing that where the precompiled tool binaries cover any mainstream system Galaxy might be used on, a local compile fall back is not needed. Also, these are both complex tools which are relatively slow to compile, and have quite a large compile time dependency set (e.g. MIRA4 requires at least a quite recent GCC, BOOST, flex, expat, and strongly recommends TCmalloc). Here at least some of the dependencies have been packaged for the ToolShed (probably by Bjoern?) but in the case of MIRA4 and BLAST+ this is still a lot of effort for no practical gain. I also feel there is an argument that the Galaxy goal of reproducibility should favour using precompiled binaries if available: A locally compiled binary will generally mean a different compiler version, perhaps with different optimisation flags, and different library versions. It will not necessarily give the same results as the tool author's provided precompiled binary. (Wow, this ended up being a long email!) Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists,
Re: [galaxy-dev] Tool Shed install best practise : Precompiled binaries vs local compile
On Fri, Mar 14, 2014 at 4:41 PM, Nate Coraor n...@bx.psu.edu wrote: Hi Peter, I just wanted to add my $0.02 USD to say that I mostly agree with this - I have long used binaries precompiled by the tool author on Main, especially for cases where, as you say, the compile-time dependency list is large and painful. The only gotcha here is to make sure that binaries support the oldest possible version of glibc that might be installed on any of the Linux distributions and versions that we support, and that no non-standard preinstalled libraries have been pulled in during the build process. --nate Good point about potential problems with glibc, and I guess any run time linked libraries which might be a different version or missing? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed install best practise : Precompiled binaries vs local compile
On Fri, Mar 14, 2014 at 12:47 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Fri, Mar 14, 2014 at 4:41 PM, Nate Coraor n...@bx.psu.edu wrote: Hi Peter, I just wanted to add my $0.02 USD to say that I mostly agree with this - I have long used binaries precompiled by the tool author on Main, especially for cases where, as you say, the compile-time dependency list is large and painful. The only gotcha here is to make sure that binaries support the oldest possible version of glibc that might be installed on any of the Linux distributions and versions that we support, and that no non-standard preinstalled libraries have been pulled in during the build process. --nate Good point about potential problems with glibc, and I guess any run time linked libraries which might be a different version or missing? Peter Right, many things (especially using autoconf) will automatically link against libraries if present and disable functionality if not. Hopefully if we were returning to mostly vanilla tool test VMs in the future, any problems would be easily discoverable. --nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] How to install bowtie2 tool in galaxy
Hi Nate Thanks for the clarifications. Is there anything specific I should watch out for when updating and deleting tools? The reason I ask is that Jennifer mentioned in previous emails the need to update metadata for tools, which I am sure I follow. Thanks Ravi On Mar 14, 2014, at 9:02 AM, Nate Coraor n...@bx.psu.edu wrote: On Wed, Mar 5, 2014 at 8:18 PM, Ravi Alla ravi.a...@berkeley.edu wrote: Hi Jennifer, Thank you for this information. I was able to troubleshoot the bowtie2_indices. Like Bjoern said the bowtie2 tool needed to be reinstalled because the tool_table does not load the correct indices. I am still trying to wrap my head around the different toolsheds. The main tool shed resides in the galaxy-dist folder and uses tool_conf.xml to load tools into the side bar and tool_data_table_conf.xml to load indices (and other data) for the default tools that come with galaxy. Hi Ravi, The tools in the galaxy-dist directory and controlled via tool_conf.xml are not a part of the tool shed. The inclusion of tools directly in Galaxy predates the existence of the tool shed. Other than that, what you have above is correct. The shed tools reside in the ../shed_tools/ directory and use shed_tools_conf.xml to load tools into the side bar and the shed_tools_data_table_conf.xml to load indices. The shed_tool xml is setup in the universe.ini file. Right. The .loc files for default main tools reside in galaxy-dist/tool-data and the .loc files for the shed_tools reside in ../shed_tools/.../tool-name/tool_id/tool-name/ By default they'll all be under galaxy-dist/tool-data/. The versions in ../shed_tools are the sample location files provided by tool authors. And when I uninstall tools and reinstall I would have to update the metadata for that tool. I am not sure what you mean by this. If ../shed_tools dir is not present then shed-tools get installed under galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/ ../shed_tools should be created if it does not exist. Tools won't be installed in galaxy-dist/tool-data/ And it is always better to install tools as wrappers + dependencies when possible. I agree. --nate Am I on the right track with this tool organization? Thanks Ravi On Mar 5, 2014, at 11:06 AM, Jennifer Jackson j...@bx.psu.edu wrote: Hi Ravi, The directory structure for the installation of ToolShed tools changed, which is why you have three directories. You perhaps had bowtie2 installed once before, then reinstalled (without completely removing the older version and associated data)? Or updated without resetting the metadata? In either case, the ../shed_tools directory is the new one. Having this as the path in your .xml configuration files (as Bjoern suggested earlier) and moving all contents data to be under the same location will be the simplest global solution ongoing. Links near the end of my reply can help explain how-to. For the specific reason why bowtie2 indices are not working: I noticed that the reference genome .fa file is not linked from the directory containing the indexes. This is required. Adding it in, the same way that you did for the bowtie2 indexes, into this dir: /global/referenceData/databases/bowtie2/hg19 will probably solve that part of the problem. I didn't see this posted - but I apologize if I am duplicating advice already given. I also tend to advise keeping all data under the same master data directory - all indexes and sequence data - as symbolic links to additional file system paths that are unknown to the 'galaxy user' cause a different set of problems. However, that said, this doesn't seem to be an issue in your specific case: if the bowtie2 indexes are functioning correctly - then environment is set up so that the other dir hierarchy where the .fa files are kept must included in the 'galaxy' user's ENV. Symbolic links that go outside of the local dir structure are known to cause problems unless the ENV config is carefully set up, and to my knowledge are best avoided entirely for certain uses such as upload by file path into libraries. For reference: NGS data set-up is described in this wiki - including expected content for each type of index: https://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup For examples, you can rsync a genome or two and examine the contents. Or, our /location dir and have a look at the .loc files. https://wiki.galaxyproject.org/Admin/DataIntegration Tool Shed help (very detailed): https://wiki.galaxyproject.org/Tool%20Shed In particular, if you had previously installed repositories (this is not clear, just suspected from the duplications), updating the Metadata with certain distribution updates can be very important. This has been necessary for the last few releases to update to changes. The News Brief noted this, and included a link to this wiki page. Also see the Related Pages lower down on the wiki.
Re: [galaxy-dev] Empty history pane
Our use was using the lastest Firefox release. I asked him to try FireFox Portable and that seemed to do the trick. It was a problem with his Firefox install. Thanks for you help, Liisa From: Dannon Baker dannon.ba...@gmail.com To: Liisa Koski liisa.ko...@basf.com Cc: Galaxy Dev galaxy-dev@lists.bx.psu.edu Date: 13/03/2014 06:39 PM Subject:Re: [galaxy-dev] Empty history pane If it's possible, can you check (or ask the user to check) if there are any javascript errors if you open the browser console when experiencing this failure? On Thu, Mar 13, 2014 at 2:07 PM, Liisa Koski liisa.ko...@basf.com wrote: Hello, Our site maintains a local Galaxy installation (Nov.4th distributuion). One of our usrers has lost the ability to view anything in his history pane. He is able to run tools and view the list of 'Saved Histories' in the middle pane but the History pane itself stays empty. When I impersonate him I can see his history pane. Very weird. Any insight would be much appreciated. Thanks, Liisa ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Internal server error when importing workflows!
Hmm... I can import older workflows on usegalaxy.org so I don't think it is a general problem with all workflows as a part of the latest release. That said I don't know what the problem is - if it is a bug with a new release or a problem with one of your tools etc If I had to guess I would say you updated a tool has been updated changing its interface without changing its version? Can you send the list (or me) the workflow and any custom tool XML files it uses? Do you still have access to the original Galaxy instance with the workflow in it - if you could try to trim the number of steps down and retry until it works or try a completely new workflow that might help track down the particular circumstances surrounding the problem. -John On Tue, Mar 11, 2014 at 11:43 AM, Hakeem Almabrazi halmabr...@idtdna.com wrote: Hi, I have upgraded my local galaxy to the latest stable version and it works fine but I get an error when I try to import workflows that were created older version of Galaxy. Here is the error I keep getting. Internal Server Error Galaxy was unable to successfully complete your request URL: http://10.7.10.21/workflow/import_workflow Module galaxy.web.framework.middleware.error:149 in __call__ app_iter = self.application(environ, sr_checker) Module paste.recursive:84 in __call__ return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request body = method( trans, **kwargs ) Module galaxy.webapps.galaxy.controllers.workflow:1072 in import_workflow workflow, missing_tool_tups = self._workflow_from_dict( trans, data, source=src, add_to_menu=add_to_menu ) Module galaxy.web.base.controller:1639 in _workflow_from_dict module.save_to_step( step ) Module galaxy.workflow.modules:274 in save_to_step step.tool_inputs = self.tool.params_to_strings( self.state.inputs, self.trans.app ) Module galaxy.tools:2418 in params_to_strings return params_to_strings( self.inputs, params, app ) Module galaxy.tools.parameters:85 in params_to_strings value = params[ key ].value_to_basic( value, app ) Module galaxy.tools.parameters.grouping:468 in value_to_basic rval[ input.name ] = input.value_to_basic( value[ input.name ], app ) KeyError: 'index' Can someone let me know how to fix this issue? Thank you Hakeem ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] How To properly use GALAXY_SLOTS
This is strange.
After restarting galaxy to include the debug tool, all galaxy_slots
values are correctly assigned (tested for BWA and GATK as rerun of
original posted jobs)
If you still need the output, let me know.
Best,
Geert
On 03/14/2014 03:25 PM, John Chilton wrote:
I am not sure I understand what you mean, but if you are setting the
job to run with a ppn of greater than one but GALAXY_SLOTS is being
evaluated as 1 then you have likely uncovered a bug - maybe in the
runner, your job configuration, or the GALAXY_SLOTS logic for
pbs/torque.
I have created a Galaxy tool for debugging the problem -
https://gist.github.com/jmchilton/9548516. If you could run in your
environment and assign it multiple cores it should spit out some
interesting information relevant to debugging the problem. It will
create Galaxy datasets corresponding to the runtime-computed Galaxy
slots, the contents of PBS_NODEFILE which Galaxy attempts to use to
compute Galaxy slots, and the full contents of your worker node
environment. This last one may contain sensitive information - but if
you could post the first two or send them to me directly it would
hopefully help debug the problem.
-John
On Wed, Mar 12, 2014 at 12:37 PM, Geert Vandeweyer
geert.vandewey...@uantwerpen.be wrote:
How would those statements translate to pbs/torque (pbs_python)
I request resources using the nodes=1:ppn:4 syntax. The -pe argument is not
available it seems...
Best,
Geert
On 03/12/2014 04:43 PM, Björn Grüning wrote:
Hi Geert,
you can give every tool a different amount of SLOTS via the job_conf.xml
file.
https://wiki.galaxyproject.org/Admin/Config/Jobs
For example:
tool
id=toolshed.g2.bx.psu.edu/repos/iuc/gatk2/gatk2_variant_recalibrator
destination=12cores_24G /
where 12cores_24G is defined with a parallel SGE environment.
Cheers,
Bjoern
Am 12.03.2014 16:32, schrieb Geert Vandeweyer:
Hi,
Is there documentation for the proper setup and utilisation of the
\${GALAXY_SLOTS:-4} style options?
I noticed that tools with this setting, including BWA and GATK run
single threaded, with the following settings respectively:
BWA \${GALAXY_SLOTS:-4}
===
ps output: python
/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/b4427dbb6ced/bwa_wrappers/bwa_wrapper.py
--threads=1 --fileSource=indexed
--ref=/galaxy/galaxy_references/hg19/bwa-0.5.9/base/hg19.fasta
--do_not_build_index
--input1=/galaxy/galaxy-dist/database/files/093/dataset_93021.dat
--input2=/galaxy/galaxy-dist/database/files/093/dataset_93023.dat
--output=/galaxy/galaxy-dist/database/files/000/112/dataset_112301.dat
--genAlignType=paired --params=pre_set --suppressHeader=false
xml config: --threads=\${GALAXY_SLOTS:-4}
GATK Base Recalibrator \${GALAXY_SLOTS:-8}
==
python
/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/gatk2/8bcc13094767/gatk2/gatk2_wrapper.py
--max_jvm_heap 6g --stdout
/galaxy/galaxy-dist/database/files/000/112/dataset_112292.dat -d -I
/galaxy/galaxy-dist/database/files/000/112/dataset_112289.dat bam
gatk_input -d
/galaxy/galaxy-dist/database/files/_metadata_files/012/metadata_12686.dat
bam_index
gatk_input -p java -jar
$GATK2_PATH/GenomeAnalysisTK.jar-T BaseRecalibrator
$GATK2_SITE_OPTIONS --num_cpu_threads_per_data_thread
${GALAXY_SLOTS:-8} --no_standard_covs -R
/galaxy/galaxy_references/hg19/srma-0.1.15/hg19.fasta --out
/galaxy/galaxy-dist/database/files/000/112/dataset_112291.dat -cov
ContextCovariate -cov CycleCovariate -d
--knownSites:dbsnp,%(file_type)s
/galaxy/galaxy-dist/database/files/000/101/dataset_101086.dat vcf
input_dbsnp_0 -p --pedigreeValidationType STRICT -d --intervals
/galaxy/galaxy-dist/database/files/000/100/dataset_100129.dat bed
input_intervals_0 -p --interval_set_rule UNION -p --downsampling_type
NONE -p --baq OFF --baqGapOpenPenalty
40.0 --defaultBaseQualities -1
--validation_strictness STRICT --interval_merging ALL
java -Xmx6g -jar
/galaxy/galaxy-dist/tool-data/shared/jars/gatk2//GenomeAnalysisTK.jar -T
BaseRecalibrator --num_cpu_threads_per_data_thread 1 --no_standard_covs
-R /galaxy/galaxy_references/hg19/srma-0.1.15/hg19.fasta --out
/galaxy/galaxy-dist/database/files/000/112/dataset_112291.dat -cov
ContextCovariate -cov CycleCovariate --pedigreeValidationType STRICT
--interval_set_rule UNION --downsampling_type NONE --baq OFF
--baqGapOpenPenalty 40.0 --defaultBaseQualities -1
--validation_strictness STRICT --interval_merging ALL -I
/tmp/tmp-gatk-fIuncH/gatk_input.bam --knownSites:dbsnp,vcf
/tmp/tmp-gatk-fIuncH/input_dbsnp_0.vcf --intervals
/tmp/tmp-gatk-fIuncH/input_intervals_0.bed
Best,
Geert
--
Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: geert.vandewe...@ua.ac.be
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/in/geertvandeweyer
Re: [galaxy-dev] Empty history pane
Good to hear you've worked around it! For future reference, in Firefox you can open the javascript console by clicking the Firefox (menu at the top)-Web Developer-Web Console. That, or Control/Command+Shift+K. On Fri, Mar 14, 2014 at 1:49 PM, Liisa Koski liisa.ko...@basf.com wrote: Our use was using the lastest Firefox release. I asked him to try FireFox Portable and that seemed to do the trick. It was a problem with his Firefox install. Thanks for you help, Liisa From:Dannon Baker dannon.ba...@gmail.com To:Liisa Koski liisa.ko...@basf.com Cc:Galaxy Dev galaxy-dev@lists.bx.psu.edu Date:13/03/2014 06:39 PM Subject:Re: [galaxy-dev] Empty history pane -- If it's possible, can you check (or ask the user to check) if there are any javascript errors if you open the browser console when experiencing this failure? On Thu, Mar 13, 2014 at 2:07 PM, Liisa Koski *liisa.ko...@basf.com*liisa.ko...@basf.com wrote: Hello, Our site maintains a local Galaxy installation (Nov.4th distributuion). One of our usrers has lost the ability to view anything in his history pane. He is able to run tools and view the list of 'Saved Histories' in the middle pane but the History pane itself stays empty. When I impersonate him I can see his history pane. Very weird. Any insight would be much appreciated. Thanks, Liisa ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: *http://lists.bx.psu.edu/* http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: *http://galaxyproject.org/search/mailinglists/*http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Puppet Modules: a new project
Dear galaxy-dev, After exchanges with Eric we have decided to create: -a common puppet module for galaxy hosted on the puppet forge: https://forge.puppetlabs.com/urgi/galaxy -a common repository hosted on a public server: We want to have the opinion of the galaxy-team and galaxy-dev community for the choice of the public server. Should we push the code on github or bitbucket ? Does this question matter anyway ? :) Thanks for your answers, Olivier Olivier Inizan Unité de Recherches en Génomique-Info (UR INRA 1164), INRA, Centre de recherche de Versailles, bat.18 RD10, route de Saint Cyr 78026 Versailles Cedex, FRANCE olivier.ini...@versailles.inra.fr Tél: +33 1 30 83 38 25 Fax: +33 1 30 83 38 99 http://urgi.versailles.inra.fr [urgi.versailles.inra.fr] Twitter: @OlivierInizan On Thu, 13 Mar 2014, Olivier Inizan wrote: Hi Eric and Sys Admin, It's a great news ! We have already published a puppet module for a basic galaxy install and update: http://forge.puppetlabs.com/urgi/galaxy We also plan to update it for configuration files management. Eric, it could be a great initiative to merge our efforts in a single module. Any ideas ? Cheers, Olivier. Olivier Inizan Unité de Recherches en Génomique-Info (UR INRA 1164), INRA, Centre de recherche de Versailles, bat.18 RD10, route de Saint Cyr 78026 Versailles Cedex, FRANCE olivier.ini...@versailles.inra.fr Tél: +33 1 30 83 38 25 Fax: +33 1 30 83 38 99 http://urgi.versailles.inra.fr [urgi.versailles.inra.fr] Twitter: @OlivierInizan On Wed, 12 Mar 2014, Eric Rasche wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Sys Admins, I'm working on some puppet modules for managing galaxy. I'm not sure how many of you use puppet, but I figured it would be useful to the community. I'll place them on puppet forge when I'm done. So far I'm doing the following: - - enabling initial deployment (updating?) with the vcsrepo module - - managing the following files: job_conf.xml, tool_conf.xml, tool_sheds_conf.xml, universe_wsgi.ini If there's anything that you find important to be managed, or features you'd like to see, please email me and let me know! Cheers, Eric - -- Eric Rasche Programmer II Center for Phage Technology Texas AM University College Station, TX 77843 404-692-2048 e...@tamu.edu rasche.e...@yandex.ru -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) iQIcBAEBAgAGBQJTIH0RAAoJEMqDXdrsMcpV2g8P/08nTnb/RLdz6WGfDbryHtBB 6Q5yr4HV2IM/idgjpLQX93jRCyahVuS1E8Sgc1x9ViGml93/ssHJZ9GIsf1PJM5/ erygrVS6njhWgefMQpoEipfTOHqxCvHhJk5xnvRc/EeUj0Lkj6YKoVpzae/6XilH Tav9/ocTNrH40HBjuFGsK7Q/IP1C+vNr/q7GPXq6Ek6dWd23qxRCYszuGgS2t3s/ i/+YmZCwHH0I25ssujGsYzsrsTfOBVBNvAj9dUmvyE+9WSLZQVw4919VKXDKNuSs 7oFBPJD2hpqSKJxf7IFChFL9WhQ4e1gykx+Z+7MhBpekUeYvwIzABxFNmQLOdl+Q 4R+mEADyAopD2enl23XkePX/FRag839zYSeEOWjvEfC8qgUWFXEJyjJ9JLqXE+Bc QOAMpmC/BMI5qIxIvKJ2ASZuqBXFxdRcGs1xaqJ2bWN+vm1fD+jzpD33KjHcegQr gf0UvhJj3aoR0Pua3vuWkWP6wPBt08F6Qab/6tQGIeQUyzZRCGyVggd4SgN+Xml5 0OB5olauA+Nr0wD+tDiCQwE+iIL/U61nU5A0yCmEMVfBem8YyKPtjuNZ7E/OZ9fy gv4uVmGDIfM+IySnWlmxqX2nYb8AODrZiQifT3kG93jh4HtNaoc4yTNeRYcFwTNi r8tJpajbO8Q0MVVj+bWn =aFP5 -END PGP SIGNATURE- ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Puppet Modules: a new project
I love this effort! I would recommend posting these on github instead of bitbucket - I think it has become kind of a puppet best practice to host collaborative repositories such as these there. For what it is worth - I have a set of puppet modules for hosting Galaxy that have a dependency on the puppetlabs Apache module - https://github.com/jmchilton/puppet-jmchilton/tree/master/modules/galaxy/ - they are not general at all but I will try to find them time to refactor my stuff against your effort and pushes enhancements of general usefulness if I can. I think most of the rest of the Galaxy team prefers Ansible to puppet (for instance Nate will likely release some Ansible roles he uses to manage usegalaxy.org in the near future) - but I think the two efforts will dovetail nicely and hopefully provide roadmaps for best practice examples for people who are deploying and managing Galaxy - however they do it. -John On Fri, Mar 14, 2014 at 9:20 AM, Olivier Inizan olivier.ini...@versailles.inra.fr wrote: Dear galaxy-dev, After exchanges with Eric we have decided to create: -a common puppet module for galaxy hosted on the puppet forge: https://forge.puppetlabs.com/urgi/galaxy -a common repository hosted on a public server: We want to have the opinion of the galaxy-team and galaxy-dev community for the choice of the public server. Should we push the code on github or bitbucket ? Does this question matter anyway ? :) Thanks for your answers, Olivier Olivier Inizan Unité de Recherches en Génomique-Info (UR INRA 1164), INRA, Centre de recherche de Versailles, bat.18 RD10, route de Saint Cyr 78026 Versailles Cedex, FRANCE olivier.ini...@versailles.inra.fr Tél: +33 1 30 83 38 25 Fax: +33 1 30 83 38 99 http://urgi.versailles.inra.fr [urgi.versailles.inra.fr] Twitter: @OlivierInizan On Thu, 13 Mar 2014, Olivier Inizan wrote: Hi Eric and Sys Admin, It's a great news ! We have already published a puppet module for a basic galaxy install and update: http://forge.puppetlabs.com/urgi/galaxy We also plan to update it for configuration files management. Eric, it could be a great initiative to merge our efforts in a single module. Any ideas ? Cheers, Olivier. Olivier Inizan Unité de Recherches en Génomique-Info (UR INRA 1164), INRA, Centre de recherche de Versailles, bat.18 RD10, route de Saint Cyr 78026 Versailles Cedex, FRANCE olivier.ini...@versailles.inra.fr Tél: +33 1 30 83 38 25 Fax: +33 1 30 83 38 99 http://urgi.versailles.inra.fr [urgi.versailles.inra.fr] Twitter: @OlivierInizan On Wed, 12 Mar 2014, Eric Rasche wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Sys Admins, I'm working on some puppet modules for managing galaxy. I'm not sure how many of you use puppet, but I figured it would be useful to the community. I'll place them on puppet forge when I'm done. So far I'm doing the following: - - enabling initial deployment (updating?) with the vcsrepo module - - managing the following files: job_conf.xml, tool_conf.xml, tool_sheds_conf.xml, universe_wsgi.ini If there's anything that you find important to be managed, or features you'd like to see, please email me and let me know! Cheers, Eric - -- Eric Rasche Programmer II Center for Phage Technology Texas AM University College Station, TX 77843 404-692-2048 e...@tamu.edu rasche.e...@yandex.ru -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) iQIcBAEBAgAGBQJTIH0RAAoJEMqDXdrsMcpV2g8P/08nTnb/RLdz6WGfDbryHtBB 6Q5yr4HV2IM/idgjpLQX93jRCyahVuS1E8Sgc1x9ViGml93/ssHJZ9GIsf1PJM5/ erygrVS6njhWgefMQpoEipfTOHqxCvHhJk5xnvRc/EeUj0Lkj6YKoVpzae/6XilH Tav9/ocTNrH40HBjuFGsK7Q/IP1C+vNr/q7GPXq6Ek6dWd23qxRCYszuGgS2t3s/ i/+YmZCwHH0I25ssujGsYzsrsTfOBVBNvAj9dUmvyE+9WSLZQVw4919VKXDKNuSs 7oFBPJD2hpqSKJxf7IFChFL9WhQ4e1gykx+Z+7MhBpekUeYvwIzABxFNmQLOdl+Q 4R+mEADyAopD2enl23XkePX/FRag839zYSeEOWjvEfC8qgUWFXEJyjJ9JLqXE+Bc QOAMpmC/BMI5qIxIvKJ2ASZuqBXFxdRcGs1xaqJ2bWN+vm1fD+jzpD33KjHcegQr gf0UvhJj3aoR0Pua3vuWkWP6wPBt08F6Qab/6tQGIeQUyzZRCGyVggd4SgN+Xml5 0OB5olauA+Nr0wD+tDiCQwE+iIL/U61nU5A0yCmEMVfBem8YyKPtjuNZ7E/OZ9fy gv4uVmGDIfM+IySnWlmxqX2nYb8AODrZiQifT3kG93jh4HtNaoc4yTNeRYcFwTNi r8tJpajbO8Q0MVVj+bWn =aFP5 -END PGP SIGNATURE- ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Persistent jobs in cluster queue even after canceling job in galaxy
On Fri, Mar 14, 2014 at 9:01 AM, John Chilton jmchil...@gmail.com wrote: I believe this problem was fixed by Nate after the latest dist release and pushed to the stable branch of galaxy-central. https://bitbucket.org/galaxy/galaxy-central/commits/1298d3f6aca59825d0eb3d32afd5686c4b1b9294 If you are eager for this bug fix, you can track the latest stable branch of galaxy-central instead of the galaxy-dist tag mentioned in the dev news. Right now it has some other good bug fixes not in the latest release. Ah, got it, thanks! Is it unfeasible to push bug fixes like those back to galaxy-dist/stable so those of us that would prefer stable to bleeding-edge don't have to cherry-pick commits? -- Brian Claywell, Systems Analyst/Programmer Fred Hutchinson Cancer Research Center bclay...@fhcrc.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Tool Testing Enhancements
Hello Tool Developers, Haven't known when to send this out, but I figure since you haven't received any e-mail from me today it might be a good time. tl;dr - Tool functional tests experienced a significant overhaul over the last release and will continue to change over the next couple releases, but in a backward compatible so hopefully you will not need to care unless you want to. The last release included a considerable overhaul to the framework used to functionally test tools (i.e. actually running those test tags in tool XML). This includes changes to how the tests are executed and an expansion in the expressiveness of these tests. As some background for why the changes are being made, currently tools are tested by starting up a test Galaxy instance and using a library called Twill to exercise the GUI and upload data and run tools through by exercising actual HTML pages. The last release added the optional ability to exercise these tool tests via Galaxy's tool API instead of via web forms, with the slightly longer term goal of eliminating the Twill testing option altogether. This will allow the Galaxy tool form API to evolve more quickly, eliminate much of the testing complexity and many of the limitations imposed by Twill while ensuring that Galaxy's tool running API is sufficiently expressive. I have put a significant amount of effort into backward compatibility - so all current tests SHOULD pass using the new method. I know some people have included significant hacks with their test definitions to sculpt them deal with Twill-isms. These may fail - I would like to catalog what these are and see if we can hack up the test framework to deal with them or transition these tests to use cleaner mechanisms allowed for by the more direct representation of tool parameters consumed by the API. If you are running tests locally with the run_functional_tests.sh script distributed with Galaxy, simply setting the environment variable GALAXY_TEST_DEFAULT_INTERACTOR to 'api' before running the script and all tests will be run through the API. Individual tests can be set to target the API (for instance if they use newer, API-only features outlined below) by adding the XML attribute interactor=api to the test tag in the tool XML. This change works right now if you want to update test tool shed tools to target the new testing method and provide feedback. As part of this transition, many bugs were fixed that affect both styles of test and new features were added. The biggest improvement is the ability to specify parameters in a nested structure matching the inputs themselves. This allows disambiguation of repeat and conditional parameters for both styles of functional tests and in the case of API driven tests allows specification of any number of repeat block instances in tests (one still can only specify at most one instance for any repeat tag while using Twill driven tests). The following annotated tool XML examples concretely demonstrate this new expressiveness. https://bitbucket.org/galaxy/galaxy-central/src/tip/test/functional/tools/disambiguate_repeats.xml https://bitbucket.org/galaxy/galaxy-central/src/tip/test/functional/tools/disambiguate_cond.xml Additional Enhancements: Enhancement (Twill and API): Allow unspecified conditional test parameters. Previously specifying a parameter in a conditional but not the value of the conditional test parameter itself (the select or boolean parameter the conditional changes based on) resulted in cryptic test errors. Now the default is assumed if unspecified. https://bitbucket.org/galaxy/galaxy-central/commits/8b99c7bc5e11cd7c452c2ad86f951109b495271c https://bitbucket.org/galaxy/galaxy-central/commits/76bd49e658876f5cc90ec5b80033f13b058593b8 Enhancement (Twill and API): Allow specifying tests for composite extra files without checking primary file. https://bitbucket.org/galaxy/galaxy-central/commits/45c1d11cfff552ce095c0561b5fba2cb5d3610e3 Enhancement (API-only): Allow testing metadata properties on test output datasets. https://bitbucket.org/galaxy/galaxy-central/commits/01f05ab8df7092b6e03f35faf7f80b47acafa0eb Enhancement (API-only): Remove restriction of output order specification. https://bitbucket.org/galaxy/galaxy-central/commits/d05be33ad6b772c2a688d875c4cf895d6afac4e3 Enhancement: Allow testing repeat blocks with min=1 (twill) or any non-zero value (API) https://bitbucket.org/galaxy/galaxy-central/commits/4615f7424dcb4d2b61be5f388c761f45a1302a03 Enhancement (API-only): Allow testing of data parameters with multiple=True. https://bitbucket.org/galaxy/galaxy-central/commits/211f30207d48101a1250983d4f0f55c937f00a2e Fuller break down of this is development is available on Trello https://trello.com/c/MsJctFem. Once Galaxy and the tool shed have been modified to target the API when testing tools by default the tool XML syntax wiki page will be updated with examples of this new functionality. -John
Re: [galaxy-dev] Verifying test output datatypes, was: Problem with change_format and conditional inputs?
On Fri, Mar 14, 2014 at 10:24 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Thanks John, I suggest making this test framework perform this check by default (the twill and API based frameworks) and seeing what - if anything - breaks as a result on the Test Tool Shed. Hey Peter, I hope it is okay, but I do not want to make this change to the Twill driven variant of tool tests, I consider that code at end of life - new development would be a waste I think. Running all tools against a modified environment that switched all tests to target the APIs would be nice, but it sounds like there is not really the infrastructure in place for doing that right now. Upon further consideration I am not sure there are really any backward compatibility concerns anyway - or at least no more so than anything else when switching over to the API driven tests. I'll let the pull request sit open for a few days and then merge it as is. Note that one area of fuzziness is subclasses, e.g. if the tool output was labelled fastqsanger, but the test just said fastq, I would say the test was broken. On the other hand, if the test used a specific datatype like fastqsanger but the tool produced a dataset tagged with a more generic datatype like fastq I think that is a again a real failure. 100% agreed on both points. I believe the implementation proposed in pull request #347 reflects this resolution of the fuzziness. Thanks and have a great weekend, -John Peter On Fri, Mar 14, 2014 at 3:15 PM, John Chilton jmchil...@gmail.com wrote: Grepping around the code I think this is the only way a ftype attribute on an output affects the evaluation of test data. if attributes.get( 'ftype', None ) == 'bam': local_fh, temp_name = self._bam_to_sam( local_name, temp_name ) local_name = local_fh.name I am not sure it was ever meant as a strict test. I worry about breaking backward compatibility but it is easy enough to implement this as an actual check when using newer API driven tests. I have opened a pull request for this functionality here: https://bitbucket.org/galaxy/galaxy-central/pull-request/347/check-ftype-attribute-if-defined-on-test/diff Thoughts? -John On Tue, Mar 11, 2014 at 10:30 AM, Peter Cock p.j.a.c...@googlemail.com wrote: That seems to work - thanks Björn :) This seems to have exposed a bug in the test framework, e.g. test param name=query value=rhodopsin_nucs.fasta ftype=fasta / param name=db_opts_selector value=file / param name=subject value=three_human_mRNA.fasta ftype=fasta / param name=database value= / param name=evalue_cutoff value=1e-40 / param name=out_format value=5 / param name=adv_opts_selector value=basic / output name=output1 file=blastn_rhodopsin_vs_three_human.xml ftype=blastxml / /test This was saying the output should have been blastxml, but until I just fixed it the output was being tagged as tabular (although run_functional_tests.sh did check the content it didn't check the datatype matched). Dave - do think this is a reasonable enhancement? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
