date:20140314

On Thu, Mar 13, 2014 at 6:40 PM, Sanka, Ravi  wrote:
> I do not think so. Several individual datasets have been deleted (clicked
> the upper-right X on the history item box) but no History has been
> permanently deleted.
>
> Is there any indication in the database if target dataset or datasets were
> marked for permanent deletion? In the dataset table, I see fields
> "deleted", "purged", and "purgable", but nothing that says permanently
> deleted.

I would welcome clarification from the Galaxy Team, here and
on the wiki page which might benefit from a flow diagram?

https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20and%20Datasets

My assumption is using "permanently delete" in the user interface
marks an entry as "purgable", and then it will be moved to "purged"
(and the associated file on disk deleted) by the cleanup scripts -
but I'm a bit hazy on this any why it takes a while for a user's
usage figures to change.

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Pick-you-own columns in BLAST+ tabular output

On Tue, Mar 4, 2014 at 12:19 PM, Peter Cock  wrote:
> On Tue, Mar 4, 2014 at 10:35 AM, Björn Grüning  
> wrote:
>> Hi Peter,
>>
>>> Hello all,
>>>
>>> I've mentioned before that a forthcoming update to the BLAST+
>>> Galaxy wrappers would be adding a most customisable output
>>> option with pick-your-own columns. This is available to test now
>>> from the Test Tool Shed, along with other enhancements like
>>> more masking tools and an update to BLAST+ 2.2.29:
>>>
>>> http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus
>>>
>>> ...
>>>
>>>
>>> Some community testing/comments would be appreciated.
>>
>> I tested it briefly on my test instance and it looks all fine and
>> seems to work :)
>>
>> Thanks, looking forward to see it in MainTS.
>> Bjoern
>
> Great - thanks. I'm about to update our local production
> server, and barring any problems will push this BLAST+
> update to the main Tool Shed later this week.

This is now live, with a regression about output formats
caught and fixed:

http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus

Peter

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Quota History not getting purged

2014-03-14 Thread Alexander Kurze

Hi there,

We have installed a local copy of galaxy but there seems to be a strange
behavior with deleting data sets in the history panel. A few users complain
that they have deleted their data sets (I can confirm that) but somehow the
quota limit still doesn't show the appropriate freed space. Looking at
their saved histories and adding up their data the quota limit should have
changed.

Any clues why this happens? I have also tried to run the clean up script
but this didn't do anything on their quota limits.

Best,

Alex
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Quota History not getting purged

On Fri, Mar 14, 2014 at 12:28 PM, Alexander Kurze
 wrote:
> Hi there,
>
> We have installed a local copy of galaxy but there seems to be a strange
> behavior with deleting data sets in the history panel. A few users complain
> that they have deleted their data sets (I can confirm that) but somehow the
> quota limit still doesn't show the appropriate freed space. Looking at their
> saved histories and adding up their data the quota limit should have
> changed.
>
> Any clues why this happens? I have also tried to run the clean up script but
> this didn't do anything on their quota limits.
>
> Best,
>
> Alex

Does this help, or make no difference?

$ cd scripts
$ python set_user_disk_usage.py

It will recalculate and update the user disk totals, and prints this
to the terminal too.

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Purging datasets as part of workflow?

I don't even think you can delete datasets in workflow presently - let
alone permanently delete them - only "hide" them.

https://trello.com/c/YfLGkJKe

The core team just meet and a number of high priorities for the next 9
months were identified and reworking workflow scheduling was very high
on this list. I would expect an number of key workflow modifications
to be made by early fall that would make this much easier.

https://trello.com/c/K2qLZCrg

My own opinion on the particulars of this topic are that whether
datasets are deleted or permanently deleted should be up to the
workflow runner not the workflow author. In particular, the workflow
"model" describes what datasets are to be deleted and the workflow
runner could opt to permanently delete them if their Galaxy instance
allows this. It is exactly because this is not a universally allowed
option that I think it should not be part of the workflow description
that can be shared between instances and on the tool shed.

-John

On Thu, Mar 13, 2014 at 1:38 PM, Peter Cock  wrote:
> On Thu, Mar 13, 2014 at 6:30 PM, Sveinung Gundersen
>  wrote:
>> Hi,
>>
>> Is there an automatic way to permanently delete and purge
>> datasets that are the result of intermittent tools in a workflow?
>> That is, how can one automatically keep only the first and
>> the last steps of an workflow without having to manually
>> delete all the intermittent datasets and running the purge
>> scripts?
>>
>> Thanks,
>> Sveinung Gundersen
>
> +1
>
> In the workflows editor we can star particular outputs to be
> kept, and the rest are "deleted". I'd much prefer they be
> more aggressively "permanently deleted".
>
> Alternately, when editing a workflow there (used to be) a
> delete dataset action - how about a "permanently delete"
> action which would be great for some big-data workflows?
>
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Quota History not getting purged

2014-03-14 Thread Alexander Kurze

Perfect! This solved my problem.

Alex


On Fri, Mar 14, 2014 at 1:23 PM, Peter Cock wrote:

> On Fri, Mar 14, 2014 at 12:28 PM, Alexander Kurze
>  wrote:
> > Hi there,
> >
> > We have installed a local copy of galaxy but there seems to be a strange
> > behavior with deleting data sets in the history panel. A few users
> complain
> > that they have deleted their data sets (I can confirm that) but somehow
> the
> > quota limit still doesn't show the appropriate freed space. Looking at
> their
> > saved histories and adding up their data the quota limit should have
> > changed.
> >
> > Any clues why this happens? I have also tried to run the clean up script
> but
> > this didn't do anything on their quota limits.
> >
> > Best,
> >
> > Alex
>
> Does this help, or make no difference?
>
> $ cd scripts
> $ python set_user_disk_usage.py
>
> It will recalculate and update the user disk totals, and prints this
> to the terminal too.
>
> Peter
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Quota History not getting purged

On Fri, Mar 14, 2014 at 1:31 PM, Alexander Kurze
 wrote:
> Perfect! This solved my problem.
>
> Alex

That's progress. I suspect something is going wrong in the
cleanup scripts failing to update the totals - but as a workaround
you could include set_user_disk_usage.py as part of the cleanup
cron job?

Note set_user_disk_usage.py isn't on the wiki - so I assume
it isn't intended to be needed this way:
https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20and%20Datasets

I've been looking at our data usage figures this week as
part of a storage move - and also observed the totals can
fail to drop after a user permanently deleted some large
datasets & I'd run the cleanup scripts.

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] How To properly use GALAXY_SLOTS

I am not sure I understand what you mean, but if you are setting the
job to run with a ppn of greater than one but GALAXY_SLOTS is being
evaluated as 1 then you have likely uncovered a bug - maybe in the
runner, your job configuration, or the GALAXY_SLOTS logic for
pbs/torque.

I have created a Galaxy tool for debugging the problem -
https://gist.github.com/jmchilton/9548516. If you could run in your
environment and assign it multiple cores it should spit out some
interesting information relevant to debugging the problem. It will
create Galaxy datasets corresponding to the runtime-computed Galaxy
slots, the contents of PBS_NODEFILE which Galaxy attempts to use to
compute Galaxy slots, and the full contents of your worker node
environment. This last one may contain sensitive information - but if
you could post the first two or send them to me directly it would
hopefully help debug the problem.

-John

On Wed, Mar 12, 2014 at 12:37 PM, Geert Vandeweyer
 wrote:
> How would those statements translate to pbs/torque (pbs_python)
>
> I request resources using the nodes=1:ppn:4 syntax.  The -pe argument is not
> available it seems...
>
> Best,
>
> Geert
>
>
> On 03/12/2014 04:43 PM, Björn Grüning wrote:
>>
>> Hi Geert,
>>
>> you can give every tool a different amount of SLOTS via the job_conf.xml
>> file.
>>
>> https://wiki.galaxyproject.org/Admin/Config/Jobs
>>
>> For example:
>>
>> > id="toolshed.g2.bx.psu.edu/repos/iuc/gatk2/gatk2_variant_recalibrator"
>> destination="12cores_24G" />
>>
>>
>> where 12cores_24G is defined with a parallel SGE environment.
>>
>> Cheers,
>> Bjoern
>>
>> Am 12.03.2014 16:32, schrieb Geert Vandeweyer:
>>>
>>> Hi,
>>>
>>> Is there documentation for the proper setup and utilisation of the
>>> \${GALAXY_SLOTS:-4} style options?
>>>
>>> I noticed that tools with this setting, including BWA and GATK run
>>> single threaded, with the following settings respectively:
>>>
>>> BWA  \${GALAXY_SLOTS:-4}"
>>> ===
>>> ps output: python
>>>
>>> /galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/b4427dbb6ced/bwa_wrappers/bwa_wrapper.py
>>> --threads=1 --fileSource=indexed
>>> --ref=/galaxy/galaxy_references/hg19/bwa-0.5.9/base/hg19.fasta
>>> --do_not_build_index
>>> --input1=/galaxy/galaxy-dist/database/files/093/dataset_93021.dat
>>> --input2=/galaxy/galaxy-dist/database/files/093/dataset_93023.dat
>>> --output=/galaxy/galaxy-dist/database/files/000/112/dataset_112301.dat
>>> --genAlignType=paired --params=pre_set --suppressHeader=false
>>>
>>> xml config: --threads="\${GALAXY_SLOTS:-4}"
>>>
>>>
>>> GATK Base Recalibrator \${GALAXY_SLOTS:-8}
>>> ==
>>> python
>>>
>>> /galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/gatk2/8bcc13094767/gatk2/gatk2_wrapper.py
>>> --max_jvm_heap 6g --stdout
>>> /galaxy/galaxy-dist/database/files/000/112/dataset_112292.dat -d -I
>>> /galaxy/galaxy-dist/database/files/000/112/dataset_112289.dat bam
>>> gatk_input -d
>>> /galaxy/galaxy-dist/database/files/_metadata_files/012/metadata_12686.dat
>>> bam_index
>>> gatk_input -p   java -jar
>>> "$GATK2_PATH/GenomeAnalysisTK.jar"-T "BaseRecalibrator"
>>> $GATK2_SITE_OPTIONS  --num_cpu_threads_per_data_thread
>>> ${GALAXY_SLOTS:-8}  --no_standard_covs  -R
>>> "/galaxy/galaxy_references/hg19/srma-0.1.15/hg19.fasta" --out
>>> "/galaxy/galaxy-dist/database/files/000/112/dataset_112291.dat" -cov
>>> "ContextCovariate" -cov "CycleCovariate" -d
>>> --knownSites:dbsnp,%(file_type)s
>>> /galaxy/galaxy-dist/database/files/000/101/dataset_101086.dat vcf
>>> input_dbsnp_0 -p --pedigreeValidationType "STRICT" -d --intervals
>>> /galaxy/galaxy-dist/database/files/000/100/dataset_100129.dat bed
>>> input_intervals_0 -p --interval_set_rule "UNION" -p --downsampling_type
>>> "NONE" -p  --baq "OFF" --baqGapOpenPenalty
>>> "40.0"  --defaultBaseQualities "-1"
>>> --validation_strictness "STRICT" --interval_merging "ALL"
>>>
>>> java -Xmx6g -jar
>>> /galaxy/galaxy-dist/tool-data/shared/jars/gatk2//GenomeAnalysisTK.jar -T
>>> BaseRecalibrator --num_cpu_threads_per_data_thread 1 --no_standard_covs
>>> -R /galaxy/galaxy_references/hg19/srma-0.1.15/hg19.fasta --out
>>> /galaxy/galaxy-dist/database/files/000/112/dataset_112291.dat -cov
>>> ContextCovariate -cov CycleCovariate --pedigreeValidationType STRICT
>>> --interval_set_rule UNION --downsampling_type NONE --baq OFF
>>> --baqGapOpenPenalty 40.0 --defaultBaseQualities -1
>>> --validation_strictness STRICT --interval_merging ALL -I
>>> /tmp/tmp-gatk-fIuncH/gatk_input.bam --knownSites:dbsnp,vcf
>>> /tmp/tmp-gatk-fIuncH/input_dbsnp_0.vcf --intervals
>>> /tmp/tmp-gatk-fIuncH/input_intervals_0.bed
>>>
>>>
>>> Best,
>>>
>>> Geert
>>>
>>>
>>>
>>>
>
>
> --
>
> Geert Vandeweyer, Ph.D.
> Department of Medical Genetics
> University of Antwerp
> Prins Boudewijnlaan 43
> 2650 Edegem
> Belgium
> Tel: +32 (0)3 275 97 56
> E-mail: geert.vandewe...@ua.ac.be
> http://ua.ac.be/cognitivegenetics
> http://w

Re: [galaxy-dev] Write permission for database/files

I did not implement this - so this is just an educated guess but I
believe the relevant line of code is this:

https://bitbucket.org/galaxy/galaxy-central/src/3fb927653301a0c06a0bf94f2b6bd71b3595ec0d/lib/galaxy/objectstore/__init__.py?at=default#cl-295

Hopefully someone will correct me if I am wrong.

-John


On Wed, Mar 12, 2014 at 1:39 PM, Alper Kucukural  wrote:
> Hi All,
> Galaxy opens a directory under database/files when the #of data files 
> increased. So, it goes 1,2,3 ... etc.
> I want to give a write permission for these specific directories when they 
> are created without changing umask in the user profile.
> Is there anybody who can tell me where galaxy creates these directories.
>
> Thanks in advance.
> Best,
>
> Alper
>
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Empty history pane

2014-03-14 Thread Liisa Koski

Thanks Dannon, 

I set java to open the console (I assume that's what you are referring to) 
but when going to galaxy it doesn't open. The console does open when i go 
to java.com and verify my version.  i've also just removed and reinstalled 
java and rebooted machine.  still no history pane :(






From:   Dannon Baker 
To: Liisa Koski 
Cc: Galaxy Dev 
Date:   13/03/2014 06:39 PM
Subject:Re: [galaxy-dev] Empty history pane



If it's possible, can you check (or ask the user to check) if there are 
any javascript errors if you open the browser console when experiencing 
this failure?


On Thu, Mar 13, 2014 at 2:07 PM, Liisa Koski  wrote:
Hello, 
Our site maintains a local Galaxy installation (Nov.4th distributuion). 
One of our usrers has lost the ability to view anything in his history 
pane. He is able to run tools and view the list of 'Saved Histories' in 
the middle pane but the History pane itself stays empty. When I 
impersonate him I can see his history pane. Very weird. Any insight would 
be much appreciated. 

Thanks, 
Liisa
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets

On Fri, Mar 14, 2014 at 11:24 AM, Peter Cock  wrote:
> On Thu, Mar 13, 2014 at 6:40 PM, Sanka, Ravi  wrote:
>> I do not think so. Several individual datasets have been deleted (clicked
>> the upper-right X on the history item box) but no History has been
>> permanently deleted.
>>
>> Is there any indication in the database if target dataset or datasets were
>> marked for permanent deletion? In the dataset table, I see fields
>> "deleted", "purged", and "purgable", but nothing that says permanently
>> deleted.
>
> I would welcome clarification from the Galaxy Team, here and
> on the wiki page which might benefit from a flow diagram?
>
> https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20and%20Datasets
>
> My assumption is using "permanently delete" in the user interface
> marks an entry as "purgable", and then it will be moved to "purged"
> (and the associated file on disk deleted) by the cleanup scripts -
> but I'm a bit hazy on this any why it takes a while for a user's
> usage figures to change.

Hmm. Right now I've unable (via the web interface) to permanently
delete a history - it stays stuck as "deleted", and thus (presumably)
won't get purged by the clean up scripts.

I've tried:

1. Load problem history
2. Rename the history "DIE DIE" to avoid confusion
3. Top right menu, "Delete permanently"
4. Prompted "Really delete the current history permanently? This
cannot be undone", OK
5. Told "History deleted, a new history is active"
6. Top right menu, "Saved Histories"
7. Click "Advanced Search", status "all"
8. Observe "DIE DIE" history is only "deleted" (while other older
histories are "deleted permanently") (BAD)
9. Run the cleanup scripts,

$ sh scripts/cleanup_datasets/delete_userless_histories.sh
$ sh scripts/cleanup_datasets/purge_histories.sh
$ sh scripts/cleanup_datasets/purge_libraries.sh
$ sh scripts/cleanup_datasets/purge_folders.sh
$ sh scripts/cleanup_datasets/purge_datasets.sh

10. Reload the saved history list, no change.
11. Using the drop down menu, select "Delete Permanently"
12. Prompted "History contents will be removed from disk, this cannot
be undone.  Continue", OK
13. No change to history status (BAD)
14. Tick the check-box, and use the "Delete Permanently" button at the
bottom of the page
15. Prompted "History contents will be removed from disk, this cannot
be undone.  Continue", OK
16. No change to history status (BAD)
17. Run the cleanup scripts, no change.

Note that in my universe_wsgi.ini I have not (yet) set:
allow_user_dataset_purge = True

If this setting is important, then the interface seems confused -
and if quotas are enforced, very frustrating :(

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Problem with and conditional inputs?

Grepping around the code I think this is the only way a ftype
attribute on an output affects the evaluation of test data.

if attributes.get( 'ftype', None ) == 'bam':
local_fh, temp_name = self._bam_to_sam(
local_name, temp_name )
local_name = local_fh.name

I am not sure it was ever meant as a strict test.

I worry about breaking backward compatibility but it is easy enough to
implement this as an actual check when using newer API driven tests. I
have opened a pull request for this functionality here:

https://bitbucket.org/galaxy/galaxy-central/pull-request/347/check-ftype-attribute-if-defined-on-test/diff

Thoughts?

-John

On Tue, Mar 11, 2014 at 10:30 AM, Peter Cock  wrote:
> On Tue, Mar 11, 2014 at 2:58 PM, Björn Grüning
>  wrote:
>> Hi Peter,
>>
>> I think you need to have:
>>
>> 
>>
>>
>>
>>
>>
>>
>>
>> 
>>
>>
>> Sorry, can't test it right now.
>
> That seems to work - thanks Björn :)
>
> This seems to have exposed a bug in the  framework, e.g.
>
> 
> 
> 
>  ftype="fasta" />
> 
> 
> 
> 
>  file="blastn_rhodopsin_vs_three_human.xml" ftype="blastxml" />
> 
>
> This was saying the output should have been "blastxml",
> but until I just fixed it the output was being tagged as
> "tabular" (although run_functional_tests.sh did check
> the content it didn't check the datatype matched).
>
> Dave - do think this is a reasonable enhancement?
>
> Peter
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Verifying test output datatypes, was: Problem with and conditional inputs?

Thanks John,

I suggest making this test framework perform this check by default
(the twill and API based frameworks) and seeing what - if anything -
breaks as a result on the Test Tool Shed.

Note that one area of fuzziness is subclasses, e.g. if the tool output
was labelled "fastqsanger", but the  just said "fastq", I would
say the test was broken. On the other hand, if the  used a
specific datatype like "fastqsanger" but the tool produced a dataset
tagged with a more generic datatype like "fastq" I think that is a
again a real failure.

Peter

On Fri, Mar 14, 2014 at 3:15 PM, John Chilton  wrote:
> Grepping around the code I think this is the only way a ftype
> attribute on an output affects the evaluation of test data.
>
> if attributes.get( 'ftype', None ) == 'bam':
> local_fh, temp_name = self._bam_to_sam(
> local_name, temp_name )
> local_name = local_fh.name
>
> I am not sure it was ever meant as a strict test.
>
> I worry about breaking backward compatibility but it is easy enough to
> implement this as an actual check when using newer API driven tests. I
> have opened a pull request for this functionality here:
>
> https://bitbucket.org/galaxy/galaxy-central/pull-request/347/check-ftype-attribute-if-defined-on-test/diff
>
> Thoughts?
>
> -John
>
> On Tue, Mar 11, 2014 at 10:30 AM, Peter Cock  
> wrote:
>>
>> That seems to work - thanks Björn :)
>>
>> This seems to have exposed a bug in the  framework, e.g.
>>
>> 
>> 
>> 
>> > ftype="fasta" />
>> 
>> 
>> 
>> 
>> > file="blastn_rhodopsin_vs_three_human.xml" ftype="blastxml" />
>> 
>>
>> This was saying the output should have been "blastxml",
>> but until I just fixed it the output was being tagged as
>> "tabular" (although run_functional_tests.sh did check
>> the content it didn't check the datatype matched).
>>
>> Dave - do think this is a reasonable enhancement?
>>
>> Peter

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Dataset's extra files

The directory can be obtained using $galaxyData.extra_files_path (be
sure to check it exists before zipping it up).

I would discourage re-using Galaxy components directly from inside of
a tool or tool wrapper - but if you want to reference that code it is
actually inside of the datatypes module -

https://bitbucket.org/galaxy/galaxy-central/src/3fb927653301a0c06a0bf94f2b6bd71b3595ec0d/lib/galaxy/datatypes/data.py?at=default#cl-228

Hope this helps.

-John

On Tue, Mar 11, 2014 at 5:35 AM, Charles Girardot
 wrote:
> Hi all,
>
> We have a local tool which role is to transfer (ie copy) a dataset file to a 
> directory on our NFS. This is extremely convenient as it can be included 
> within workflows and therefore save the time of clicking download button (we 
> also have configurable renaming/compression as part of it). It is heavily 
> used by our users.
>
> The problem is with datasets that have associated files like FASTQC as these 
> extra files are simple not ignored...
> We'd like to improve our 'NFS_transfer' tool so it can deal with this in a 
> similar fashion as the download button.
>
> Foreseen solution :
> * Check if a directory named 'dataset__files' exists within the dataset 
> store
> * if so, 'cp -r' it into a tmp dir, cp the dataset itself into same tmp dir 
> (with renaming on the fly)
> * zip/tar.gz the tmp dir
> * copy it to final NFS location
>
> Question is : is this the right way to do it ?  As a non python specialist, 
> it is a little tricky to find the right way to it (I can t locate the piece 
> of code that does this in galaxy ie behind the download button). In 
> particular, can I get the list of extra files using the '$galaxyFile' object 
> given in the tool by :
>
> 
>
> i.e. in the same way we get the dataset name or file extension 
> ($galaxyFile.dataset.name and $galaxyFile.ext) ?
>
> Any advise on how best to implement this, in a portable way, very appreciated.
>
> Thanks for your time,
>
> Charles
>
> =
> Charles Girardot
> Head of Genome Biology Computational Support (GBCS)
> European Molecular Biology Laboratory
> Tel: +49 6221 387 -8585
> Fax: +49-(0)6221-387-8166
> Email: charles.girar...@embl.de
> Room V205
> Meyerhofstraße 1,
> 69117 Heidelberg, Germany
> =
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] FTP password and web interface password

Hi Yec'han,

Sorry for the delayed response - I would guess that something is
different about the configuration of the SFTP server. Could you check
the server logs?

--nate

On Thu, Mar 6, 2014 at 8:13 AM, Yec'han Laizet
 wrote:
> Hi Nate,
>
> I have reseted the password of a newly created user, so now, it does not
> begin with $PBKDF2$. With the reseted password, I can access the web
> interface but I can not connect by SFTP.
>
> Here is the log of filezilla:
>
> Statut :Connexion à galaxy-pgtp.pierroton.inra.fr...
> Suivi :Going to execute /usr/bin/fzsftp
> Réponse :fzSftp started
> Suivi :CSftpControlSocket::ConnectParseResponse(fzSftp started)
> Suivi :CSftpControlSocket::SendNextCommand()
> Suivi :CSftpControlSocket::ConnectSend()
> Commande :open "new_u...@mydomain.com@galaxy-pgtp.pierroton.inra.fr" 22
> Suivi :Server version: SSH-2.0-mod_sftp/0.9.8
> Suivi :Using SSH protocol version 2
> Suivi :We claim version: SSH-2.0-PuTTY_Local:_Sep_14_2013_01:12:43
> Suivi :Doing Diffie-Hellman group exchange
> Suivi :Doing Diffie-Hellman key exchange with hash SHA-256
> Suivi :Host key fingerprint is:
> Suivi :ssh-rsa ***
> Suivi :Initialised AES-256 SDCTR client->server encryption
> Suivi :Initialised HMAC-SHA1 client->server MAC algorithm
> Suivi :Initialised AES-256 SDCTR server->client encryption
> Suivi :Initialised HMAC-SHA1 server->client MAC algorithm
> Suivi :Pageant is running. Requesting keys.
> Suivi :Pageant has 1 SSH-2 keys
> Commande :Pass: **
> Suivi :Sent password
> Suivi :Access denied
> Erreur :Échec de l'authentification.
> Suivi :CControlSocket::DoClose(1030)
> Suivi :CSftpControlSocket::ResetOperation(1094)
> Suivi :CControlSocket::ResetOperation(1094)
> Erreur :Erreur critique
> Erreur :Impossible d'établir une connexion au serveur
> Suivi :CFileZillaEnginePrivate::ResetOperation(1094)
>
>
>
> If I use my own account which has been created a long time ago (understand
> here that some updates of galaxy have been done since this time...), the
> password is not PBKDF2$ encrypted and I can access both the web interface
> and the sftp. The filezilla log here is similar to the one shown above but
> of course, I get an "Access granted" instead of "denied".
>
> I don't understand why old accounts can connect whereas new ones cannot
> although passwords are not PBKDF2$.
>
>
> Yec'han
>
>
> 
>
> Dr. Yec'han LAIZET
> Ingenieur Bioinformatique
> Tel: +33 (0)5 57 12 27 75
> _
>
> INRA-UMR BIOGECO 1202
> Equipe Genetique
> 69 route d'Arcachon
> 33612 CESTAS
> 
>
> Le 05/03/2014 20:44, Nate Coraor a écrit :
>
>> Hi Yec'han,
>>
>> Could you check that the 'password' column for the user in question in
>> the 'galaxy_user' table in the database does not begin with $PBKDF2$?
>>
>> If not, do you have any debug logs from the FTP session and server
>> that provide details on the failure?
>>
>> --nate
>>
>> On Wed, Mar 5, 2014 at 10:36 AM, Yec'han Laizet
>>  wrote:
>>>
>>> Hello,
>>>
>>> does anybody have any idea of what I can do to fix the problem?
>>>
>>> Maybe an update is required? I currently use the changeset:
>>> 11219:5c789ab4144a
>>>
>>> thanks
>>>
>>>
>>> Yec'han
>>>
>>>
>>> 
>>>
>>> Dr. Yec'han LAIZET
>>> Ingenieur Bioinformatique
>>> Tel: +33 (0)5 57 12 27 75
>>> _
>>>
>>> INRA-UMR BIOGECO 1202
>>> Equipe Genetique
>>> 69 route d'Arcachon
>>> 33612 CESTAS
>>> 
>>>
>>> Le 18/02/2014 08:39, Yec'han Laizet a écrit :
>>>
 Hi Bjoern,

 I indeed followed the wiki tutorial to set up my FTP service some time
 ago. It seems, as you suggest, that newly created users cannot connect
 by
 SFTP.
 I followed the fix by putting the use_pbkdf2 = False line just below the
 [app:main] and restarted the galaxy server. I have reseted a newly
 created
 user password but it still does not work.

 Yec'han


 

 Dr. Yec'han LAIZET
 Ingenieur Bioinformatique
 Tel: +33 (0)5 57 12 27 75
 _

 INRA-UMR BIOGECO 1202
 Equipe Genetique
 69 route d'Arcachon
 33612 CESTAS
 

 Le 17/02/2014 18:12, Björn Grüning a écrit :
>
> Hi Yec'han,
>
> please have a look at
>
> https://wiki.galaxyproject.org/Admin/Config/Upload%20via%20FTP
>
> If you are running postgres and you only newly created users can't
> access
> the server its probably due to encryption changes. Set use_pbkdf2 =
> False
> and reset all passwort for new users.
>
>

Re: [galaxy-dev] Job never starts

Hmm... unfortunately I have no particular guess about either of these issues.

It sounds like you upgraded Galaxy and made large changes to your
configuration at the same time. Can you try just one process and no
load balancer so you can determine if the issue is with the Apache
configuration or the latest version of Galaxy. If that works - can you
just do one web process and one handler and see if the problems
persist? The goal would be to scale up, but still figure out at what
point the problems start.

-John

On Tue, Mar 11, 2014 at 1:30 PM, Alper Kucukural  wrote:
> Hi All,
> I have installed the recent version of galaxy and starting multiple web and
> job handlers(six each) on a Centos 5.1 machine.
> It is working almost perfectly.
> 1. The first problem is sometimes jobs never start and they are in grey
> stage. After I killed them and start again, they work fine. There are no any
> log about why they haven't started. So, if you direct me to find the reason,
> it would be great.
>
>
> 2. Second problem is about UCSC visualization issue using the links from
> history. I haven't found a solution yet, my XSendFile is all set and working
> in the previous version but not this one.
> The only difference is I am using the new version and balancer in Apache
> since I run multiple instances. So, balancer might not use XSendFile
> properly or there is a configuration problem.
>
> http://lists.bx.psu.edu/pipermail/galaxy-user/2012-November/005508.html
> https://wiki.galaxyproject.org/Admin/Config/ApacheProxy
>
>
> Let me know, if anybody has a similar settings and solved this problem with
> Apache.
> Best,
>
> Alper Kucukural, PhD
> Bioinformatics Core,
> University of Massachusetts Medical School
> 368 Plantation St.Room AS4.2067
> Worcester, MA 01605-2324
> Phone: 774-312-4493
> E-mail: al...@kucukural.com
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Persistent jobs in cluster queue even after canceling job in galaxy

I believe this problem was fixed by Nate after the latest dist release
and pushed to the stable branch of galaxy-central.

https://bitbucket.org/galaxy/galaxy-central/commits/1298d3f6aca59825d0eb3d32afd5686c4b1b9294

If you are eager for this bug fix, you can track the latest stable
branch of galaxy-central instead of the galaxy-dist tag mentioned in
the dev news. Right now it has some other good bug fixes not in the
latest release.

-John

On Tue, Mar 11, 2014 at 9:53 PM, Brian Claywell  wrote:
> We've had the same problem since updating from the April 2013 stable
> to Feb 2014 stable. Our jobs are going off to a slurm cluster via the
> SlurmJobRunner plugin (though this was happening with the
> DRMAAJobRunner plugin too, if I remember right). Removing pending
> datasets occasionally removes the entry in Admin/Jobs, but not
> reliably, and regardless the jobs stay queued in slurm with no noise
> in galaxy.log at DEBUG or higher. We're not using parallelism either.
>
> I noticed this around the same time as I noticed the new in-progress
> animation in the history pane; perhaps there's an ajax-y callback
> that's not firing? Otherwise I'd expect to see something in the log
> about Galaxy at least trying to cancel the job.
>
>
> On Tue, Mar 11, 2014 at 10:57 AM, Ravi Alla  wrote:
>> Hi All,
>> I've been able to submit jobs to the cluster through galaxy, it works great. 
>> But when the job is in queue to run (it is gray in the galaxy history pane) 
>> and I cancel the job, it still remains in queue on the cluster. Why does 
>> this happen? How can I delete the jobs in queue as well? I tried qdel 
>>  as galaxy user but it says I am not authenticated to delete the job.
>> Any help would be greatly appreciated.
>> Thanks
>> Ravi.
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>
>
>
> --
> Brian Claywell, Systems Analyst/Programmer
> Fred Hutchinson Cancer Research Center
> bclay...@fhcrc.org
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] How to install bowtie2 tool in galaxy

On Wed, Mar 5, 2014 at 8:18 PM, Ravi Alla  wrote:
> Hi Jennifer,
> Thank you for this information. I was able to troubleshoot the
> bowtie2_indices. Like Bjoern said the bowtie2 tool needed to be reinstalled
> because the tool_table does not load the correct indices.
>
> I am still trying to wrap my head around the different toolsheds.
>
> The main tool shed resides in the galaxy-dist folder and uses tool_conf.xml
> to load tools into the side bar and tool_data_table_conf.xml to load indices
> (and other data) for the default tools that come with galaxy.

Hi Ravi,

The tools in the galaxy-dist directory and controlled via
tool_conf.xml are not a part of the tool shed. The inclusion of tools
directly in Galaxy predates the existence of the tool shed. Other than
that, what you have above is correct.

> The shed tools reside in the ../shed_tools/ directory and use
> shed_tools_conf.xml to load tools into the side bar and the
> shed_tools_data_table_conf.xml to load indices.
>
> The shed_tool xml is setup in the universe.ini file.

Right.

> The .loc files for default main tools reside in galaxy-dist/tool-data and
> the .loc files for the shed_tools reside in
> ../shed_tools/...

By default they'll all be under galaxy-dist/tool-data/.  The versions
in ../shed_tools are the sample location files provided by tool
authors.

> And when I uninstall tools and reinstall I would have to update the metadata
> for that tool.

I am not sure what you mean by this.

> If ../shed_tools dir is not present then shed-tools get installed under
> galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/

../shed_tools should be created if it does not exist. Tools won't be
installed in galaxy-dist/tool-data/

> And it is always better to install tools as wrappers + dependencies when
> possible.

I agree.

--nate

>
> Am I on the right track with this tool organization?
> Thanks
> Ravi
>
>
>
> On Mar 5, 2014, at 11:06 AM, Jennifer Jackson  wrote:
>
> Hi Ravi,
>
> The directory structure for the installation of ToolShed tools changed,
> which is why you have three directories. You perhaps had bowtie2 installed
> once before, then reinstalled (without completely removing the older version
> and associated data)? Or updated without resetting the metadata? In either
> case, the ../shed_tools directory is the new one. Having this as the path in
> your .xml configuration files (as Bjoern suggested earlier) and moving all
> contents & data to be under the same location will be the simplest global
> solution ongoing. Links near the end of my reply can help explain how-to.
>
> For the specific reason why bowtie2 indices are not working: I noticed that
> the reference genome ".fa" file is not linked from the directory containing
> the indexes. This is required. Adding it in, the same way that you did for
> the bowtie2 indexes, into this dir:
> "/global/referenceData/databases/bowtie2/hg19" will probably solve that part
> of the problem. I didn't see this posted - but I apologize if I am
> duplicating advice already given.
>
> I also tend to advise keeping all data under the same master "data"
> directory - all indexes and sequence data - as symbolic links to additional
> file system paths that are unknown to the 'galaxy user' cause a different
> set of problems. However, that said, this doesn't seem to be an issue in
> your specific case: if the bowtie2 indexes are functioning correctly - then
> environment is set up so that the other dir hierarchy where the .fa files
> are kept must included in the 'galaxy' user's ENV. Symbolic links that go
> outside of the local dir structure are known to cause problems unless the
> ENV config is carefully set up, and to my knowledge are best avoided
> entirely for certain uses such as "upload by file path" into libraries.
>
> For reference:
>
> NGS data set-up is described in this wiki - including expected content for
> each type of index:
> https://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup
>
> For examples, you can rsync a genome or two and examine the contents. Or,
> our /location dir and have a look at the .loc files.
> https://wiki.galaxyproject.org/Admin/DataIntegration
>
> Tool Shed help (very detailed):
> https://wiki.galaxyproject.org/Tool%20Shed
> In particular, if you had previously installed repositories (this is not
> clear, just suspected from the duplications), updating the Metadata with
> certain distribution updates can be very important. This has been necessary
> for the last few releases to update to changes. The News Brief noted this,
> and included a link to this wiki page. Also see the "Related Pages" lower
> down on the wiki.
> https://wiki.galaxyproject.org/ResettingMetadataForInstalledRepositories
> This may also be useful:
> https://wiki.galaxyproject.org/RepairingInstalledRepositories
>
> Hopefully you have sorted most of this out by now, or this helps!
>
> Jen
> Galaxy team
>
>
> On 3/4/14 12:21 PM, Ravi Alla wrote:
>
> Bjoern,
> This is getting frustrating.
> There are thr

Re: [galaxy-dev] Tool Shed install best practise : Precompiled binaries vs local compile

Hi Peter,

I just wanted to add my $0.02 USD to say that I mostly agree with this
- I have long used binaries precompiled by the tool author on Main,
especially for cases where, as you say, the compile-time dependency
list is large and painful. The only "gotcha" here is to make sure that
binaries support the oldest possible version of glibc that might be
installed on any of the Linux distributions and versions that we
support, and that no non-standard preinstalled libraries have been
pulled in during the build process.

--nate

On Thu, Mar 6, 2014 at 12:46 PM, Peter Cock  wrote:
> Hi Greg,
>
> I've retitled the thread, previously about a ToolShed nightly test
> failure.
>
> A brief recap, we're talking about the Galaxy ToolShed XML
> installation recipes for the NCBI BLAST+ packages and my
> MIRA4 wrapper in their tool_dependencies.xml files:
>
> http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_29
> http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_29
> http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler
>
> These use the pattern of having os/arch specific  tags
> (which download and install the tool author's precompiled
> binaries) and a fall back default  which is to report
> an error with the os/arch combination and that there are no
> ready made binaries available.
>
> Greg is instead advocating the fall back action be to download
> the source code, and do a local compile.
>
> My reply is below...
>
> On Thu, Mar 6, 2014 at 5:24 PM, Peter Cock  wrote:
>> On Thu, Mar 6, 2014 at 4:53 PM, Greg Von Kuster  wrote:
>>>
>>> As we briefly discussed earlier, your mira4 recipe is not currently
>>> following best practices.  Although you uncovered a problem in
>>> the framework which has now been corrected, your recipe's fall
>>> back  tag set should be the recipe for installing mira4
>>> from source ( http://sourceforge.net/projects/mira-assembler/ )
>>> since there is no licensing issues for doing so.  This would be a
>>> more ideal approach than echoing the error messages.
>>>
>>> Thanks very much for helping us discover this problem though!
>>>
>>> Greg Von Kuster
>>
>> Hi Greg,
>>
>> No problem - I'm good at discovering problems ;)
>>
>> If the download approach failed, it it most likely due to a
>> transient error (e.g. network issues with download). Here I
>> would much prefer Galaxy aborted and reported this as an
>> error (and does not attempt the default action). Is that what
>> you just fixed?
>>
>> As to best practice for the fall back action, I think that needs
>> a new thread.
>>
>> Regards,
>>
>> Peter
>
> As to best practice, I do not agree that in cases like this
> (MIRA4, NCBI BLAST+) where there are provided binaries
> for the major platforms that the fall back should be compiling
> from source.
>
> The NCBI BLAST+ provide binaries for 32 bit and 64 bit
> Linux and Mac OS X (which I believe covers all the
> mainstream platforms Galaxy runs on).
>
> Similarly, MIRA4 provides binaries for 64 bit Linux and
> Mac OS X. Note that 32 bit binaries are not provided,
> but would be very restricted in terms of the datasets
> they could be used on anyway - and I doubt many of
> the systems Galaxy runs on these days are 32 bits.
>
> If the os/arch combination is exotic enough that precompiled
> binaries are not available, then it is likely compilation will be
> tricky anyway - or not supported for that tool, or Galaxy itself.
>
> Essentially I am arguing that where the precompiled tool
> binaries cover any mainstream system Galaxy might
> be used on, a local compile fall back is not needed.
>
> Also, these are both complex tools which are relatively slow
> to compile, and have quite a large compile time dependency
> set (e.g. MIRA4 requires at least a quite recent GCC, BOOST,
> flex, expat, and strongly recommends TCmalloc). Here at
> least some of the dependencies have been packaged for
> the ToolShed (probably by Bjoern?) but in the case of
> MIRA4 and BLAST+ this is still a lot of effort for no
> practical gain.
>
> I also feel there is an argument that the Galaxy goal of
> reproducibility should favour using precompiled binaries if
> available: A locally compiled binary will generally mean a
> different compiler version, perhaps with different optimisation
> flags, and different library versions. It will not necessarily
> give the same results as the tool author's provided
> precompiled binary.
>
> (Wow, this ended up being a long email!)
>
> Regards,
>
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail cli

Re: [galaxy-dev] Tool Shed install best practise : Precompiled binaries vs local compile

On Fri, Mar 14, 2014 at 4:41 PM, Nate Coraor  wrote:
> Hi Peter,
>
> I just wanted to add my $0.02 USD to say that I mostly agree with this
> - I have long used binaries precompiled by the tool author on Main,
> especially for cases where, as you say, the compile-time dependency
> list is large and painful. The only "gotcha" here is to make sure that
> binaries support the oldest possible version of glibc that might be
> installed on any of the Linux distributions and versions that we
> support, and that no non-standard preinstalled libraries have been
> pulled in during the build process.
>
> --nate

Good point about potential problems with glibc, and I guess any run
time linked libraries which might be a different version or missing?

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tool Shed install best practise : Precompiled binaries vs local compile

On Fri, Mar 14, 2014 at 12:47 PM, Peter Cock  wrote:
> On Fri, Mar 14, 2014 at 4:41 PM, Nate Coraor  wrote:
>> Hi Peter,
>>
>> I just wanted to add my $0.02 USD to say that I mostly agree with this
>> - I have long used binaries precompiled by the tool author on Main,
>> especially for cases where, as you say, the compile-time dependency
>> list is large and painful. The only "gotcha" here is to make sure that
>> binaries support the oldest possible version of glibc that might be
>> installed on any of the Linux distributions and versions that we
>> support, and that no non-standard preinstalled libraries have been
>> pulled in during the build process.
>>
>> --nate
>
> Good point about potential problems with glibc, and I guess any run
> time linked libraries which might be a different version or missing?
>
> Peter

Right, many things (especially using autoconf) will automatically link
against libraries if present and disable functionality if not.
Hopefully if we were returning to mostly vanilla tool test VMs in the
future, any problems would be easily discoverable.

--nate
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] How to install bowtie2 tool in galaxy

2014-03-14 Thread Ravi Alla

Hi Nate 
Thanks for the clarifications. Is there anything specific I should watch out 
for when updating and deleting tools? The reason I ask is that Jennifer 
mentioned in previous emails the need to update metadata for tools, which I am 
sure I follow.
Thanks
Ravi
On Mar 14, 2014, at 9:02 AM, Nate Coraor  wrote:

> On Wed, Mar 5, 2014 at 8:18 PM, Ravi Alla  wrote:
>> Hi Jennifer,
>> Thank you for this information. I was able to troubleshoot the
>> bowtie2_indices. Like Bjoern said the bowtie2 tool needed to be reinstalled
>> because the tool_table does not load the correct indices.
>> 
>> I am still trying to wrap my head around the different toolsheds.
>> 
>> The main tool shed resides in the galaxy-dist folder and uses tool_conf.xml
>> to load tools into the side bar and tool_data_table_conf.xml to load indices
>> (and other data) for the default tools that come with galaxy.
> 
> Hi Ravi,
> 
> The tools in the galaxy-dist directory and controlled via
> tool_conf.xml are not a part of the tool shed. The inclusion of tools
> directly in Galaxy predates the existence of the tool shed. Other than
> that, what you have above is correct.
> 
>> The shed tools reside in the ../shed_tools/ directory and use
>> shed_tools_conf.xml to load tools into the side bar and the
>> shed_tools_data_table_conf.xml to load indices.
>> 
>> The shed_tool xml is setup in the universe.ini file.
> 
> Right.
> 
>> The .loc files for default main tools reside in galaxy-dist/tool-data and
>> the .loc files for the shed_tools reside in
>> ../shed_tools/...
> 
> By default they'll all be under galaxy-dist/tool-data/.  The versions
> in ../shed_tools are the sample location files provided by tool
> authors.
> 
>> And when I uninstall tools and reinstall I would have to update the metadata
>> for that tool.
> 
> I am not sure what you mean by this.
> 
>> If ../shed_tools dir is not present then shed-tools get installed under
>> galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/
> 
> ../shed_tools should be created if it does not exist. Tools won't be
> installed in galaxy-dist/tool-data/
> 
>> And it is always better to install tools as wrappers + dependencies when
>> possible.
> 
> I agree.
> 
> --nate
> 
>> 
>> Am I on the right track with this tool organization?
>> Thanks
>> Ravi
>> 
>> 
>> 
>> On Mar 5, 2014, at 11:06 AM, Jennifer Jackson  wrote:
>> 
>> Hi Ravi,
>> 
>> The directory structure for the installation of ToolShed tools changed,
>> which is why you have three directories. You perhaps had bowtie2 installed
>> once before, then reinstalled (without completely removing the older version
>> and associated data)? Or updated without resetting the metadata? In either
>> case, the ../shed_tools directory is the new one. Having this as the path in
>> your .xml configuration files (as Bjoern suggested earlier) and moving all
>> contents & data to be under the same location will be the simplest global
>> solution ongoing. Links near the end of my reply can help explain how-to.
>> 
>> For the specific reason why bowtie2 indices are not working: I noticed that
>> the reference genome ".fa" file is not linked from the directory containing
>> the indexes. This is required. Adding it in, the same way that you did for
>> the bowtie2 indexes, into this dir:
>> "/global/referenceData/databases/bowtie2/hg19" will probably solve that part
>> of the problem. I didn't see this posted - but I apologize if I am
>> duplicating advice already given.
>> 
>> I also tend to advise keeping all data under the same master "data"
>> directory - all indexes and sequence data - as symbolic links to additional
>> file system paths that are unknown to the 'galaxy user' cause a different
>> set of problems. However, that said, this doesn't seem to be an issue in
>> your specific case: if the bowtie2 indexes are functioning correctly - then
>> environment is set up so that the other dir hierarchy where the .fa files
>> are kept must included in the 'galaxy' user's ENV. Symbolic links that go
>> outside of the local dir structure are known to cause problems unless the
>> ENV config is carefully set up, and to my knowledge are best avoided
>> entirely for certain uses such as "upload by file path" into libraries.
>> 
>> For reference:
>> 
>> NGS data set-up is described in this wiki - including expected content for
>> each type of index:
>> https://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup
>> 
>> For examples, you can rsync a genome or two and examine the contents. Or,
>> our /location dir and have a look at the .loc files.
>> https://wiki.galaxyproject.org/Admin/DataIntegration
>> 
>> Tool Shed help (very detailed):
>> https://wiki.galaxyproject.org/Tool%20Shed
>> In particular, if you had previously installed repositories (this is not
>> clear, just suspected from the duplications), updating the Metadata with
>> certain distribution updates can be very important. This has been necessary
>> for the last few releases to update to changes. The News Brie

Re: [galaxy-dev] Empty history pane

2014-03-14 Thread Liisa Koski

Our use was using the lastest Firefox release. I asked him to try FireFox 
Portable and that seemed to do the trick. It was a problem with his 
Firefox install.

Thanks for you help,
Liisa




From:   Dannon Baker 
To: Liisa Koski 
Cc: Galaxy Dev 
Date:   13/03/2014 06:39 PM
Subject:Re: [galaxy-dev] Empty history pane



If it's possible, can you check (or ask the user to check) if there are 
any javascript errors if you open the browser console when experiencing 
this failure?


On Thu, Mar 13, 2014 at 2:07 PM, Liisa Koski  wrote:
Hello, 
Our site maintains a local Galaxy installation (Nov.4th distributuion). 
One of our usrers has lost the ability to view anything in his history 
pane. He is able to run tools and view the list of 'Saved Histories' in 
the middle pane but the History pane itself stays empty. When I 
impersonate him I can see his history pane. Very weird. Any insight would 
be much appreciated. 

Thanks, 
Liisa
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Internal server error when importing workflows!

Hmm... I can import older workflows on usegalaxy.org so I don't think
it is a general problem with all workflows as a part of the latest
release. That said I don't know what the problem is - if it is a bug
with a new release or a problem with one of your tools etc

If I had to guess I would say you updated a tool has been updated
changing its "interface" without changing its version? Can you send
the list (or me) the workflow and any custom tool XML files it uses?
Do you still have access to the original Galaxy instance with the
workflow in it - if you could try to trim the number of steps down and
retry until it works or try a completely new workflow that might help
track down the particular circumstances surrounding the problem.

-John


On Tue, Mar 11, 2014 at 11:43 AM, Hakeem Almabrazi
 wrote:
> Hi,
>
>
>
> I have upgraded my local galaxy to the latest stable version and it works
> fine but I get an error when I try to import workflows that were created
> older version of Galaxy.  Here is the error I keep getting.
>
>
>
> Internal Server Error
>
> Galaxy was unable to successfully complete your request
>
> URL: http://10.7.10.21/workflow/import_workflow
> Module galaxy.web.framework.middleware.error:149 in __call__
>>>  app_iter = self.application(environ, sr_checker)
> Module paste.recursive:84 in __call__
>>>  return self.application(environ, start_response)
> Module paste.httpexceptions:633 in __call__
>>>  return self.application(environ, start_response)
> Module galaxy.web.framework.base:132 in __call__
>>>  return self.handle_request( environ, start_response )
> Module galaxy.web.framework.base:190 in handle_request
>>>  body = method( trans, **kwargs )
> Module galaxy.webapps.galaxy.controllers.workflow:1072 in import_workflow
>>>  workflow, missing_tool_tups = self._workflow_from_dict( trans, data,
>>> source=src, add_to_menu=add_to_menu )
> Module galaxy.web.base.controller:1639 in _workflow_from_dict
>>>  module.save_to_step( step )
> Module galaxy.workflow.modules:274 in save_to_step
>>>  step.tool_inputs = self.tool.params_to_strings( self.state.inputs,
>>> self.trans.app )
> Module galaxy.tools:2418 in params_to_strings
>>>  return params_to_strings( self.inputs, params, app )
> Module galaxy.tools.parameters:85 in params_to_strings
>>>  value = params[ key ].value_to_basic( value, app )
> Module galaxy.tools.parameters.grouping:468 in value_to_basic
>>>  rval[ input.name ] = input.value_to_basic( value[ input.name ], app )
> KeyError: 'index'
>
>
>
> Can someone let me know how to fix this issue?
>
>
>
> Thank you
>
> Hakeem
>
>
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] How To properly use GALAXY_SLOTS

2014-03-14 Thread Geert Vandeweyer


This is strange.

After restarting galaxy to include the debug tool, all galaxy_slots 
values are correctly assigned (tested for BWA and GATK as rerun of 
original posted jobs)


If you still need the output, let me know.

Best,

Geert

On 03/14/2014 03:25 PM, John Chilton wrote:

I am not sure I understand what you mean, but if you are setting the
job to run with a ppn of greater than one but GALAXY_SLOTS is being
evaluated as 1 then you have likely uncovered a bug - maybe in the
runner, your job configuration, or the GALAXY_SLOTS logic for
pbs/torque.

I have created a Galaxy tool for debugging the problem -
https://gist.github.com/jmchilton/9548516. If you could run in your
environment and assign it multiple cores it should spit out some
interesting information relevant to debugging the problem. It will
create Galaxy datasets corresponding to the runtime-computed Galaxy
slots, the contents of PBS_NODEFILE which Galaxy attempts to use to
compute Galaxy slots, and the full contents of your worker node
environment. This last one may contain sensitive information - but if
you could post the first two or send them to me directly it would
hopefully help debug the problem.

-John

On Wed, Mar 12, 2014 at 12:37 PM, Geert Vandeweyer
 wrote:

How would those statements translate to pbs/torque (pbs_python)

I request resources using the nodes=1:ppn:4 syntax.  The -pe argument is not
available it seems...

Best,

Geert


On 03/12/2014 04:43 PM, Björn Grüning wrote:

Hi Geert,

you can give every tool a different amount of SLOTS via the job_conf.xml
file.

https://wiki.galaxyproject.org/Admin/Config/Jobs

For example:

 


where 12cores_24G is defined with a parallel SGE environment.

Cheers,
Bjoern

Am 12.03.2014 16:32, schrieb Geert Vandeweyer:

Hi,

Is there documentation for the proper setup and utilisation of the
\${GALAXY_SLOTS:-4} style options?

I noticed that tools with this setting, including BWA and GATK run
single threaded, with the following settings respectively:

BWA  \${GALAXY_SLOTS:-4}"
===
ps output: python

/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/b4427dbb6ced/bwa_wrappers/bwa_wrapper.py
--threads=1 --fileSource=indexed
--ref=/galaxy/galaxy_references/hg19/bwa-0.5.9/base/hg19.fasta
--do_not_build_index
--input1=/galaxy/galaxy-dist/database/files/093/dataset_93021.dat
--input2=/galaxy/galaxy-dist/database/files/093/dataset_93023.dat
--output=/galaxy/galaxy-dist/database/files/000/112/dataset_112301.dat
--genAlignType=paired --params=pre_set --suppressHeader=false

xml config: --threads="\${GALAXY_SLOTS:-4}"


GATK Base Recalibrator \${GALAXY_SLOTS:-8}
==
python

/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/gatk2/8bcc13094767/gatk2/gatk2_wrapper.py
--max_jvm_heap 6g --stdout
/galaxy/galaxy-dist/database/files/000/112/dataset_112292.dat -d -I
/galaxy/galaxy-dist/database/files/000/112/dataset_112289.dat bam
gatk_input -d
/galaxy/galaxy-dist/database/files/_metadata_files/012/metadata_12686.dat
bam_index
gatk_input -p   java -jar
"$GATK2_PATH/GenomeAnalysisTK.jar"-T "BaseRecalibrator"
$GATK2_SITE_OPTIONS  --num_cpu_threads_per_data_thread
${GALAXY_SLOTS:-8}  --no_standard_covs  -R
"/galaxy/galaxy_references/hg19/srma-0.1.15/hg19.fasta" --out
"/galaxy/galaxy-dist/database/files/000/112/dataset_112291.dat" -cov
"ContextCovariate" -cov "CycleCovariate" -d
--knownSites:dbsnp,%(file_type)s
/galaxy/galaxy-dist/database/files/000/101/dataset_101086.dat vcf
input_dbsnp_0 -p --pedigreeValidationType "STRICT" -d --intervals
/galaxy/galaxy-dist/database/files/000/100/dataset_100129.dat bed
input_intervals_0 -p --interval_set_rule "UNION" -p --downsampling_type
"NONE" -p  --baq "OFF" --baqGapOpenPenalty
"40.0"  --defaultBaseQualities "-1"
--validation_strictness "STRICT" --interval_merging "ALL"

java -Xmx6g -jar
/galaxy/galaxy-dist/tool-data/shared/jars/gatk2//GenomeAnalysisTK.jar -T
BaseRecalibrator --num_cpu_threads_per_data_thread 1 --no_standard_covs
-R /galaxy/galaxy_references/hg19/srma-0.1.15/hg19.fasta --out
/galaxy/galaxy-dist/database/files/000/112/dataset_112291.dat -cov
ContextCovariate -cov CycleCovariate --pedigreeValidationType STRICT
--interval_set_rule UNION --downsampling_type NONE --baq OFF
--baqGapOpenPenalty 40.0 --defaultBaseQualities -1
--validation_strictness STRICT --interval_merging ALL -I
/tmp/tmp-gatk-fIuncH/gatk_input.bam --knownSites:dbsnp,vcf
/tmp/tmp-gatk-fIuncH/input_dbsnp_0.vcf --intervals
/tmp/tmp-gatk-fIuncH/input_intervals_0.bed


Best,

Geert






--

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: geert.vandewe...@ua.ac.be
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/in/geertvandeweyer

___
Please keep all replies on the list by using "reply all"
in your mail client.  To ma

Re: [galaxy-dev] Empty history pane

2014-03-14 Thread Dannon Baker

Good to hear you've worked around it!  For future reference, in Firefox you
can open the javascript console by clicking the "Firefox" (menu at the
top)->"Web Developer"->"Web Console".  That, or Control/Command+Shift+K.


On Fri, Mar 14, 2014 at 1:49 PM, Liisa Koski  wrote:

> Our use was using the lastest Firefox release. I asked him to try FireFox
> Portable and that seemed to do the trick. It was a problem with his Firefox
> install.
>
> Thanks for you help,
> Liisa
>
>
>
>
> From:Dannon Baker 
> To:Liisa Koski 
> Cc:Galaxy Dev 
> Date:13/03/2014 06:39 PM
> Subject:Re: [galaxy-dev] Empty history pane
> --
>
>
>
> If it's possible, can you check (or ask the user to check) if there are
> any javascript errors if you open the browser console when experiencing
> this failure?
>
>
> On Thu, Mar 13, 2014 at 2:07 PM, Liisa Koski 
> <*liisa.ko...@basf.com*>
> wrote:
> Hello,
> Our site maintains a local Galaxy installation (Nov.4th distributuion).
> One of our usrers has lost the ability to view anything in his history
> pane. He is able to run tools and view the list of 'Saved Histories' in the
> middle pane but the History pane itself stays empty. When I impersonate him
> I can see his history pane. Very weird. Any insight would be much
> appreciated.
>
> Thanks,
> Liisa
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   *http://lists.bx.psu.edu/* 
>
> To search Galaxy mailing lists use the unified search at:
>   
> *http://galaxyproject.org/search/mailinglists/*
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Puppet Modules: a new project

2014-03-14 Thread Olivier Inizan


Dear galaxy-dev,

After exchanges with Eric we have decided to create:
-a common puppet module for galaxy hosted on the puppet forge:
https://forge.puppetlabs.com/urgi/galaxy
-a common repository hosted on a public server:
We want to have the opinion of the galaxy-team and galaxy-dev community 
for the choice of the public server.

Should we push the code on github or bitbucket ?
Does this question matter anyway ? :)

Thanks for your answers,

Olivier


Olivier Inizan
Unité de Recherches en Génomique-Info (UR INRA 1164),
INRA, Centre de recherche de Versailles, bat.18
RD10, route de Saint Cyr
78026 Versailles Cedex, FRANCE

olivier.ini...@versailles.inra.fr

Tél: +33 1 30 83 38 25
Fax: +33 1 30 83 38 99
http://urgi.versailles.inra.fr [urgi.versailles.inra.fr]
Twitter: @OlivierInizan


On Thu, 13 Mar 2014, Olivier Inizan wrote:


Hi Eric and Sys Admin,

It's a great news !

We have already published a puppet module for a basic galaxy install and 
update:

http://forge.puppetlabs.com/urgi/galaxy

We also plan to update it for configuration files management.
Eric, it could be a great initiative to merge our efforts in a single module.
Any ideas ?

Cheers,
Olivier.

Olivier Inizan
Unité de Recherches en Génomique-Info (UR INRA 1164),
INRA, Centre de recherche de Versailles, bat.18
RD10, route de Saint Cyr
78026 Versailles Cedex, FRANCE

olivier.ini...@versailles.inra.fr

Tél: +33 1 30 83 38 25
Fax: +33 1 30 83 38 99
http://urgi.versailles.inra.fr [urgi.versailles.inra.fr]
Twitter: @OlivierInizan


On Wed, 12 Mar 2014, Eric Rasche wrote:


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Sys Admins,

I'm working on some puppet modules for managing galaxy. I'm not sure
how many of you use puppet, but I figured it would be useful to the
community. I'll place them on puppet forge when I'm done.

So far I'm doing the following:

- - enabling initial deployment (updating?) with the vcsrepo module

- - managing the following files: job_conf.xml, tool_conf.xml,
tool_sheds_conf.xml, universe_wsgi.ini

If there's anything that you find important to be managed, or features
you'd like to see, please email me and let me know!

Cheers,
Eric

- --
Eric Rasche
Programmer II
Center for Phage Technology
Texas A&M University
College Station, TX 77843
404-692-2048
e...@tamu.edu
rasche.e...@yandex.ru
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (GNU/Linux)

iQIcBAEBAgAGBQJTIH0RAAoJEMqDXdrsMcpV2g8P/08nTnb/RLdz6WGfDbryHtBB
6Q5yr4HV2IM/idgjpLQX93jRCyahVuS1E8Sgc1x9ViGml93/ssHJZ9GIsf1PJM5/
erygrVS6njhWgefMQpoEipfTOHqxCvHhJk5xnvRc/EeUj0Lkj6YKoVpzae/6XilH
Tav9/ocTNrH40HBjuFGsK7Q/IP1C+vNr/q7GPXq6Ek6dWd23qxRCYszuGgS2t3s/
i/+YmZCwHH0I25ssujGsYzsrsTfOBVBNvAj9dUmvyE+9WSLZQVw4919VKXDKNuSs
7oFBPJD2hpqSKJxf7IFChFL9WhQ4e1gykx+Z+7MhBpekUeYvwIzABxFNmQLOdl+Q
4R+mEADyAopD2enl23XkePX/FRag839zYSeEOWjvEfC8qgUWFXEJyjJ9JLqXE+Bc
QOAMpmC/BMI5qIxIvKJ2ASZuqBXFxdRcGs1xaqJ2bWN+vm1fD+jzpD33KjHcegQr
gf0UvhJj3aoR0Pua3vuWkWP6wPBt08F6Qab/6tQGIeQUyzZRCGyVggd4SgN+Xml5
0OB5olauA+Nr0wD+tDiCQwE+iIL/U61nU5A0yCmEMVfBem8YyKPtjuNZ7E/OZ9fy
gv4uVmGDIfM+IySnWlmxqX2nYb8AODrZiQifT3kG93jh4HtNaoc4yTNeRYcFwTNi
r8tJpajbO8Q0MVVj+bWn
=aFP5
-END PGP SIGNATURE-
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Puppet Modules: a new project

I love this effort! I would recommend posting these on github instead
of bitbucket - I think it has become kind of a puppet best practice to
host collaborative repositories such as these there.

For what it is worth - I have a set of puppet modules for hosting
Galaxy that have a dependency on the puppetlabs Apache module -
https://github.com/jmchilton/puppet-jmchilton/tree/master/modules/galaxy/
- they are not general at all but I will try to find them time to
refactor my stuff against your effort and pushes enhancements of
general usefulness if I can.

I think most of the rest of the Galaxy team prefers Ansible to puppet
(for instance Nate will likely release some Ansible roles he uses to
manage usegalaxy.org in the near future) - but I think the two efforts
will dovetail nicely and hopefully provide roadmaps for best practice
examples for people who are deploying and managing Galaxy - however
they do it.

-John

On Fri, Mar 14, 2014 at 9:20 AM, Olivier Inizan
 wrote:
> Dear galaxy-dev,
>
> After exchanges with Eric we have decided to create:
> -a common puppet module for galaxy hosted on the puppet forge:
> https://forge.puppetlabs.com/urgi/galaxy
> -a common repository hosted on a public server:
> We want to have the opinion of the galaxy-team and galaxy-dev community for
> the choice of the public server.
> Should we push the code on github or bitbucket ?
> Does this question matter anyway ? :)
>
> Thanks for your answers,
>
> Olivier
>
>
> Olivier Inizan
> Unité de Recherches en Génomique-Info (UR INRA 1164),
> INRA, Centre de recherche de Versailles, bat.18
> RD10, route de Saint Cyr
> 78026 Versailles Cedex, FRANCE
>
> olivier.ini...@versailles.inra.fr
>
> Tél: +33 1 30 83 38 25
> Fax: +33 1 30 83 38 99
> http://urgi.versailles.inra.fr [urgi.versailles.inra.fr]
> Twitter: @OlivierInizan
>
>
> On Thu, 13 Mar 2014, Olivier Inizan wrote:
>
>> Hi Eric and Sys Admin,
>>
>> It's a great news !
>>
>> We have already published a puppet module for a basic galaxy install and
>> update:
>> http://forge.puppetlabs.com/urgi/galaxy
>>
>> We also plan to update it for configuration files management.
>> Eric, it could be a great initiative to merge our efforts in a single
>> module.
>> Any ideas ?
>>
>> Cheers,
>> Olivier.
>>
>> Olivier Inizan
>> Unité de Recherches en Génomique-Info (UR INRA 1164),
>> INRA, Centre de recherche de Versailles, bat.18
>> RD10, route de Saint Cyr
>> 78026 Versailles Cedex, FRANCE
>>
>> olivier.ini...@versailles.inra.fr
>>
>> Tél: +33 1 30 83 38 25
>> Fax: +33 1 30 83 38 99
>> http://urgi.versailles.inra.fr [urgi.versailles.inra.fr]
>> Twitter: @OlivierInizan
>>
>>
>> On Wed, 12 Mar 2014, Eric Rasche wrote:
>>
>>> -BEGIN PGP SIGNED MESSAGE-
>>> Hash: SHA1
>>>
>>> Sys Admins,
>>>
>>> I'm working on some puppet modules for managing galaxy. I'm not sure
>>> how many of you use puppet, but I figured it would be useful to the
>>> community. I'll place them on puppet forge when I'm done.
>>>
>>> So far I'm doing the following:
>>>
>>> - - enabling initial deployment (updating?) with the vcsrepo module
>>>
>>> - - managing the following files: job_conf.xml, tool_conf.xml,
>>> tool_sheds_conf.xml, universe_wsgi.ini
>>>
>>> If there's anything that you find important to be managed, or features
>>> you'd like to see, please email me and let me know!
>>>
>>> Cheers,
>>> Eric
>>>
>>> - --
>>> Eric Rasche
>>> Programmer II
>>> Center for Phage Technology
>>> Texas A&M University
>>> College Station, TX 77843
>>> 404-692-2048
>>> e...@tamu.edu
>>> rasche.e...@yandex.ru
>>> -BEGIN PGP SIGNATURE-
>>> Version: GnuPG v2.0.17 (GNU/Linux)
>>>
>>> iQIcBAEBAgAGBQJTIH0RAAoJEMqDXdrsMcpV2g8P/08nTnb/RLdz6WGfDbryHtBB
>>> 6Q5yr4HV2IM/idgjpLQX93jRCyahVuS1E8Sgc1x9ViGml93/ssHJZ9GIsf1PJM5/
>>> erygrVS6njhWgefMQpoEipfTOHqxCvHhJk5xnvRc/EeUj0Lkj6YKoVpzae/6XilH
>>> Tav9/ocTNrH40HBjuFGsK7Q/IP1C+vNr/q7GPXq6Ek6dWd23qxRCYszuGgS2t3s/
>>> i/+YmZCwHH0I25ssujGsYzsrsTfOBVBNvAj9dUmvyE+9WSLZQVw4919VKXDKNuSs
>>> 7oFBPJD2hpqSKJxf7IFChFL9WhQ4e1gykx+Z+7MhBpekUeYvwIzABxFNmQLOdl+Q
>>> 4R+mEADyAopD2enl23XkePX/FRag839zYSeEOWjvEfC8qgUWFXEJyjJ9JLqXE+Bc
>>> QOAMpmC/BMI5qIxIvKJ2ASZuqBXFxdRcGs1xaqJ2bWN+vm1fD+jzpD33KjHcegQr
>>> gf0UvhJj3aoR0Pua3vuWkWP6wPBt08F6Qab/6tQGIeQUyzZRCGyVggd4SgN+Xml5
>>> 0OB5olauA+Nr0wD+tDiCQwE+iIL/U61nU5A0yCmEMVfBem8YyKPtjuNZ7E/OZ9fy
>>> gv4uVmGDIfM+IySnWlmxqX2nYb8AODrZiQifT3kG93jh4HtNaoc4yTNeRYcFwTNi
>>> r8tJpajbO8Q0MVVj+bWn
>>> =aFP5
>>> -END PGP SIGNATURE-
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>  http://lists.bx.psu.edu/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>  http://galaxyproject.org/search/mailinglists/
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To man

Re: [galaxy-dev] Persistent jobs in cluster queue even after canceling job in galaxy

2014-03-14 Thread Brian Claywell

On Fri, Mar 14, 2014 at 9:01 AM, John Chilton  wrote:
> I believe this problem was fixed by Nate after the latest dist release
> and pushed to the stable branch of galaxy-central.
>
> https://bitbucket.org/galaxy/galaxy-central/commits/1298d3f6aca59825d0eb3d32afd5686c4b1b9294
>
> If you are eager for this bug fix, you can track the latest stable
> branch of galaxy-central instead of the galaxy-dist tag mentioned in
> the dev news. Right now it has some other good bug fixes not in the
> latest release.

Ah, got it, thanks! Is it unfeasible to push bug fixes like those back
to galaxy-dist/stable so those of us that would prefer stable to
bleeding-edge don't have to cherry-pick commits?


-- 
Brian Claywell, Systems Analyst/Programmer
Fred Hutchinson Cancer Research Center
bclay...@fhcrc.org
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Tool Testing Enhancements

Hello Tool Developers,

Haven't known when to send this out, but I figure since you haven't
received any e-mail from me today it might be a good time.

tl;dr - Tool functional tests experienced a significant overhaul over
the last release and will continue to change over the next couple
releases, but in a backward compatible so hopefully you will not need
to care unless you want to.

The last release included a considerable overhaul to the framework
used to functionally test tools (i.e. actually running those
tags in tool XML). This includes changes to how the tests are executed
and an expansion in the expressiveness of these tests.

As some background for why the changes are being made, currently tools
are tested by starting up a test Galaxy instance and using a library
called Twill to exercise the GUI and upload data and run tools through
by exercising actual HTML pages.

The last release added the optional ability to exercise these tool
tests via Galaxy's tool API instead of via web forms, with the
slightly longer term goal of eliminating the Twill testing option
altogether. This will allow the Galaxy tool form API to evolve more
quickly, eliminate much of the testing complexity and many of the
limitations imposed by Twill while ensuring that Galaxy's tool running
API is sufficiently expressive.

I have put a significant amount of effort into backward compatibility
- so all current tests SHOULD pass using the new method. I know some
people have included significant hacks with their test definitions to
sculpt them deal with Twill-isms. These may fail - I would like to
catalog what these are and see if we can hack up the test framework to
deal with them or transition these tests to use cleaner mechanisms
allowed for by the more direct representation of tool parameters
consumed by the API.

If you are running tests locally with the run_functional_tests.sh
script distributed with Galaxy, simply setting the environment
variable GALAXY_TEST_DEFAULT_INTERACTOR to 'api' before running the
script and all tests will be run through the API. Individual tests can
be set to target the API (for instance if they use newer, API-only
features outlined below) by adding the XML attribute interactor="api"
to the test tag in the tool XML. This change works right now if you
want to update test tool shed tools to target the new testing method
and provide feedback.

As part of this transition, many bugs were fixed that affect both
styles of test and new features were added. The biggest improvement is
the ability to specify parameters in a nested structure matching the
inputs themselves. This allows disambiguation of repeat and
conditional parameters for both styles of functional tests and in the
case of API driven tests allows specification of any number of repeat
block instances in tests (one still can only specify at most one
instance for any repeat tag while using Twill driven tests).

The following annotated tool XML examples concretely demonstrate this
new expressiveness.

https://bitbucket.org/galaxy/galaxy-central/src/tip/test/functional/tools/disambiguate_repeats.xml
https://bitbucket.org/galaxy/galaxy-central/src/tip/test/functional/tools/disambiguate_cond.xml

Additional Enhancements:

Enhancement (Twill and API): Allow unspecified conditional test
parameters. Previously specifying a parameter in a conditional but not
the value of the conditional test parameter itself (the select or
boolean parameter the conditional changes based on) resulted in
cryptic test errors. Now the default is assumed if unspecified.
https://bitbucket.org/galaxy/galaxy-central/commits/8b99c7bc5e11cd7c452c2ad86f951109b495271c
https://bitbucket.org/galaxy/galaxy-central/commits/76bd49e658876f5cc90ec5b80033f13b058593b8

Enhancement (Twill and API): Allow specifying tests for composite
extra files without checking primary file.
https://bitbucket.org/galaxy/galaxy-central/commits/45c1d11cfff552ce095c0561b5fba2cb5d3610e3

Enhancement (API-only): Allow testing metadata properties on test
output datasets.
https://bitbucket.org/galaxy/galaxy-central/commits/01f05ab8df7092b6e03f35faf7f80b47acafa0eb

Enhancement (API-only): Remove restriction of output order
specification.
https://bitbucket.org/galaxy/galaxy-central/commits/d05be33ad6b772c2a688d875c4cf895d6afac4e3

Enhancement: Allow testing repeat blocks with min="1" (twill) or any
non-zero value (API)
https://bitbucket.org/galaxy/galaxy-central/commits/4615f7424dcb4d2b61be5f388c761f45a1302a03

Enhancement (API-only): Allow testing of data parameters with multiple="True".
https://bitbucket.org/galaxy/galaxy-central/commits/211f30207d48101a1250983d4f0f55c937f00a2e

Fuller break down of this is development is available on Trello
https://trello.com/c/MsJctFem.

Once Galaxy and the tool shed have been modified to target the API
when testing tools by default the tool XML syntax wiki page will be
updated with examples of this new functionality.

-John
_

Re: [galaxy-dev] Persistent jobs in cluster queue even after canceling job in galaxy

2014-03-14 Thread Ravi Alla

What are the best practices for updating galaxy? Also is there a quick command 
I can run to see what version of galaxy I am running?
On Mar 14, 2014, at 12:23 PM, Brian Claywell  wrote:

> On Fri, Mar 14, 2014 at 9:01 AM, John Chilton  wrote:
>> I believe this problem was fixed by Nate after the latest dist release
>> and pushed to the stable branch of galaxy-central.
>> 
>> https://bitbucket.org/galaxy/galaxy-central/commits/1298d3f6aca59825d0eb3d32afd5686c4b1b9294
>> 
>> If you are eager for this bug fix, you can track the latest stable
>> branch of galaxy-central instead of the galaxy-dist tag mentioned in
>> the dev news. Right now it has some other good bug fixes not in the
>> latest release.
> 
> Ah, got it, thanks! Is it unfeasible to push bug fixes like those back
> to galaxy-dist/stable so those of us that would prefer stable to
> bleeding-edge don't have to cherry-pick commits?
> 
> 
> -- 
> Brian Claywell, Systems Analyst/Programmer
> Fred Hutchinson Cancer Research Center
> bclay...@fhcrc.org


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Verifying test output datatypes, was: Problem with and conditional inputs?