date:20120207

Re: [galaxy-dev] Syntax issue for adding new tool

2012-02-07 Thread Louise-Amélie Schmitt


Hello,

I'm not sure this will fix the issue but you might have to use
-p $param_file.params
instead of
-p $params
in your command line.

Best,
L-A


Le 06/02/2012 22:15, Jeffrey Long a écrit :

Hi all,
I am having trouble with what looked to me at first to be a "simple" 
syntactical issue with writing the XML wrapper for a new Galaxy tool. 
 I had thought this would be easy, but perhaps I am missing something 
incredibly simple, as I am having no success at all.  The relevant 
parts look something like this:



mycommand.sh
-i $input
#if $param_file.use_params == "true"#
-p $params
#end if





Yes
No


value=""/>





It's very simple, I just want the line "-p $params" to appear in the 
command line when the parameter "use_params" parameter is set to true.


However, when building the command line statement to run, I get this 
error:


--
Traceback (most recent call last):
  File 
"/Users/jrl909/Galaxy/galaxy-dist/lib/galaxy/jobs/runners/local.py", 
line 59, in run_job

job_wrapper.prepare()
  File "/Users/jrl909/Galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", 
line 424, in prepare

self.command_line = self.tool.build_command_line( param_dict )
  File 
"/Users/jrl909/Galaxy/galaxy-dist/lib/galaxy/tools/__init__.py", line 
1827, in build_command_line

command_line = fill_template( self.command, context=param_dict )
  File "/Users/jrl909/Galaxy/galaxy-dist/lib/galaxy/util/template.py", 
line 9, in fill_template

return str( Template( source=template_text, searchList=[context] ) )
  File 
"/Users/jrl909/Galaxy/galaxy-dist/eggs/Cheetah-2.2.2-py2.7-macosx-10.6-intel-ucs2.egg/Cheetah/Template.py", 
line 1004, in __str__

return getattr(self, mainMethName)()
  File 
"cheetah_DynamicallyCompiledCheetahTemplate_1328561167_73_97196.py", 
line 84, in respond
NotFound: cannot find 'use_params' while searching for 
'param_file.use_params'

michael
---

As far as I can tell, the syntax I've used here is the same as that 
found in the example file suggested by the Galaxy documentation: 
interval2maf.xml  (what I see as the "relevant" sections being 
reproduced below):


---



#if $maf_source_type.maf_source == "user" #interval2maf.py 
--dbkey=${input1.dbkey} --chromCol=${input1.metadata.chromCol} 
--startCol=${input1.metadata.startCol} 
--endCol=${input1.metadata.endCol} 
--strandCol=${input1.metadata.strandCol} 
--mafFile=$maf_source_type.mafFile 
--mafIndex=$maf_source_type.mafFile.metadata.maf_index 
--interval_file=$input1 --output_file=$out_file1 
--mafIndexFile=${GALAXY_DATA_INDEX_DIR}/maf_index.loc 
--species=$maf_source_type.species


#else #interval2maf.py 
--dbkey=${input1.dbkey} --chromCol=${input1.metadata.chromCol} 
--startCol=${input1.metadata.startCol} 
--endCol=${input1.metadata.endCol} 
--strandCol=${input1.metadata.strandCol} 
--mafType=$maf_source_type.mafType --interval_file=$input1 
--output_file=$out_file1 
--mafIndexFile=${GALAXY_DATA_INDEX_DIR}/maf_index.loc 
--species=$maf_source_type.species


#end if# 
--split_blocks_by_species=$split_blocks_by_species_selector.split_blocks_by_species


#if $split_blocks_by_species_selector.split_blocks_by_species == 
"split_blocks_by_species"#



--remove_all_gap_columns=$split_blocks_by_species_selector.remove_all_gap_columns


#end if





...

blahblahblah

...






Split by species

Do not 
split















Collapse empty 
columns


Do not collapse











---
As you can see, the field 'split_blocks_by_species' is referenced in 
an #if-block in the command specification the same way my 'use_params' 
field is.


I would like very much to know the apparently crucial difference that 
I'm missing!


Thanks,
-Jeff




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

   http://lists.bx.psu.edu/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

2012-02-07 Thread Florent Angly

Hi Amanda,
I would certainly be interested in using your helpful QIIME wrappers if
you put them on the Toolshed.

Best,
Florent

On 06/02/12 06:22, Amanda Zuzolo wrote:

Hello, all.

I have been working on getting the Qiime scripts into Galaxy as
mentioned before, and they are working with Qiime 1.3.0. I have edited
the wrapper file that Jim Johnson wrote to create more flexibility,
especially in cases where the tool looks for a specific file type
extension (for example, a .fna file), or where the tool normally
outputs something to the command line that is not normally picked up
in Galaxy.

So far, I have completely finished fixing the XML files to the latest
documentation for the entire Pick OTU process, Alpha Diversity, and
Beta Diversity, as well as other miscellaneous functions. Currently, I
am working on making scripts for jack-knifing functional. I determined
that it would be easier to get individual scripts functional, rather
than workflow scripts, since that allows the end-user to have more
control. Additionally, the workflow scripts can easily be recreated by
using Galaxy's workflows.

As far as the toolshed goes, I don't believe I know the ins and outs
yet, but I would be more than willing to learn if people would benefit
from having these versions in that repository.

2012/1/29 Jim Johnson:

Pat,

That sounds great. Do one of you want to take ownership of the toolshed
repository?
At minimum, we should add developers to the list that can push changes.

Thanks,

On 1/28/12 9:37 AM, Gillevet Patrick wrote:

Jim et al

Amanda has most of the scripts working now and will be putting them up on
the toolshed.
She will be in touch as soon as the scripts are validated a couple of times
with different datasets.

cheers...
Pat

On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:

It is easiest to generate tools for galaxy when the applications or scripts
can take arbitrarily named input files and generate output to given path
names.
Input directories, output directories are very convenient on the command
line, but more of a challenge when crafting a galaxy tool.
That said, many applications require a wrapper script to work with in
galaxy.
Thank you for the consistent script_info[] help/usage syntax in the qiime
scripts, which enabled me to generate a skeleton galaxy tool_config file
for each qiime script.

I had some time last spring to work on integrating qiime into galaxy.
Unfortunately, I haven't had any time since to work on this.
I put those partial results on the Galaxy Tool Shed:
http://toolshed.g2.bx.psu.edu/
There's a continuing effort at George Mason University to incorporate qiime
into galaxy tools, so you may want to ask them what they need.

I started by generating galaxy tool_config files, e.g. align_seqs.xml, by
using python to get the script_info[] from the qiime script:

$ cat generate_tool_config.bash
#!/usr/bin/env bash
python $1> ${1%.*}.help
cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h>
${1%.*}.log

(I'll attach tool_template.txt )

This generated skeleton tool_config .xml files that I could then edit as
needed.
( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax )

I originally was calling all qiime scripts from a tool wrapper:
qiime_wrapper.py
But, if a script can be called with any input filepaths and write its
results to any filepaths, and only writes to STDERR when it fails, then you
could call that script directly.

When should you use a tool_wrapper or call the qiime script directly?
Many of the qiime scripts could probably be called directly, especially if
it can be called with arbitary input/output file pathnames.
The reasons for using a tool wrapper may be if input/output needs to be
manipulated, moved, renamed in order to be used by the qiime script.
You'll also need a tool wrapper if the names or number of the output files
can not be determined from the parameter settings.
( http://wiki.g2.bx.psu.edu/Admin/Tools/Multiple%20Output%20Files )
If your tool relies on a file ext to determine a format, you'll have to
rename the input.
( Galaxy dataset pathnames will look something like:
//072/dataset_72931.dat )
The format/type of a dataset is stored in its metadata, so the tool_config
can use that information, especially if a script can take muliple
alternative input formats.
A tool_wrapper can also be used to manage the stdout or stderr from a
tool. Galaxy currently interprets any output on stderr as a failure.

A couple changes in galaxy should make somethings easier than when I first
attempted this:
- galaxy now accepts dataset requests with sub directories. (
https://bitbucket.org/galaxy/galaxy-central/issue/494/support-sub-dirs-in-extra_files_path-patch
)
That means that output HTML files with links into sub directories can be
left intact, with the html copied to the output dataset and the linked files
to its "extra_files_path".
- if you know the pathname of an output relative to the wor

[galaxy-dev] Importing data from password protected UCSC Genome Browser

2012-02-07 Thread Federico De Masi

Dear colleagues,

I have been struggling with this issue for quite a while now, and I
don't seem to find a solution (neither in situ nor in webo).

here it is:

Because of the confidentiality of our data, I have setup both a Galaxy
server and a UCSC Genome Browser locally on my server. Both are run
through Apache, using the same .htpasswd authentication method and file
(ie: same usernmae/password combo for each).

Everything works great except that I do not seem to be able to
send/receive data from Galaxy to UCSC and vice versa.

When I try to "get data" from UCSC main (my local instance), I get the
followin error:

"Tool execution generated the following error message:
The remote data source application may be off line, please try again
later. Error: EOF when reading a line

The tool produced the following additional output:
Enter username for SG at 127.0.0.1:"

where SG is the AuthName in vhost.conf and 127.0.0.1 is UCSC's IP.

Whenever I try to to send data to UCSC ("Display at UCSC main" in the
history boxes), I get this guy:

"Expected 200
http://127.0.0.3/display_application/48916fac0de9a85d/ucsc_bigwig/main/b9ba5eb40e0fec32/param/track:

401 Authorization Required"

Where 127.0.0.3 is galaxy's IP.

I have:
- hardcoded username:passowrd on UCSC's "send to galaxy" routine but to
no avail
- hardcode username:passowrd on Galaxy's definiton of UCSC's "main" URL,
but to no avail

- I have removed UCSC authentication, and everything works fine
- I have kept UCSC authentication and let Galaxy "open", doesn't work
- I remove authentication to both servers, and it all works cool.

Somehow, Galaxy doesn't like to talk to a protected UCSC browser.

I'll be happy to post my vhost.conf files if needed.

Can anyone help, please?

Thanks a lot,

Fred

--
Federico De Masi, PhD,
Assistant Professor
The Technical University of Denmark - DTU
Center for Biological Sequence Analysis - CBS
Kemitorvet 208/002
DK-2800 KGS. LYNGBY, DENMARK
Telephone: (+45) 45 25 24 21
Fax: (+45) 45 93 15 85
http://rg.cbs.dtu.dk
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

[galaxy-dev] Adding a new python tool

2012-02-07 Thread Sarah Maman


Hello,

I managed to add a new tool coded in bash but I am having some 
difficulty adding a tool coded in python.


Here are the contents of the test files:
* Content of test.xml file in galaxy/galaxy-dist/tools/test/test.xml :

 Test description
 test.py


* Content of test.py file in galaxy/galaxy-dist/tools/test/test.py :
#!/usr/bin/python

import os

for param in os.environ.keys():
   print "%20s %s" % (param,os.environ[param])

* Content of tool_conf_xml file in galaxy/galaxy-dist/ :


 

   

   
   
   ..

Could you, please, explain me why the new tool "test" isn't avalaible in 
Tools/Get Data windows (on the left, before uploadFile and upload tools ) ?


Thanks in advance,
Sarah

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

 http://lists.bx.psu.edu/

Re: [galaxy-dev] Adding a new python tool

2012-02-07 Thread Langhorst, Brad

Sarah:

I've had trouble before with invalid xml files (in your case test.xml)
I don't see an obvious problem with your file, but I've never tried such a
minimal file.

Have a look at galaxy's log during startup to see if it tells you what it
does not like about the file.

Brad
-- 
Brad Langhorst
New England Biolabs
langho...@neb.com

On 2/7/12 6:32 AM, "Sarah Maman"  wrote:

>Hello,
>
>I managed to add a new tool coded in bash but I am having some
>difficulty adding a tool coded in python.
>
>Here are the contents of the test files:
>* Content of test.xml file in galaxy/galaxy-dist/tools/test/test.xml :
>
>  Test description
>  test.py
>
>
>* Content of test.py file in galaxy/galaxy-dist/tools/test/test.py :
>#!/usr/bin/python
>
>import os
>
>for param in os.environ.keys():
>print "%20s %s" % (param,os.environ[param])
>
>* Content of tool_conf_xml file in galaxy/galaxy-dist/ :
>
>
>  
>
>
>
>
>..
>
>Could you, please, explain me why the new tool "test" isn't avalaible in
>Tools/Get Data windows (on the left, before uploadFile and upload tools )
>?
>
>Thanks in advance,
>Sarah
>
>___
>Please keep all replies on the list by using "reply all"
>in your mail client.  To manage your subscriptions to this
>and other Galaxy lists, please use the interface at:
>
>  http://lists.bx.psu.edu/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Looking for recommendations: How to run galaxy workflows in batch

2012-02-07 Thread Louise-Amélie Schmitt


Hello Dannon

Could it be possible to have the input dataset's display name appended 
to the new history's name instead of plain numbers when the "Send 
results in a new history" option is checked?


This new feature is indeed very useful (thanks a million for it) but 
the numbered suffixes make it hard to track what new history belongs to 
which dataset.


Thanks,
L-A


Le 06/02/2012 23:00, Dannon Baker a écrit :

This method only works for single inputs at the moment, though eventually it'd 
be nice to allow pairing.  Another option for you would be to use the workflows 
API, with which you can definitely specify multiple inputs.  See 
workflow_execute.py in the scripts/api folder of your galaxy installation for 
one method of doing this.

-Dannon


On Feb 6, 2012, at 4:53 PM, Dave Lin wrote:


Thank you Dannon. That is helpful.


What if I need to specify multiple inputs per run (i.e. .csfasta + .qual file)?

-Dave

On Mon, Feb 6, 2012 at 1:27 PM, Dannon Baker  wrote:
Hi Dave,

Yes, galaxy's standard run-workflow dialog has a feature where you can select multiple 
datasets as input for a single "Input Dataset" step.  To do this, click the 
icon referenced by the tooltip in the screenshot below to select multiple files.  All 
parameters remain static between executions except for the single input dataset that gets 
modified for each run, and that only one input dataset can be set to multiple files in 
this fashion.

-Dannon







On Feb 6, 2012, at 4:18 PM, Dave Lin wrote:


Hi All,

I'm looking to batch process 40 large data sets with the same galaxy workflow.

This obviously can be done in a brute-force manual manner.

However, is there a better way to schedule/invoke these jobs in batch

1) from the UI with a plugin
2) command-line
3) web-service

Thanks in advance for any pointers.
Dave

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

   http://lists.bx.psu.edu/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

 http://lists.bx.psu.edu/

Re: [galaxy-dev] Looking for recommendations: How to run galaxy workflows in batch

2012-02-07 Thread Dannon Baker

Thanks for the suggestion, I like that!  I'll make the change shortly.

-Dannon

On Feb 7, 2012, at 8:03 AM, Louise-Amélie Schmitt wrote:

> Hello Dannon
> 
> Could it be possible to have the input dataset's display name appended to the 
> new history's name instead of plain numbers when the "Send results in a new 
> history" option is checked?
> 
> This new feature is indeed very useful (thanks a million for it) but the 
> numbered suffixes make it hard to track what new history belongs to which 
> dataset.
> 
> Thanks,
> L-A
> 
> 
> Le 06/02/2012 23:00, Dannon Baker a écrit :
>> This method only works for single inputs at the moment, though eventually 
>> it'd be nice to allow pairing.  Another option for you would be to use the 
>> workflows API, with which you can definitely specify multiple inputs.  See 
>> workflow_execute.py in the scripts/api folder of your galaxy installation 
>> for one method of doing this.
>> 
>> -Dannon
>> 
>> 
>> On Feb 6, 2012, at 4:53 PM, Dave Lin wrote:
>> 
>>> Thank you Dannon. That is helpful.
>>> 
>>> 
>>> What if I need to specify multiple inputs per run (i.e. .csfasta + .qual 
>>> file)?
>>> 
>>> -Dave
>>> 
>>> On Mon, Feb 6, 2012 at 1:27 PM, Dannon Baker  wrote:
>>> Hi Dave,
>>> 
>>> Yes, galaxy's standard run-workflow dialog has a feature where you can 
>>> select multiple datasets as input for a single "Input Dataset" step.  To do 
>>> this, click the icon referenced by the tooltip in the screenshot below to 
>>> select multiple files.  All parameters remain static between executions 
>>> except for the single input dataset that gets modified for each run, and 
>>> that only one input dataset can be set to multiple files in this fashion.
>>> 
>>> -Dannon
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Feb 6, 2012, at 4:18 PM, Dave Lin wrote:
>>> 
 Hi All,
 
 I'm looking to batch process 40 large data sets with the same galaxy 
 workflow.
 
 This obviously can be done in a brute-force manual manner.
 
 However, is there a better way to schedule/invoke these jobs in batch
 
 1) from the UI with a plugin
 2) command-line
 3) web-service
 
 Thanks in advance for any pointers.
 Dave
 
 ___
 Please keep all replies on the list by using "reply all"
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
 
  http://lists.bx.psu.edu/
>>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>> 
>>   http://lists.bx.psu.edu/
> 


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Galaxy Main Server Tool Repository

2012-02-07 Thread Nate Coraor

Hi Charlie,

Just a minor correction, the Main server is Solaris 10, the clusters on which 
jobs run are Debian testing/unstable and RHEL 5.7.

--nate

On Feb 6, 2012, at 9:01 PM, Dannon Baker wrote:

> Hi Charlie,
> 
> Our main galaxy server is actually running CentOS 5.10.  The galaxy cloud 
> instances are built on Ubuntu 10.04 LTS.  That said, as far as I know, any 
> *nix with python 2.5 or above should work fine.  
> 
> The following two links have lots of information regarding setting up your 
> own server:
> 
> http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Production%20Server
> http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup
> 
> Automated setup of many tools is available currently via mi-deployment, see 
> https://bitbucket.org/afgane/mi-deployment for more information on that.  In 
> coming months, automated deployment of many tool binaries will be available 
> directly from within Galaxy via the Tool Shed.
> 
> And lastly, this is definitely the right list for this type of question.  
> Thanks for your interest in Galaxy!
> 
> Dannon
> 
> 
> On Feb 6, 2012, at 4:53 PM, Li Charlie Xia wrote:
> 
>> Dear all,
>> 
>> New bee here. Wonder if this is the right place to post following questions:
>> 
>> We are interested in setting up a server like galaxy main server.
>> Wonder which Linux Platform is the main server running on.
>> Also is there a method for batch installing tools that now are available on 
>> the main galaxy server?
>> If not, wonder if can point me to the documentation of the main galaxy 
>> server installation.
>> 
>> Please excuse me if this is not the right list! 
>> Thanks a lot!
>> 
>> Charlie
>> 
>> -- 
>> Li Charlie Xia
>> PhD Candidate
>> Computational Biology and Bioinformatics Program at University of Southern 
>> California
>> 1050 Childs Way, RRI 416C
>> Los Angeles, CA 90089-2910, USA
>> OP: 213-740-5168
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>> 
>> http://lists.bx.psu.edu/
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> 
>  http://lists.bx.psu.edu/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] manage_db.sh error

2012-02-07 Thread Nate Coraor

On Feb 6, 2012, at 1:27 PM, Zachary Charlop-Powers wrote:

> Hi Galaxy Team,
> 
> I am running a small local instance of Galaxy and I've run into a database  
> error following an upgrade to the Jan27th  galaxy-dist release.
> 
> The last line when I run run.sh tells me to upgrade my database:
> 
>Exception: Your database has version '85' but this code expects 
> version '90'.  Please backup your database and then migrate the schema by 
> running 'sh manage_db.sh upgrade'.
> 
> But when I run the manage_db.sh  I run into the following errors:
> 
> 
> $ sh manage_db.sh 
> Traceback (most recent call last):
>  File "./scripts/manage_db.py", line 12, in 
>from migrate.versioning.shell import main
>  File 
> "/home/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/shell.py",
>  line 7, in 
>from migrate.versioning.base import *
>  File 
> "/home/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/base/__init__.py",
>  line 5, in 
>from const import databases,operations
>  File 
> "/home/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/base/const.py",
>  line 7, in 
>from sqlalchemy.util import OrderedDict
> ImportError: No module named sqlalchemy.util
> 
> 
> The first thing that pops out is that I the sqlalchemy egg is part of py2.7 
> while I am using python2.6.  Any suggestions on how to troubleshoot this?

Hi Zach,

You could try removing your eggs/ directory and fetching them again with:

% python ./scripts/fetch_eggs.py

--nate

> 
> 
> thanks,
> zach cp
> 
> 
> 
> 
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> 
>  http://lists.bx.psu.edu/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

2012-02-07 Thread Jeffrey Long

Hello Amanda,
I was just about to embark on EXACTLY this process, so I would certainly be
very interested in saving myself some work.
Would there be any issue (that you're aware of, of course) with using QIIME
1.4.0 instead of 1.3?

-Jeff

On Tue, Feb 7, 2012 at 2:32 AM, Florent Angly wrote:

> Hi Amanda,
> I would certainly be interested in using your helpful QIIME wrappers if
> you put them on the Toolshed.
> Best,
> Florent
>
> On 06/02/12 06:22, Amanda Zuzolo wrote:
>
>> Hello, all.
>>
>> I have been working on getting the Qiime scripts into Galaxy as
>> mentioned before, and they are working with Qiime 1.3.0. I have edited
>> the wrapper file that Jim Johnson wrote to create more flexibility,
>> especially in cases where the tool looks for a specific file type
>> extension (for example, a .fna file), or where the tool normally
>> outputs something to the command line that is not normally picked up
>> in Galaxy.
>>
>> So far, I have completely finished fixing the XML files to the latest
>> documentation for the entire Pick OTU process, Alpha Diversity, and
>> Beta Diversity, as well as other miscellaneous functions. Currently, I
>> am working on making scripts for jack-knifing functional. I determined
>> that it would be easier to get individual scripts functional, rather
>> than workflow scripts, since that allows the end-user to have more
>> control. Additionally, the workflow scripts can easily be recreated by
>> using Galaxy's workflows.
>>
>> As far as the toolshed goes, I don't believe I know the ins and outs
>> yet, but I would be more than willing to learn if people would benefit
>> from having these versions in that repository.
>>
>> 2012/1/29 Jim Johnson:
>>
>>> Pat,
>>>
>>> That sounds great.   Do one of you want to take ownership of the toolshed
>>> repository?
>>> At minimum, we should add developers to the list that can push changes.
>>>
>>> Thanks,
>>>
>>> JJ
>>>
>>> On 1/28/12 9:37 AM, Gillevet Patrick wrote:
>>>
>>> Jim et al
>>>
>>> Amanda has most of the scripts working now and will be putting them up on
>>> the toolshed.
>>> She will be in touch as soon as the scripts are validated a couple of
>>> times
>>> with different datasets.
>>>
>>> cheers...
>>> Pat
>>>
>>>
>>>
>>> On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:
>>>
>>>
>>> It is easiest to generate tools for galaxy when the applications or
>>> scripts
>>> can take arbitrarily named input files and generate output to given path
>>> names.
>>> Input directories, output directories are very convenient on the command
>>> line, but more of a challenge when crafting a galaxy tool.
>>> That said, many applications require a wrapper script to work with in
>>> galaxy.
>>> Thank you for the consistent script_info[] help/usage syntax in the qiime
>>> scripts,  which enabled me to generate a skeleton galaxy tool_config file
>>> for each qiime script.
>>>
>>> I had some time last spring to work on integrating qiime into galaxy.
>>> Unfortunately, I haven't had any time since to work on this.
>>> I put those partial results  on the Galaxy Tool Shed:
>>> http://toolshed.g2.bx.psu.edu/
>>> There's a continuing effort at George Mason University to incorporate
>>> qiime
>>> into galaxy tools, so you may want to ask them what they need.
>>>
>>>
>>> I started by generating galaxy tool_config files, e.g. align_seqs.xml,
>>>  by
>>> using python to get the script_info[] from the qiime script:
>>>
>>> $ cat generate_tool_config.bash
>>> #!/usr/bin/env bash
>>> python $1>  ${1%.*}.help
>>> cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h>
>>> ${1%.*}.log
>>>
>>> (I'll attach tool_template.txt )
>>>
>>> This generated skeleton tool_config .xml files that I could then edit as
>>> needed.
>>> ( 
>>> http://wiki.g2.bx.psu.edu/**Admin/Tools/Tool%20Config%**20Syntax)
>>>
>>> I originally was calling all qiime scripts from a tool wrapper:
>>> qiime_wrapper.py
>>> But, if a script can be called with any input filepaths and write its
>>> results to any filepaths, and only writes to STDERR when it fails, then
>>> you
>>> could call that script directly.
>>>
>>>
>>> When should you use a tool_wrapper or call the qiime script directly?
>>>   Many of the qiime scripts could probably be called directly,
>>> especially if
>>> it can be called with arbitary input/output file pathnames.
>>>   The reasons for using a tool wrapper may be if input/output needs to be
>>> manipulated, moved, renamed in order to be used by the qiime script.
>>>   You'll also need a tool wrapper if the names or number of the output
>>> files
>>> can not be determined from the parameter settings.
>>>   ( 
>>> http://wiki.g2.bx.psu.edu/**Admin/Tools/Multiple%20Output%**20Files)
>>>   If your tool relies on a file ext to determine a format, you'll have to
>>> rename the input.
>>>   ( Galaxy dataset pathnames will look something like:
>>> //07

Re: [galaxy-dev] Postgresql To Galaxy

2012-02-07 Thread Nate Coraor

On Feb 7, 2012, at 6:28 AM, Rehan Saleem wrote:

> Hi nate ,
> thanks for reply , actually i want to know how can i add  my own tool in 
> galaxy , how can i show the output of my postgreSQL function in galaxy just 
> asume my postgresql database is connected to galaxy . i need the python and 
> xml code for that . 
> thanks

Hi Rehan,

Your database shouldn't be connected to Galaxy, it should be connected to your 
tool, and then your tool will write the results as the tool's output, which 
will then be available in Galaxy.  The documentation on how to write the tool 
config XML is in the wiki at:

http://wiki.g2.bx.psu.edu/Admin/Tools/Add%20Tool%20Tutorial
http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax

I'd suggest using one of the existing tool config files in Galaxy as a 
reference.  As for the script which queries the database, that would be a new 
tool to write.  Two of our tools use sqlite, which may be useful as a reference 
when writing your tool:

tools/next_gen_conversion/solid2fastq.py
tools/taxonomy/find_diag_hits.py

You are not limited to using Python to write the tool, however, it can be any 
language which you can run on the command line.

--nate

> 
> From: Nate Coraor 
> To: Rehan Saleem  
> Cc: "galaxy-dev@lists.bx.psu.edu"  
> Sent: Monday, February 6, 2012 10:19 PM
> Subject: Re: [galaxy-dev] Postgresql To Galaxy
> 
> On Feb 6, 2012, at 10:56 AM, Rehan Saleem wrote:
> 
> > Hi Nate,
> > well i don’t want to use galaxy default database , i want to set my own 
> > tools for example, i have created a function totaloverlapcount and its work 
> > perfectly fine in my separate  postgresql database called ova , now i want 
> > to add a tool in galaxy with a name Total Overlap Count , as we know 
> > whenever we want to add a tool we configure it in tool_config file.
> > Now if a user click on Total Overlap Count then my postgresql function 
> > totaloverlapcount should be called , and that function will perform query 
> > on my own created database(ova) and then return the result into galaxy 
> > interface
> 
> Hi Rehan,
> 
> Please use 'reply all' to keep replies on the mailing list.
> 
> Since your tool already works with a separate postgres database outside of 
> Galaxy, you should be able to add the tool to Galaxy without modification.  I 
> presume the tool has some means of knowing what host, user and port to 
> connect to, and the setup of this information is the same as it would be when 
> running it on the command line (via a config file, environment, hardcoded in 
> the tool...?).
> 
> --nate
> 
> > 
> > From: Nate Coraor 
> > To: Rehan Saleem  
> > Cc: "galaxy-dev@lists.bx.psu.edu"  
> > Sent: Monday, February 6, 2012 8:07 PM
> > Subject: Re: [galaxy-dev] Postgresql To Galaxy
> > 
> > On Feb 6, 2012, at 6:37 AM, Rehan Saleem wrote:
> > 
> > > Hi ,
> > > i have created postgreSQL database where i have created all functions and 
> > > views which performs annotations and some other functions , how can i 
> > > show my postgresql database table to galaxyas output . i have connected 
> > > my database from universe_wsgi.ini to my linux host where i am running 
> > > galaxy and my database is on windows host, i need the python script which 
> > > take values from table and show them back to galaxy. 
> > > THANKS
> > 
> > Hi Rehan,
> > 
> > I am a little unsure what you're trying to do.  Do you want  to display 
> > contents of this database in some sort of display within Galaxy, use them 
> > as tool inputs, or outputs?
> > 
> > I would not suggest using this custom database as Galaxy's database (set in 
> > database_connection) if it contains separate data from the Galaxy 
> > application.  Galaxy should live in its own database away from any other 
> > tables you use for other purposes.
> > 
> > --nate
> > 
> > > ___
> > > Please keep all replies on the list by using "reply all"
> > > in your mail client.  To manage your subscriptions to this
> > > and other Galaxy lists, please use the interface at:
> > > 
> > >  http://lists.bx.psu.edu/
> > 
> > 
> > 
> 
> 
> 


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Problem with latest database upgrade

2012-02-07 Thread Greg Von Kuster

Hi Thon,

The fix for this issue in now available in the central repo in change set  
revision 271ccd09bc23.  Even though you have a working version of Galaxy, you 
should update your instance to at least this revision, and then do the 
following as soon as possible.

%sh manage_db.sh downgrade 90
91 -> 90... 
done

%sh manage_db.sh upgrade
90 -> 91... 

The problem was caused because you are running postgres version 9 and the 
original version of my migration script did not handle hex bytea for that 
version of postgres.

Thanks for finding this problem!

Greg

On Feb 6, 2012, at 10:08 PM, Anthonius deBoer wrote:

> Oh, and whatever you did in the little patch, it worked...At least I have a 
> working version of galaxy again and the database seems to be all intact..
> 
> Thanks!
> 
> Thon
> 
> On Feb 06, 2012, at 03:52 PM, Greg Von Kuster  wrote:
> 
>> Can you replace lines 75 - 86 of the file 
>> ~/lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py with the 
>> following for loop and send me your paster log?
>> 
>> Thanks!
>> 
>> for row in result:
>> tool_shed_repository_id = row[0]
>> try:
>> repository_metadata = from_json_string( str( row[1] ) )
>> # Create a new row in the tool table for each tool included in 
>> repository.  We will NOT
>> # handle tool_version_associaions because we do not have the 
>> information we need to do so.
>> tools = repository_metadata.get( 'tools', [] )
>> for tool_dict in tools:
>> cmd = "INSERT INTO tool_version VALUES (%s, %s, %s, '%s', 
>> %s)" % \
>> ( nextval( 'tool_version' ), localtimestamp(), 
>> localtimestamp(), tool_dict[ 'guid' ], tool_shed_repository_id )
>> db_session.execute( cmd )
>> count += 1
>> except Exception, e:
>> print "Repository metadata: ", str( row[1] )
>> print "Exception setting tool versions for tools in 
>> tool_shed_repository with id: ", str( tool_shed_repository_id )
>> print str( e )
>> 
>> The lines of code you are replacing look like this:
>> 
>> for row in result:
>> if row[1]:
>> tool_shed_repository_id = row[0]
>> repository_metadata = from_json_string( str( row[1] ) )
>> # Create a new row in the tool table for each tool included in 
>> repository.  We will NOT
>> # handle tool_version_associaions because we do not have the 
>> information we need to do so.
>> tools = repository_metadata.get( 'tools', [] )
>> for tool_dict in tools:
>> cmd = "INSERT INTO tool_version VALUES (%s, %s, %s, '%s', 
>> %s)" % \
>> ( nextval( 'tool_version' ), localtimestamp(), 
>> localtimestamp(), tool_dict[ 'guid' ], tool_shed_repository_id )
>> db_session.execute( cmd )
>> count += 1
>> 
>> 
>> On Feb 6, 2012, at 5:23 PM, Anthonius deBoer wrote:
>> 
>>> I manually dropped the tool_version and tool_association table in the 
>>> database and I am left with the JSON error as shown below:
>>> 
>>> 90 -> 91...
>>> 
>>> Migration script to create the tool_version and tool_version_association 
>>> tables and drop the tool_id_guid_map table.
>>> 
>>> Traceback (most recent call last):
>>>   File "./scripts/manage_db.py", line 63, in 
>>> main( repository=repo, url=db_url )
>>>   File 
>>> "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py",
>>>  line 150, in main
>>> ret = command_func(**kwargs)
>>>   File 
>>> "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py",
>>>  line 221, in upgrade
>>> return _migrate(url, repository, version, upgrade=True, err=err, **opts)
>>>   File 
>>> "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py",
>>>  line 349, in _migrate
>>> schema.runchange(ver, change, changeset.step)
>>>   File 
>>> "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py",
>>>  line 184, in runchange
>>> change.run(self.engine, step)
>>>   File 
>>> "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py",
>>>  line 101, in run
>>> func()
>>>   File "lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py", 
>>> line 78, in upgrade
>>> repository_metadata = from_json_string( str( row[1] ) )
>>>   File 
>>> "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/__init__.py",
>>>  line 384, in loads
>>>   File 
>>> "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/decoder.py",
>>>  line 402, in decode
>>>   File 
>>> "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/decoder.py",
>>>

Re: [galaxy-dev] More helpful cluster job names

2012-02-07 Thread Peter Cock

On Mon, Feb 6, 2012 at 6:17 PM, Nate Coraor  wrote:
>
> On Feb 6, 2012, at 1:05 PM, Nate Coraor wrote:
>
>>
>> On Feb 3, 2012, at 7:30 AM, Peter Cock wrote:
>>
>>> On Fri, Feb 3, 2012 at 11:55 AM, Ryan Golhar
>>>  wrote:
 'qstat -f | grep jobname' will give you the full jobname without 
 truncation.
>>>
>>> Sadly not on our version of SGE - which appears to be SGE 6.2u5 - you
>>> get a bit more information but the job name is still truncated as before.
>>>
 Personally, I think the qstat reporting gives too little information.
>>>
>>> I agree.
>>
>> Try qstat -j 
>>
 I've written a perl script to parse the output of qstat -f to give a bit 
 more
 information.  So truncation of jobnames for qstat shouldn't be an issue.
>>>
>>> Maybe you're right that it is possible to interrogate SGE for the full
>>> name, but I still think Galaxy's job names could be more informative.
>>
>> This was implemented for the PBS runner, I've carried over
>> the format from that to the DRMAA runner in e4d1dd3bdd0d.
>> The format is:
>>
>>       g__
>
> Correction, you'll need 74b6319b38b4 since the only valid
> characters in a DRMAA job name are alphanumeric or
> underscores.
>

Thanks Nate - I'll check this out when I'm back in the office.

Peter

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Problems with custom data types and sniffers in shed tools

2012-02-07 Thread Greg Von Kuster

Hello Ira,

Very sorry for the back-and-forth on this.  The behavior you encountered was 
accurate - the environment in which I was testing was not pristine.  I have 
committed a fix for this issue in revision b4ba8b20d78d, which is now available 
from our central repo.  

I started looking at changing things so that proprietary sniffers are loaded 
before the sniffers defined in the Galaxy distribution, and determined that 
this will be a bit non-trivial.  Problems arise due to potential conflicts 
between a proprietary datatype that has the same extension as a datatype 
defined in the Galaxy distribution.  The approach I've taken to deal with this 
is that datatypes in the Galaxy distribution get loaded first, and then any 
conflicting proprietary datatypes will be ignored.  Loading proprietary 
datatypes first will not allow for this rule.  

I'll gladly listen to advice from the community on this, but in the meantime, 
proprietary datatypes and sniffers are loaded after those in the distribution.

Thanks for your patience and help on this issue.

Greg

On Feb 6, 2012, at 11:37 PM, Ira Cooke wrote:

> Hi Greg, 
> 
> As far as I know there is nothing in my local setup that affects this.  As I 
> test I did the following; 
> 
> Checked out a clean copy of galaxy-central 
> Edited my universe_wsgi.ini as follows;
>   - Changed the port to 8300
>   - Added an admin user
>   - Uncommented this line
>   tool_config_file = tool_conf.xml,shed_tool_conf.xml
> 
> Edited shed_tool_conf.xml to point to ../shed_tools_galaxy_central (which is 
> an empty directory)
> 
> Started up galaxy 
> ./run.sh --reload
> 
> I then went straight to admin -> "Search and browse tool sheds" and selected 
> the main galaxy toolshed. 
> I then selected the gmap tool and told it to install.  My paster log is at 
> the bottom of this mail.
> 
> My system is Mac OSX 10.7.2 and I'm running python 2.7.1
> 
> I then went to the relevant bit of code and inserted some print statements.  
> I found that; 
> 
> On line 191 in registry.py there is;
> 
> module = __import__( datatype_module )
> 
> and the value of datatype_module is galaxy.datatypes.gmap
> 
> this line seems to be causing the exception ... I'm not a python programmer 
> so I don't really know how module importing works ... could this be related 
> to the version of python I'm running?  
> alternatively should the "imported_modules" variable be used here .. since 
> wouldn't galaxy.datatypes.gmap suggest that the module needs to be located 
> inside the main galaxy lib rather than in the shed_tools.
> 
> Strange that I'm the only one who gets these errors?  Any ideas what it could 
> be?
> 
> Ira
> 
> 
> -- paster log --
> 
> Cloning http://toolshed.g2.bx.psu.edu/repos/jjohnson/gmap
> destination directory: gmap
> requesting all changes
> adding changesets
> adding manifests
> adding file changes
> added 11 changesets with 32 changes to 17 files
> updating to branch default
> resolving manifests
> getting README
> getting gmap.xml
> getting gmap_build.xml
> getting gsnap.xml
> getting iit_store.xml
> getting lib/galaxy/datatypes/gmap.py
> getting snpindex.xml
> getting tool-data/datatypes_conf.xml
> getting tool-data/gmap_indices.loc.sample
> 9 files updated, 0 files merged, 0 files removed, 0 files unresolved
> galaxy.util.shed_util DEBUG 2012-02-07 14:25:35,100 Updating cloned 
> repository to revision "93911bac43da"
> resolving manifests
> 0 files updated, 0 files merged, 0 files removed, 0 files unresolved
> docutils WARNING 2012-02-07 14:25:35,512 :10: (WARNING/2) Explicit 
> markup ends without a blank line; unexpected unindent.
> 
> galaxy.util.shed_util DEBUG 2012-02-07 14:25:35,589 Adding new row (or 
> updating an existing row) for repository 'gmap' in the tool_shed_repository 
> table.
> docutils WARNING 2012-02-07 14:25:35,701 :10: (WARNING/2) Explicit 
> markup ends without a blank line; unexpected unindent.
> 
> docutils WARNING 2012-02-07 14:25:35,863 :10: (WARNING/2) Explicit 
> markup ends without a blank line; unexpected unindent.
> 
> galaxy.tools DEBUG 2012-02-07 14:25:35,932 Reloading section: gmap
> galaxy.tools DEBUG 2012-02-07 14:25:35,997 Loaded tool id: 
> toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap/2.0.1, version: 2.0.1.
> galaxy.tools DEBUG 2012-02-07 14:25:36,026 Loaded tool id: 
> toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap_build/2.0.0, version: 2.0.0.
> docutils WARNING 2012-02-07 14:25:36,077 :10: (WARNING/2) Explicit 
> markup ends without a blank line; unexpected unindent.
> 
> galaxy.tools DEBUG 2012-02-07 14:25:36,103 Loaded tool id: 
> toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gsnap/2.0.1, version: 2.0.1.
> galaxy.tools DEBUG 2012-02-07 14:25:36,268 Loaded tool id: 
> toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap_iit_store/2.0.0, version: 
> 2.0.0.
> galaxy.tools DEBUG 2012-02-07 14:25:36,316 Loaded tool id: 
> toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap_snpindex/2.0.0, version: 
> 2.0.0.
> galaxy.datatypes.regis

[galaxy-dev] set_metadata.py

2012-02-07 Thread Bram Slabbinck


Hi

I am running into a problem with the scripts/set_metadata.py file since
the last update of galaxy-dist. In this script the name of the universe
file is hard-coded as 'universe_wsgi.ini', see line 44:

config_file_name = 'universe_wsgi.ini' # Safe assumption?

I renamed the universe file in our setting, due to different Galaxy
installs. Running the set_metadata.py script crashes however with this
coding as it is constantly trying to make a tmpfile in './database/tmp'
which is the default tmp path in universe_wsgi.ini. Replacing
'universe_wsgi.ini' by the name of our universe file solved the problem.
So, how can I dynamically retrieve the name of the launched universe file?

Thx for any help
regards
Bram

--
==
Bram Slabbinck, PhD

Bioinformatics&  Systems Biology Division
VIB Department of Plant Systems Biology, UGent
Technologiepark 927, 9052 Gent, BELGIUM

Email: bram.slabbi...@psb.ugent.be
WWW: http://bioinformatics.psb.ugent.be
==
Please consider the environment before printing this email

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

 http://lists.bx.psu.edu/

Re: [galaxy-dev] Problem with latest database upgrade

2012-02-07 Thread Anthonius deBoer

Thanks for fixing it so fast!

Regards,

Thon

Thon de Boer, Ph.D
Bioinformatics Guru

T: +1.650.799.6839 | E-mail: thondeb...@me.com

http://www.linkedin.com/pub/thon-de-boer/1/1ba/a5b

"IMPORTANT NOTICE: This email message is legally privileged, confidential and 
is for the use of the individual or entity to whom it is addressed. If you have 
received this email message by error, please immediately notify us by email and 
delete the message. Thank you."

On Feb 7, 2012, at 8:11 AM, Greg Von Kuster  wrote:

> Hi Thon,
> 
> The fix for this issue in now available in the central repo in change set  
> revision 271ccd09bc23.  Even though you have a working version of Galaxy, you 
> should update your instance to at least this revision, and then do the 
> following as soon as possible.
> 
> %sh manage_db.sh downgrade 90
> 91 -> 90... 
> done
> 
> %sh manage_db.sh upgrade
> 90 -> 91... 
> 
> The problem was caused because you are running postgres version 9 and the 
> original version of my migration script did not handle hex bytea for that 
> version of postgres.
> 
> Thanks for finding this problem!
> 
> Greg
> 
> On Feb 6, 2012, at 10:08 PM, Anthonius deBoer wrote:
> 
>> Oh, and whatever you did in the little patch, it worked...At least I have a 
>> working version of galaxy again and the database seems to be all intact..
>> 
>> Thanks!
>> 
>> Thon
>> 
>> On Feb 06, 2012, at 03:52 PM, Greg Von Kuster  wrote:
>> 
>>> Can you replace lines 75 - 86 of the file 
>>> ~/lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py with 
>>> the following for loop and send me your paster log?
>>> 
>>> Thanks!
>>> 
>>> for row in result:
>>> tool_shed_repository_id = row[0]
>>> try:
>>> repository_metadata = from_json_string( str( row[1] ) )
>>> # Create a new row in the tool table for each tool included in 
>>> repository.  We will NOT
>>> # handle tool_version_associaions because we do not have the 
>>> information we need to do so.
>>> tools = repository_metadata.get( 'tools', [] )
>>> for tool_dict in tools:
>>> cmd = "INSERT INTO tool_version VALUES (%s, %s, %s, '%s', 
>>> %s)" % \
>>> ( nextval( 'tool_version' ), localtimestamp(), 
>>> localtimestamp(), tool_dict[ 'guid' ], tool_shed_repository_id )
>>> db_session.execute( cmd )
>>> count += 1
>>> except Exception, e:
>>> print "Repository metadata: ", str( row[1] )
>>> print "Exception setting tool versions for tools in 
>>> tool_shed_repository with id: ", str( tool_shed_repository_id )
>>> print str( e )
>>> 
>>> The lines of code you are replacing look like this:
>>> 
>>> for row in result:
>>> if row[1]:
>>> tool_shed_repository_id = row[0]
>>> repository_metadata = from_json_string( str( row[1] ) )
>>> # Create a new row in the tool table for each tool included in 
>>> repository.  We will NOT
>>> # handle tool_version_associaions because we do not have the 
>>> information we need to do so.
>>> tools = repository_metadata.get( 'tools', [] )
>>> for tool_dict in tools:
>>> cmd = "INSERT INTO tool_version VALUES (%s, %s, %s, '%s', 
>>> %s)" % \
>>> ( nextval( 'tool_version' ), localtimestamp(), 
>>> localtimestamp(), tool_dict[ 'guid' ], tool_shed_repository_id )
>>> db_session.execute( cmd )
>>> count += 1
>>> 
>>> 
>>> On Feb 6, 2012, at 5:23 PM, Anthonius deBoer wrote:
>>> 
 I manually dropped the tool_version and tool_association table in the 
 database and I am left with the JSON error as shown below:
 
 90 -> 91...
 
 Migration script to create the tool_version and tool_version_association 
 tables and drop the tool_id_guid_map table.
 
 Traceback (most recent call last):
   File "./scripts/manage_db.py", line 63, in 
 main( repository=repo, url=db_url )
   File 
 "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py",
  line 150, in main
 ret = command_func(**kwargs)
   File 
 "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py",
  line 221, in upgrade
 return _migrate(url, repository, version, upgrade=True, err=err, 
 **opts)
   File 
 "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py",
  line 349, in _migrate
 schema.runchange(ver, change, changeset.step)
   File 
 "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py",
  line 184, in runchange
 change.run(self.engine, step)
   File 
 "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versio

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

2012-02-07 Thread Amanda Zuzolo

Rob,

I am not aware that there would be any issue, as I've verified all the
options with the Qiime documentation that is up now (and I've
eliminated those being deprecated).

Amanda Zuzolo

On 2/7/12, Jeffrey Long  wrote:
> Hello Amanda,
> I was just about to embark on EXACTLY this process, so I would certainly be
> very interested in saving myself some work.
> Would there be any issue (that you're aware of, of course) with using QIIME
> 1.4.0 instead of 1.3?
>
> -Jeff
>
> On Tue, Feb 7, 2012 at 2:32 AM, Florent Angly
> wrote:
>
>> Hi Amanda,
>> I would certainly be interested in using your helpful QIIME wrappers if
>> you put them on the Toolshed.
>> Best,
>> Florent
>>
>> On 06/02/12 06:22, Amanda Zuzolo wrote:
>>
>>> Hello, all.
>>>
>>> I have been working on getting the Qiime scripts into Galaxy as
>>> mentioned before, and they are working with Qiime 1.3.0. I have edited
>>> the wrapper file that Jim Johnson wrote to create more flexibility,
>>> especially in cases where the tool looks for a specific file type
>>> extension (for example, a .fna file), or where the tool normally
>>> outputs something to the command line that is not normally picked up
>>> in Galaxy.
>>>
>>> So far, I have completely finished fixing the XML files to the latest
>>> documentation for the entire Pick OTU process, Alpha Diversity, and
>>> Beta Diversity, as well as other miscellaneous functions. Currently, I
>>> am working on making scripts for jack-knifing functional. I determined
>>> that it would be easier to get individual scripts functional, rather
>>> than workflow scripts, since that allows the end-user to have more
>>> control. Additionally, the workflow scripts can easily be recreated by
>>> using Galaxy's workflows.
>>>
>>> As far as the toolshed goes, I don't believe I know the ins and outs
>>> yet, but I would be more than willing to learn if people would benefit
>>> from having these versions in that repository.
>>>
>>> 2012/1/29 Jim Johnson:
>>>
 Pat,

 That sounds great.   Do one of you want to take ownership of the
 toolshed
 repository?
 At minimum, we should add developers to the list that can push changes.

 Thanks,

 JJ

 On 1/28/12 9:37 AM, Gillevet Patrick wrote:

 Jim et al

 Amanda has most of the scripts working now and will be putting them up
 on
 the toolshed.
 She will be in touch as soon as the scripts are validated a couple of
 times
 with different datasets.

 cheers...
 Pat

 On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:

 It is easiest to generate tools for galaxy when the applications or
 scripts
 can take arbitrarily named input files and generate output to given
 path
 names.
 Input directories, output directories are very convenient on the
 command
 line, but more of a challenge when crafting a galaxy tool.
 That said, many applications require a wrapper script to work with in
 galaxy.
 Thank you for the consistent script_info[] help/usage syntax in the
 qiime
 scripts,  which enabled me to generate a skeleton galaxy tool_config
 file
 for each qiime script.

 I had some time last spring to work on integrating qiime into galaxy.
 Unfortunately, I haven't had any time since to work on this.
 I put those partial results  on the Galaxy Tool Shed:
 http://toolshed.g2.bx.psu.edu/
 There's a continuing effort at George Mason University to incorporate
 qiime
 into galaxy tools, so you may want to ask them what they need.

 I started by generating galaxy tool_config files, e.g. align_seqs.xml,
  by
 using python to get the script_info[] from the qiime script:

 $ cat generate_tool_config.bash
 #!/usr/bin/env bash
 python $1>  ${1%.*}.help
 cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1
 -h>
 ${1%.*}.log

 (I'll attach tool_template.txt )

 This generated skeleton tool_config .xml files that I could then edit
 as
 needed.
 (
 http://wiki.g2.bx.psu.edu/**Admin/Tools/Tool%20Config%**20Syntax)

 I originally was calling all qiime scripts from a tool wrapper:
 qiime_wrapper.py
 But, if a script can be called with any input filepaths and write its
 results to any filepaths, and only writes to STDERR when it fails, then
 you
 could call that script directly.

 When should you use a tool_wrapper or call the qiime script directly?
   Many of the qiime scripts could probably be called directly,
 especially if
 it can be called with arbitary input/output file pathnames.
   The reasons for using a tool wrapper may be if input/output needs to
 be
 manipulated, moved, renamed in order to be used by the qiime script.
   You'll also need a tool wrapper if the names or

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

2012-02-07 Thread Rob Knight

I can't make the call at that time (am in Dhaka) but am very enthusiastic about that effort; please keep me in the loop. I am cc:ing a couple of the people in my lab who also indicated interest in the qiime/galaxy integration effort (though Antonio won't be able to make it either, for the same reason). Thanks!RobOn Feb 7, 2012, at 8:48 AM, Jeffrey Long wrote:Hello Amanda,I was just about to embark on EXACTLY this process, so I would certainly be very interested in saving myself some work. Would there be any issue (that you're aware of, of course) with using QIIME 1.4.0 instead of 1.3?
-JeffOn Tue, Feb 7, 2012 at 2:32 AM, Florent Angly wrote:
Hi Amanda,
I would certainly be interested in using your helpful QIIME wrappers if you put them on the Toolshed.
Best,
Florent

On 06/02/12 06:22, Amanda Zuzolo wrote:

Hello, all.

As far as the toolshed goes, I don't believe I know the ins and outs
yet, but I would be more than willing to learn if people would benefit
from having these versions in that repository.

2012/1/29 Jim Johnson:

Pat,

That sounds great. Do one of you want to take ownership of the toolshed
repository?
At minimum, we should add developers to the list that can push changes.

Thanks,

On 1/28/12 9:37 AM, Gillevet Patrick wrote:

Jim et al

Amanda has most of the scripts working now and will be putting them up on
the toolshed.
She will be in touch as soon as the scripts are validated a couple of times
with different datasets.

cheers...
Pat

On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:

I started by generating galaxy tool_config files, e.g. align_seqs.xml, by
using python to get the script_info[] from the qiime script:

$ cat generate_tool_config.bash
#!/usr/bin/env bash
python $1> ${1%.*}.help
cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h>
${1%.*}.log

(I'll attach tool_template.txt )

This generated skeleton tool_config .xml files that I could then edit as
needed.
( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax )

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

2012-02-07 Thread Daniel McDonald

I'll be there
Daniel



On Feb 7, 2012, at 10:30, Greg Caporaso  wrote:

> OK, let's plan on a Skype call at 1pm MT/3pm ET this Thursday (9 Feb
> 2012). I will initiate the call - my Skype ID is gregcaporaso. Please
> let me know if you'd like to join the call, and send my your skype id.
>
> Looking forward to talking about this!
>
> Greg
>
> 2012/2/7 Rob Knight :
>> I can't make the call at that time (am in Dhaka) but am very enthusiastic
>> about that effort; please keep me in the loop. I am cc:ing a couple of the
>> people in my lab who also indicated interest in the qiime/galaxy integration
>> effort (though Antonio won't be able to make it either, for the same
>> reason). Thanks!
>>
>> Rob
>>
>> On Feb 7, 2012, at 8:48 AM, Jeffrey Long wrote:
>>
>> Hello Amanda,
>> I was just about to embark on EXACTLY this process, so I would certainly be
>> very interested in saving myself some work.
>> Would there be any issue (that you're aware of, of course) with using QIIME
>> 1.4.0 instead of 1.3?
>>
>> -Jeff
>>
>> On Tue, Feb 7, 2012 at 2:32 AM, Florent Angly 
>> wrote:
>>>
>>> Hi Amanda,
>>> I would certainly be interested in using your helpful QIIME wrappers if
>>> you put them on the Toolshed.
>>> Best,
>>> Florent
>>>
>>> On 06/02/12 06:22, Amanda Zuzolo wrote:

 Hello, all.

 I have been working on getting the Qiime scripts into Galaxy as
 mentioned before, and they are working with Qiime 1.3.0. I have edited
 the wrapper file that Jim Johnson wrote to create more flexibility,
 especially in cases where the tool looks for a specific file type
 extension (for example, a .fna file), or where the tool normally
 outputs something to the command line that is not normally picked up
 in Galaxy.

 So far, I have completely finished fixing the XML files to the latest
 documentation for the entire Pick OTU process, Alpha Diversity, and
 Beta Diversity, as well as other miscellaneous functions. Currently, I
 am working on making scripts for jack-knifing functional. I determined
 that it would be easier to get individual scripts functional, rather
 than workflow scripts, since that allows the end-user to have more
 control. Additionally, the workflow scripts can easily be recreated by
 using Galaxy's workflows.

 As far as the toolshed goes, I don't believe I know the ins and outs
 yet, but I would be more than willing to learn if people would benefit
 from having these versions in that repository.

 2012/1/29 Jim Johnson:
>
> Pat,
>
> That sounds great.   Do one of you want to take ownership of the
> toolshed
> repository?
> At minimum, we should add developers to the list that can push changes.
>
> Thanks,
>
> JJ
>
> On 1/28/12 9:37 AM, Gillevet Patrick wrote:
>
> Jim et al
>
> Amanda has most of the scripts working now and will be putting them up
> on
> the toolshed.
> She will be in touch as soon as the scripts are validated a couple of
> times
> with different datasets.
>
> cheers...
> Pat
>
>
>
> On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:
>
>
> It is easiest to generate tools for galaxy when the applications or
> scripts
> can take arbitrarily named input files and generate output to given path
> names.
> Input directories, output directories are very convenient on the command
> line, but more of a challenge when crafting a galaxy tool.
> That said, many applications require a wrapper script to work with in
> galaxy.
> Thank you for the consistent script_info[] help/usage syntax in the
> qiime
> scripts,  which enabled me to generate a skeleton galaxy tool_config
> file
> for each qiime script.
>
> I had some time last spring to work on integrating qiime into galaxy.
> Unfortunately, I haven't had any time since to work on this.
> I put those partial results  on the Galaxy Tool Shed:
> http://toolshed.g2.bx.psu.edu/
> There's a continuing effort at George Mason University to incorporate
> qiime
> into galaxy tools, so you may want to ask them what they need.
>
>
> I started by generating galaxy tool_config files, e.g. align_seqs.xml,
>  by
> using python to get the script_info[] from the qiime script:
>
> $ cat generate_tool_config.bash
> #!/usr/bin/env bash
> python $1>  ${1%.*}.help
> cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h>
> ${1%.*}.log
>
> (I'll attach tool_template.txt )
>
> This generated skeleton tool_config .xml files that I could then edit as
> needed.
> ( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax )
>
> I originally was calling all qiime scripts from a tool wrapper:
> qiime_wrapper.py
> But, if a script can be called with any input filepaths and write its
>

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

2012-02-07 Thread Gillevet Patrick

Skype ID : patrickgillevet


On Feb 7, 2012, at 12:33 PM, Daniel McDonald wrote:

> I'll be there
> Daniel
> 
> 
> 
> On Feb 7, 2012, at 10:30, Greg Caporaso  wrote:
> 
>> OK, let's plan on a Skype call at 1pm MT/3pm ET this Thursday (9 Feb
>> 2012). I will initiate the call - my Skype ID is gregcaporaso. Please
>> let me know if you'd like to join the call, and send my your skype id.
>> 
>> Looking forward to talking about this!
>> 
>> Greg
>> 
>> 2012/2/7 Rob Knight :
>>> I can't make the call at that time (am in Dhaka) but am very enthusiastic
>>> about that effort; please keep me in the loop. I am cc:ing a couple of the
>>> people in my lab who also indicated interest in the qiime/galaxy integration
>>> effort (though Antonio won't be able to make it either, for the same
>>> reason). Thanks!
>>> 
>>> Rob
>>> 
>>> On Feb 7, 2012, at 8:48 AM, Jeffrey Long wrote:
>>> 
>>> Hello Amanda,
>>> I was just about to embark on EXACTLY this process, so I would certainly be
>>> very interested in saving myself some work.
>>> Would there be any issue (that you're aware of, of course) with using QIIME
>>> 1.4.0 instead of 1.3?
>>> 
>>> -Jeff
>>> 
>>> On Tue, Feb 7, 2012 at 2:32 AM, Florent Angly 
>>> wrote:
 
 Hi Amanda,
 I would certainly be interested in using your helpful QIIME wrappers if
 you put them on the Toolshed.
 Best,
 Florent
 
 On 06/02/12 06:22, Amanda Zuzolo wrote:
> 
> Hello, all.
> 
> I have been working on getting the Qiime scripts into Galaxy as
> mentioned before, and they are working with Qiime 1.3.0. I have edited
> the wrapper file that Jim Johnson wrote to create more flexibility,
> especially in cases where the tool looks for a specific file type
> extension (for example, a .fna file), or where the tool normally
> outputs something to the command line that is not normally picked up
> in Galaxy.
> 
> So far, I have completely finished fixing the XML files to the latest
> documentation for the entire Pick OTU process, Alpha Diversity, and
> Beta Diversity, as well as other miscellaneous functions. Currently, I
> am working on making scripts for jack-knifing functional. I determined
> that it would be easier to get individual scripts functional, rather
> than workflow scripts, since that allows the end-user to have more
> control. Additionally, the workflow scripts can easily be recreated by
> using Galaxy's workflows.
> 
> As far as the toolshed goes, I don't believe I know the ins and outs
> yet, but I would be more than willing to learn if people would benefit
> from having these versions in that repository.
> 
> 2012/1/29 Jim Johnson:
>> 
>> Pat,
>> 
>> That sounds great.   Do one of you want to take ownership of the
>> toolshed
>> repository?
>> At minimum, we should add developers to the list that can push changes.
>> 
>> Thanks,
>> 
>> JJ
>> 
>> On 1/28/12 9:37 AM, Gillevet Patrick wrote:
>> 
>> Jim et al
>> 
>> Amanda has most of the scripts working now and will be putting them up
>> on
>> the toolshed.
>> She will be in touch as soon as the scripts are validated a couple of
>> times
>> with different datasets.
>> 
>> cheers...
>> Pat
>> 
>> 
>> 
>> On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:
>> 
>> 
>> It is easiest to generate tools for galaxy when the applications or
>> scripts
>> can take arbitrarily named input files and generate output to given path
>> names.
>> Input directories, output directories are very convenient on the command
>> line, but more of a challenge when crafting a galaxy tool.
>> That said, many applications require a wrapper script to work with in
>> galaxy.
>> Thank you for the consistent script_info[] help/usage syntax in the
>> qiime
>> scripts,  which enabled me to generate a skeleton galaxy tool_config
>> file
>> for each qiime script.
>> 
>> I had some time last spring to work on integrating qiime into galaxy.
>> Unfortunately, I haven't had any time since to work on this.
>> I put those partial results  on the Galaxy Tool Shed:
>> http://toolshed.g2.bx.psu.edu/
>> There's a continuing effort at George Mason University to incorporate
>> qiime
>> into galaxy tools, so you may want to ask them what they need.
>> 
>> 
>> I started by generating galaxy tool_config files, e.g. align_seqs.xml,
>> by
>> using python to get the script_info[] from the qiime script:
>> 
>> $ cat generate_tool_config.bash
>> #!/usr/bin/env bash
>> python $1>  ${1%.*}.help
>> cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h>
>> ${1%.*}.log
>> 
>> (I'll attach tool_template.txt )
>> 
>> This generated skeleton tool_config .xml files that I could then edit as
>> n

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

2012-02-07 Thread Rob Knight

That's great news! Thanks for checking.

Rob

On Feb 7, 2012, at 10:59 AM, Amanda Zuzolo wrote:

> Rob,
> 
> I am not aware that there would be any issue, as I've verified all the
> options with the Qiime documentation that is up now (and I've
> eliminated those being deprecated).
> 
> Amanda Zuzolo
> 
> On 2/7/12, Jeffrey Long  wrote:
>> Hello Amanda,
>> I was just about to embark on EXACTLY this process, so I would certainly be
>> very interested in saving myself some work.
>> Would there be any issue (that you're aware of, of course) with using QIIME
>> 1.4.0 instead of 1.3?
>> 
>> -Jeff
>> 
>> On Tue, Feb 7, 2012 at 2:32 AM, Florent Angly
>> wrote:
>> 
>>> Hi Amanda,
>>> I would certainly be interested in using your helpful QIIME wrappers if
>>> you put them on the Toolshed.
>>> Best,
>>> Florent
>>> 
>>> On 06/02/12 06:22, Amanda Zuzolo wrote:
>>> 
 Hello, all.
 
 I have been working on getting the Qiime scripts into Galaxy as
 mentioned before, and they are working with Qiime 1.3.0. I have edited
 the wrapper file that Jim Johnson wrote to create more flexibility,
 especially in cases where the tool looks for a specific file type
 extension (for example, a .fna file), or where the tool normally
 outputs something to the command line that is not normally picked up
 in Galaxy.
 
 So far, I have completely finished fixing the XML files to the latest
 documentation for the entire Pick OTU process, Alpha Diversity, and
 Beta Diversity, as well as other miscellaneous functions. Currently, I
 am working on making scripts for jack-knifing functional. I determined
 that it would be easier to get individual scripts functional, rather
 than workflow scripts, since that allows the end-user to have more
 control. Additionally, the workflow scripts can easily be recreated by
 using Galaxy's workflows.
 
 As far as the toolshed goes, I don't believe I know the ins and outs
 yet, but I would be more than willing to learn if people would benefit
 from having these versions in that repository.
 
 2012/1/29 Jim Johnson:
 
> Pat,
> 
> That sounds great.   Do one of you want to take ownership of the
> toolshed
> repository?
> At minimum, we should add developers to the list that can push changes.
> 
> Thanks,
> 
> JJ
> 
> On 1/28/12 9:37 AM, Gillevet Patrick wrote:
> 
> Jim et al
> 
> Amanda has most of the scripts working now and will be putting them up
> on
> the toolshed.
> She will be in touch as soon as the scripts are validated a couple of
> times
> with different datasets.
> 
> cheers...
> Pat
> 
> 
> 
> On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:
> 
> 
> It is easiest to generate tools for galaxy when the applications or
> scripts
> can take arbitrarily named input files and generate output to given
> path
> names.
> Input directories, output directories are very convenient on the
> command
> line, but more of a challenge when crafting a galaxy tool.
> That said, many applications require a wrapper script to work with in
> galaxy.
> Thank you for the consistent script_info[] help/usage syntax in the
> qiime
> scripts,  which enabled me to generate a skeleton galaxy tool_config
> file
> for each qiime script.
> 
> I had some time last spring to work on integrating qiime into galaxy.
> Unfortunately, I haven't had any time since to work on this.
> I put those partial results  on the Galaxy Tool Shed:
> http://toolshed.g2.bx.psu.edu/
> There's a continuing effort at George Mason University to incorporate
> qiime
> into galaxy tools, so you may want to ask them what they need.
> 
> 
> I started by generating galaxy tool_config files, e.g. align_seqs.xml,
> by
> using python to get the script_info[] from the qiime script:
> 
> $ cat generate_tool_config.bash
> #!/usr/bin/env bash
> python $1>  ${1%.*}.help
> cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1
> -h>
> ${1%.*}.log
> 
> (I'll attach tool_template.txt )
> 
> This generated skeleton tool_config .xml files that I could then edit
> as
> needed.
> (
> http://wiki.g2.bx.psu.edu/**Admin/Tools/Tool%20Config%**20Syntax)
> 
> I originally was calling all qiime scripts from a tool wrapper:
> qiime_wrapper.py
> But, if a script can be called with any input filepaths and write its
> results to any filepaths, and only writes to STDERR when it fails, then
> you
> could call that script directly.
> 
> 
> When should you use a tool_wrapper or call the qiime script directly?
>  Many of the qiime scripts could probably be called directly,
> especially if
>

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

2012-02-07 Thread Greg Caporaso

OK, let's plan on a Skype call at 1pm MT/3pm ET this Thursday (9 Feb
2012). I will initiate the call - my Skype ID is gregcaporaso. Please
let me know if you'd like to join the call, and send my your skype id.

Looking forward to talking about this!

Greg

2012/2/7 Rob Knight :
> I can't make the call at that time (am in Dhaka) but am very enthusiastic
> about that effort; please keep me in the loop. I am cc:ing a couple of the
> people in my lab who also indicated interest in the qiime/galaxy integration
> effort (though Antonio won't be able to make it either, for the same
> reason). Thanks!
>
> Rob
>
> On Feb 7, 2012, at 8:48 AM, Jeffrey Long wrote:
>
> Hello Amanda,
> I was just about to embark on EXACTLY this process, so I would certainly be
> very interested in saving myself some work.
> Would there be any issue (that you're aware of, of course) with using QIIME
> 1.4.0 instead of 1.3?
>
> -Jeff
>
> On Tue, Feb 7, 2012 at 2:32 AM, Florent Angly 
> wrote:
>>
>> Hi Amanda,
>> I would certainly be interested in using your helpful QIIME wrappers if
>> you put them on the Toolshed.
>> Best,
>> Florent
>>
>> On 06/02/12 06:22, Amanda Zuzolo wrote:
>>>
>>> Hello, all.
>>>
>>> I have been working on getting the Qiime scripts into Galaxy as
>>> mentioned before, and they are working with Qiime 1.3.0. I have edited
>>> the wrapper file that Jim Johnson wrote to create more flexibility,
>>> especially in cases where the tool looks for a specific file type
>>> extension (for example, a .fna file), or where the tool normally
>>> outputs something to the command line that is not normally picked up
>>> in Galaxy.
>>>
>>> So far, I have completely finished fixing the XML files to the latest
>>> documentation for the entire Pick OTU process, Alpha Diversity, and
>>> Beta Diversity, as well as other miscellaneous functions. Currently, I
>>> am working on making scripts for jack-knifing functional. I determined
>>> that it would be easier to get individual scripts functional, rather
>>> than workflow scripts, since that allows the end-user to have more
>>> control. Additionally, the workflow scripts can easily be recreated by
>>> using Galaxy's workflows.
>>>
>>> As far as the toolshed goes, I don't believe I know the ins and outs
>>> yet, but I would be more than willing to learn if people would benefit
>>> from having these versions in that repository.
>>>
>>> 2012/1/29 Jim Johnson:

 Pat,

 That sounds great.   Do one of you want to take ownership of the
 toolshed
 repository?
 At minimum, we should add developers to the list that can push changes.

 Thanks,

 JJ

 On 1/28/12 9:37 AM, Gillevet Patrick wrote:

 Jim et al

 Amanda has most of the scripts working now and will be putting them up
 on
 the toolshed.
 She will be in touch as soon as the scripts are validated a couple of
 times
 with different datasets.

 cheers...
 Pat

 On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:

 It is easiest to generate tools for galaxy when the applications or
 scripts
 can take arbitrarily named input files and generate output to given path
 names.
 Input directories, output directories are very convenient on the command
 line, but more of a challenge when crafting a galaxy tool.
 That said, many applications require a wrapper script to work with in
 galaxy.
 Thank you for the consistent script_info[] help/usage syntax in the
 qiime
 scripts,  which enabled me to generate a skeleton galaxy tool_config
 file
 for each qiime script.

 I had some time last spring to work on integrating qiime into galaxy.
 Unfortunately, I haven't had any time since to work on this.
 I put those partial results  on the Galaxy Tool Shed:
 http://toolshed.g2.bx.psu.edu/
 There's a continuing effort at George Mason University to incorporate
 qiime
 into galaxy tools, so you may want to ask them what they need.

 I started by generating galaxy tool_config files, e.g. align_seqs.xml,
  by
 using python to get the script_info[] from the qiime script:

 $ cat generate_tool_config.bash
 #!/usr/bin/env bash
 python $1>  ${1%.*}.help
 cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h>
 ${1%.*}.log

 (I'll attach tool_template.txt )

 This generated skeleton tool_config .xml files that I could then edit as
 needed.
 ( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax )

 I originally was calling all qiime scripts from a tool wrapper:
 qiime_wrapper.py
 But, if a script can be called with any input filepaths and write its
 results to any filepaths, and only writes to STDERR when it fails, then
 you
 could call that script directly.

 When should you use a tool_wrapper or call the qiime script directly?
   Many

[galaxy-dev] 01/27 galaxy-dist startup problem (toolshed/pgsql related)

2012-02-07 Thread Oleksandr Moskalenko

I've upgraded my developmental Galaxy instance to the latest galaxy-dist 
release with a transplanted the drmaa.py patch from galaxy-central. All .conf 
files have been updated. However, I can't start the instance now The error 
listed below is generated. I tried enabling/disabling the toolshed configs, but 
that didn't resolve the issue. The database was upgraded from version 85 to 90 
cleanly.

Regards,

Alex

galaxy.datatypes.registry DEBUG 2012-02-07 12:32:12,345 Loaded converter: 
pbed2ldindepconvert
galaxy.tools.imp_exp DEBUG 2012-02-07 12:32:12,360 Loaded history export tool: 
__EXPORT_HISTORY__
galaxy.tools.imp_exp DEBUG 2012-02-07 12:32:12,367 Loaded history import tool: 
__IMPORT_HISTORY__
galaxy.datatypes.registry DEBUG 2012-02-07 12:32:12,383 Loaded external 
metadata tool: __SET_METADATA__




sqlalchemy.pool.QueuePool.0x...1290 WARNING 2012-02-07 12:32:12,831 Error 
closing cursor: current transaction is aborted, commands ignored until end of 
transaction block

Traceback (most recent call last):
  File "/galaxy/run/dev/lib/galaxy/web/buildapp.py", line 82, in app_factory
app = UniverseApplication( global_conf = global_conf, **kwargs )
  File "/galaxy/run/dev/lib/galaxy/app.py", line 83, in __init__
self.installed_repository_manager.load_proprietary_datatypes()
  File "/galaxy/run/dev/lib/galaxy/tool_shed/__init__.py", line 19, in 
load_proprietary_datatypes
.order_by( self.model.ToolShedRepository.table.c.id ):
  File 
"/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/query.py",
 line 1361, in __iter__
return self._execute_and_instances(context)
  File 
"/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/query.py",
 line 1364, in _execute_and_instances
result = self.session.execute(querycontext.statement, params=self._params, 
mapper=self._mapper_zero_or_none())
  File 
"/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/session.py",
 line 755, in execute
clause, params or {})
  File 
"/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
 line 824, in execute
return Connection.executors[c](self, object, multiparams, params)
  File 
"/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
 line 874, in _execute_clauseelement
return self.__execute_context(context)
  File 
"/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
 line 896, in __execute_context
self._cursor_execute(context.cursor, context.statement, 
context.parameters[0], context=context)
  File 
"/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
 line 950, in _cursor_execute
self._handle_dbapi_exception(e, statement, parameters, cursor, context)
  File 
"/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
 line 931, in _handle_dbapi_exception
raise exc.DBAPIError.instance(statement, parameters, e, 
connection_invalidated=is_disconnect)
ProgrammingError: (ProgrammingError) column tool_shed_repository.uninstalled 
does not exist
LINE 1: ...pository.deleted AS tool_shed_repository_deleted, tool_shed_...
 ^
 'SELECT tool_shed_repository.id AS tool_shed_repository_id, 
tool_shed_repository.create_time AS tool_shed_repository_create_time, 
tool_shed_repository.update_time AS tool_shed_repository_update_time, 
tool_shed_repository.tool_shed AS tool_shed_repository_tool_shed, 
tool_shed_repository.name AS tool_shed_repository_name, 
tool_shed_repository.description AS tool_shed_repository_description, 
tool_shed_repository.owner AS tool_shed_repository_owner, 
tool_shed_repository.installed_changeset_revision AS 
tool_shed_repository_installed_changeset_revision, 
tool_shed_repository.changeset_revision AS 
tool_shed_repository_changeset_revision, tool_shed_repository.metadata AS 
tool_shed_repository_metadata, tool_shed_repository.includes_datatypes AS 
tool_shed_repository_includes_datatypes, tool_shed_repository.update_available 
AS tool_shed_repository_update_available, tool_shed_repository.deleted AS 
tool_shed_repository_deleted, tool_shed_repository.uninstalled AS 
tool_shed_repository_uni!
 nstalled, tool_shed_repository.dist_to_shed AS 
tool_shed_repository_dist_to_shed \nFROM tool_shed_repository \nWHERE 
tool_shed_repository.includes_datatypes = %(includes_datatypes_1)s AND 
tool_shed_repository.deleted = %(deleted_1)s ORDER BY tool_shed_repository.id' 
{'includes_datatypes_1': True, 'deleted_1': False}
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] 01/27 galaxy-dist startup problem (toolshed/pgsql related)

2012-02-07 Thread Greg Von Kuster

Hi Alex,

If your migration scripts through 90 executed cleanly, then your 
tool_shed_repository.uninstalled column should exist as that column is created 
in the 0090 script.  What was the log of your db migration upgrades that 
occurred when your database was migrated from version 85 to 90?

Thanks,

Greg

On Feb 7, 2012, at 3:32 PM, Oleksandr Moskalenko wrote:

> I've upgraded my developmental Galaxy instance to the latest galaxy-dist 
> release with a transplanted the drmaa.py patch from galaxy-central. All .conf 
> files have been updated. However, I can't start the instance now The error 
> listed below is generated. I tried enabling/disabling the toolshed configs, 
> but that didn't resolve the issue. The database was upgraded from version 85 
> to 90 cleanly.
> 
> Regards,
> 
> Alex
> 
> galaxy.datatypes.registry DEBUG 2012-02-07 12:32:12,345 Loaded converter: 
> pbed2ldindepconvert
> galaxy.tools.imp_exp DEBUG 2012-02-07 12:32:12,360 Loaded history export 
> tool: __EXPORT_HISTORY__
> galaxy.tools.imp_exp DEBUG 2012-02-07 12:32:12,367 Loaded history import 
> tool: __IMPORT_HISTORY__
> galaxy.datatypes.registry DEBUG 2012-02-07 12:32:12,383 Loaded external 
> metadata tool: __SET_METADATA__
> 
> 
> 
> 
> sqlalchemy.pool.QueuePool.0x...1290 WARNING 2012-02-07 12:32:12,831 Error 
> closing cursor: current transaction is aborted, commands ignored until end of 
> transaction block
> 
> Traceback (most recent call last):
>  File "/galaxy/run/dev/lib/galaxy/web/buildapp.py", line 82, in app_factory
>app = UniverseApplication( global_conf = global_conf, **kwargs )
>  File "/galaxy/run/dev/lib/galaxy/app.py", line 83, in __init__
>self.installed_repository_manager.load_proprietary_datatypes()
>  File "/galaxy/run/dev/lib/galaxy/tool_shed/__init__.py", line 19, in 
> load_proprietary_datatypes
>.order_by( self.model.ToolShedRepository.table.c.id ):
>  File 
> "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/query.py",
>  line 1361, in __iter__
>return self._execute_and_instances(context)
>  File 
> "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/query.py",
>  line 1364, in _execute_and_instances
>result = self.session.execute(querycontext.statement, params=self._params, 
> mapper=self._mapper_zero_or_none())
>  File 
> "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/session.py",
>  line 755, in execute
>clause, params or {})
>  File 
> "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
>  line 824, in execute
>return Connection.executors[c](self, object, multiparams, params)
>  File 
> "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
>  line 874, in _execute_clauseelement
>return self.__execute_context(context)
>  File 
> "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
>  line 896, in __execute_context
>self._cursor_execute(context.cursor, context.statement, 
> context.parameters[0], context=context)
>  File 
> "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
>  line 950, in _cursor_execute
>self._handle_dbapi_exception(e, statement, parameters, cursor, context)
>  File 
> "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
>  line 931, in _handle_dbapi_exception
>raise exc.DBAPIError.instance(statement, parameters, e, 
> connection_invalidated=is_disconnect)
> ProgrammingError: (ProgrammingError) column tool_shed_repository.uninstalled 
> does not exist
> LINE 1: ...pository.deleted AS tool_shed_repository_deleted, tool_shed_...
> ^
> 'SELECT tool_shed_repository.id AS tool_shed_repository_id, 
> tool_shed_repository.create_time AS tool_shed_repository_create_time, 
> tool_shed_repository.update_time AS tool_shed_repository_update_time, 
> tool_shed_repository.tool_shed AS tool_shed_repository_tool_shed, 
> tool_shed_repository.name AS tool_shed_repository_name, 
> tool_shed_repository.description AS tool_shed_repository_description, 
> tool_shed_repository.owner AS tool_shed_repository_owner, 
> tool_shed_repository.installed_changeset_revision AS 
> tool_shed_repository_installed_changeset_revision, 
> tool_shed_repository.changeset_revision AS 
> tool_shed_repository_changeset_revision, tool_shed_repository.metadata AS 
> tool_shed_repository_metadata, tool_shed_repository.includes_datatypes AS 
> tool_shed_repository_includes_datatypes, 
> tool_shed_repository.update_available AS 
> tool_shed_repository_update_available, tool_shed_repository.deleted AS 
> tool_shed_repository_deleted, tool_shed_repository.uninstalled AS 
> tool_shed_repository_un!
 installed, tool_shed_repository.dist_to_shed AS 
tool_shed_repository_dist_to_shed \nFROM tool_shed_repository \nWHERE 
tool_shed_repository.includes_datatypes = %(includes_datatypes_1)s AND 
tool

Re: [galaxy-dev] 01/27 galaxy-dist startup problem (toolshed/pgsql related)

2012-02-07 Thread Oleksandr Moskalenko

$ sh manage_db.sh -c ./dev_universe_wsgi.runner.ini upgrade
85 -> 86... 
Migration script to add the metadata, update_available and includes_datatypes 
columns to the tool_shed_repository table.
done
86 -> 87... 
Migration script to create the tool_id_guid_map table.
done
87 -> 88...
Migration script to add the installed_changeset_revision column to the 
tool_shed_repository table.
Updated the installed_changeset_revision column for  0  rows in the 
tool_shed_repository table.  
done
88 -> 89... 
Migration script to add 'object_store_id' column to various tables
done
89 -> 90... 
Migration script to add the uninstalled and dist_to_shed columns to the 
tool_shed_repository table.
done

Looking around in psql seems to confirm a clean update:

galaxydev-> \d tool_shed_repository
  Table 
"public.tool_shed_repository"
Column|Type |   
  Modifiers 
--+-+---
 id   | integer | not null default 
nextval('tool_shed_repository_id_seq'::regclass)
 create_time  | timestamp without time zone | 
 update_time  | timestamp without time zone | 
 tool_shed| character varying(255)  | 
 name | character varying(255)  | 
 description  | text| 
 owner| character varying(255)  | 
 changeset_revision   | character varying(255)  | 
 deleted  | boolean | 
 metadata | bytea   | 
 includes_datatypes   | boolean | 
 update_available | boolean | 
 installed_changeset_revision | character varying(255)  | 
 uninstalled  | boolean | 
 dist_to_shed | boolean | 
Indexes:
"tool_shed_repository_pkey" PRIMARY KEY, btree (id)
"ix_tool_shed_repository_changeset_revision" btree (changeset_revision)
"ix_tool_shed_repository_deleted" btree (deleted)
"ix_tool_shed_repository_name" btree (name)
"ix_tool_shed_repository_owner" btree (owner)
"ix_tool_shed_repository_tool_shed" btree (tool_shed)


The table is empty though:

galaxydev=> select * from tool_shed_repository;
 id | create_time | update_time | tool_shed | name | description | owner | 
changeset_revision | deleted | metadata | includes_datatypes | 
update_available | installed_changeset_revision | uninstalled | dist_to_shed 
+-+-+---+--+-+---++-+--++-
-+--+-+--
(0 rows)

On Feb 7, 2012, at 3:42 PM, Greg Von Kuster wrote:

> Hi Alex,
> 
> If your migration scripts through 90 executed cleanly, then your 
> tool_shed_repository.uninstalled column should exist as that column is 
> created in the 0090 script.  What was the log of your db migration upgrades 
> that occurred when your database was migrated from version 85 to 90?
> 
> Thanks,
> 
> Greg
> 
> On Feb 7, 2012, at 3:32 PM, Oleksandr Moskalenko wrote:
> 
>> I've upgraded my developmental Galaxy instance to the latest galaxy-dist 
>> release with a transplanted the drmaa.py patch from galaxy-central. All 
>> .conf files have been updated. However, I can't start the instance now The 
>> error listed below is generated. I tried enabling/disabling the toolshed 
>> configs, but that didn't resolve the issue. The database was upgraded from 
>> version 85 to 90 cleanly.
>> 
>> Regards,
>> 
>> Alex
>> 
>> galaxy.datatypes.registry DEBUG 2012-02-07 12:32:12,345 Loaded converter: 
>> pbed2ldindepconvert
>> galaxy.tools.imp_exp DEBUG 2012-02-07 12:32:12,360 Loaded history export 
>> tool: __EXPORT_HISTORY__
>> galaxy.tools.imp_exp DEBUG 2012-02-07 12:32:12,367 Loaded history import 
>> tool: __IMPORT_HISTORY__
>> galaxy.datatypes.registry DEBUG 2012-02-07 12:32:12,383 Loaded external 
>> metadata tool: __SET_METADATA__
>> 
>> 
>> 
>> 
>> sqlalchemy.pool.QueuePool.0x...1290 WARNING 2012-02-07 12:32:12,831 Error 
>> closing cursor: current transaction is aborted, commands ignored until end 
>> of transaction block
>> 
>> Traceback (most recent call last):
>> File "/galaxy/run/dev/lib/galaxy/web/buildapp.py", line 82, in app_factory
>>   app = UniverseApplication( global_conf = global_conf, **kwargs )
>> File "/galaxy/run/dev/lib/galaxy/app.py", line 83, in __init__
>>   self.installed_repository_manager.load_proprietary_datatypes()
>> File "/galaxy/run/dev/lib/galaxy/tool_shed/__init__.py", line 19, in 
>> load_proprietary_datatypes
>>   .order_by( self.model.ToolShedRepository

[galaxy-dev] generate dynamic select list based on other input dataset

2012-02-07 Thread Holger Klein

Dear all,

I have a working module which generates wig files for genomic annotation
from a single column of a bigger input data matrix (Input A). In the
current state, the user has to input the column name (Input B) from
which to calculate the values in the wig file.

Now I'd like to modify the xml in such a way, that depending on the
input dataset (Input A) a dynamic list for Input B is generated.

I found Hans-Rudolf Hotz' hints from some time ago on this list and
thought that the following would be a good start:





getInputBOptionsFromInputA.py contains a single function
def getInputBOptions($InputA):
## parse Input A
## create list InputBOptions
return(InputBOptions)

Using this approach I get an invalid syntax message when trying to even
open the module - in any case I have the feeling that something is still
missing here.
Did anybody solve a similar problem already and could give me a hint on
how to solve that?

Cheers,
Holger




-- 
Dr. Holger Klein
Core Facility Bioinformatics
Institute of Molecular Biology gGmbH (IMB)
http://www.imb-mainz.de/
Tel: +49(6131) 39 21511
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] 01/27 galaxy-dist startup problem (toolshed/pgsql related)

2012-02-07 Thread Greg Von Kuster

Alex, this is strange, so it'll be tricky to figure out what's causing the 
problem.  I can't think of anything that would cause this unless 2 different 
databases are somehow being used.   I assume your database configured in your 
file dev_universe_wsgi.runner.ini is the same as the database used by your 
Galaxy instance.  Can you make sure of this?  

When you start up your Galaxy instance, your paster log should include 
something like the following - this happens right after loading the Galaxy 
datatypes, and before any tools are loaded.  Can you make sure what database 
version is displayed when you start up?  Yours should be at database version 90.

galaxy.tool_shed.tool_shed_registry DEBUG 2012-02-07 15:58:56,069 Loaded 
reference to tool shed: Bx tool shed
galaxy.tool_shed.tool_shed_registry DEBUG 2012-02-07 15:58:56,069 Loaded 
reference to tool shed: Galaxy main tool shed
galaxy.tool_shed.tool_shed_registry DEBUG 2012-02-07 15:58:56,069 Loaded 
reference to tool shed: Galaxy test tool shed
galaxy.model.migrate.check DEBUG 2012-02-07 15:58:56,111 psycopg2 egg 
successfully loaded for postgres dialect
galaxy.model.migrate.check INFO 2012-02-07 15:58:56,273 At database version 91



On Feb 7, 2012, at 3:55 PM, Oleksandr Moskalenko wrote:

> $ sh manage_db.sh -c ./dev_universe_wsgi.runner.ini upgrade
> 85 -> 86... 
> Migration script to add the metadata, update_available and includes_datatypes 
> columns to the tool_shed_repository table.
> done
> 86 -> 87... 
> Migration script to create the tool_id_guid_map table.
> done
> 87 -> 88...
> Migration script to add the installed_changeset_revision column to the 
> tool_shed_repository table.
> Updated the installed_changeset_revision column for  0  rows in the 
> tool_shed_repository table.  
> done
> 88 -> 89... 
> Migration script to add 'object_store_id' column to various tables
> done
> 89 -> 90... 
> Migration script to add the uninstalled and dist_to_shed columns to the 
> tool_shed_repository table.
> done
> 
> Looking around in psql seems to confirm a clean update:
> 
> galaxydev-> \d tool_shed_repository
>  Table 
> "public.tool_shed_repository"
>Column|Type |  
>Modifiers 
> --+-+---
> id   | integer | not null default 
> nextval('tool_shed_repository_id_seq'::regclass)
> create_time  | timestamp without time zone | 
> update_time  | timestamp without time zone | 
> tool_shed| character varying(255)  | 
> name | character varying(255)  | 
> description  | text| 
> owner| character varying(255)  | 
> changeset_revision   | character varying(255)  | 
> deleted  | boolean | 
> metadata | bytea   | 
> includes_datatypes   | boolean | 
> update_available | boolean | 
> installed_changeset_revision | character varying(255)  | 
> uninstalled  | boolean | 
> dist_to_shed | boolean | 
> Indexes:
>"tool_shed_repository_pkey" PRIMARY KEY, btree (id)
>"ix_tool_shed_repository_changeset_revision" btree (changeset_revision)
>"ix_tool_shed_repository_deleted" btree (deleted)
>"ix_tool_shed_repository_name" btree (name)
>"ix_tool_shed_repository_owner" btree (owner)
>"ix_tool_shed_repository_tool_shed" btree (tool_shed)
> 
> 
> The table is empty though:
> 
> galaxydev=> select * from tool_shed_repository;
> id | create_time | update_time | tool_shed | name | description | owner | 
> changeset_revision | deleted | metadata | includes_datatypes | 
> update_available | installed_changeset_revision | uninstalled | dist_to_shed 
> +-+-+---+--+-+---++-+--++-
> -+--+-+--
> (0 rows)
> 
> On Feb 7, 2012, at 3:42 PM, Greg Von Kuster wrote:
> 
>> Hi Alex,
>> 
>> If your migration scripts through 90 executed cleanly, then your 
>> tool_shed_repository.uninstalled column should exist as that column is 
>> created in the 0090 script.  What was the log of your db migration upgrades 
>> that occurred when your database was migrated from version 85 to 90?
>> 
>> Thanks,
>> 
>> Greg
>> 
>> On Feb 7, 2012, at 3:32 PM, Oleksandr Moskalenko wrote:
>> 
>>> I've upgraded my developmental Galaxy instance to the latest galaxy-dist 
>>> release with a transplanted the drmaa.py patch from galaxy-central.

Re: [galaxy-dev] 01/27 galaxy-dist startup problem (toolshed/pgsql related)

2012-02-07 Thread Oleksandr Moskalenko

Greg,

Startup log:

galaxy.tool_shed.tool_shed_registry DEBUG 2012-02-07 16:13:29,613 Loading 
references to tool sheds from tool_sheds_conf.xml
galaxy.tool_shed.tool_shed_registry DEBUG 2012-02-07 16:13:29,614 Loaded 
reference to tool shed: Galaxy main tool shed
galaxy.tool_shed.tool_shed_registry DEBUG 2012-02-07 16:13:29,614 Loaded 
reference to tool shed: Galaxy test tool shed
galaxy.model.migrate.check DEBUG 2012-02-07 16:13:29,852 psycopg2 egg 
successfully loaded for postgres dialect
galaxy.model.migrate.check INFO 2012-02-07 16:13:29,981 At database version 90
galaxy.model.custom_types DEBUG 2012-02-07 16:13:29,986 psycopg2 egg 
successfully loaded for postgres dialect


I forced a db downgrade, then ran the db upgrade again. Then,  the instance 
came up. It's weird, but the issue is resolved I guess.

Thanks for looking into it.

Cheers,

Alex

On Feb 7, 2012, at 4:10 PM, Greg Von Kuster wrote:

> Alex, this is strange, so it'll be tricky to figure out what's causing the 
> problem.  I can't think of anything that would cause this unless 2 different 
> databases are somehow being used.   I assume your database configured in your 
> file dev_universe_wsgi.runner.ini is the same as the database used by your 
> Galaxy instance.  Can you make sure of this?  
> 
> When you start up your Galaxy instance, your paster log should include 
> something like the following - this happens right after loading the Galaxy 
> datatypes, and before any tools are loaded.  Can you make sure what database 
> version is displayed when you start up?  Yours should be at database version 
> 90.
> 
> galaxy.tool_shed.tool_shed_registry DEBUG 2012-02-07 15:58:56,069 Loaded 
> reference to tool shed: Bx tool shed
> galaxy.tool_shed.tool_shed_registry DEBUG 2012-02-07 15:58:56,069 Loaded 
> reference to tool shed: Galaxy main tool shed
> galaxy.tool_shed.tool_shed_registry DEBUG 2012-02-07 15:58:56,069 Loaded 
> reference to tool shed: Galaxy test tool shed
> galaxy.model.migrate.check DEBUG 2012-02-07 15:58:56,111 psycopg2 egg 
> successfully loaded for postgres dialect
> galaxy.model.migrate.check INFO 2012-02-07 15:58:56,273 At database version 91
> 
> 
> 
> On Feb 7, 2012, at 3:55 PM, Oleksandr Moskalenko wrote:
> 
>> $ sh manage_db.sh -c ./dev_universe_wsgi.runner.ini upgrade
>> 85 -> 86... 
>> Migration script to add the metadata, update_available and 
>> includes_datatypes columns to the tool_shed_repository table.
>> done
>> 86 -> 87... 
>> Migration script to create the tool_id_guid_map table.
>> done
>> 87 -> 88...
>> Migration script to add the installed_changeset_revision column to the 
>> tool_shed_repository table.
>> Updated the installed_changeset_revision column for  0  rows in the 
>> tool_shed_repository table.  
>> done
>> 88 -> 89... 
>> Migration script to add 'object_store_id' column to various tables
>> done
>> 89 -> 90... 
>> Migration script to add the uninstalled and dist_to_shed columns to the 
>> tool_shed_repository table.
>> done
>> 
>> Looking around in psql seems to confirm a clean update:
>> 
>> galaxydev-> \d tool_shed_repository
>> Table 
>> "public.tool_shed_repository"
>>   Column|Type |  
>>Modifiers 
>> --+-+---
>> id   | integer | not null 
>> default nextval('tool_shed_repository_id_seq'::regclass)
>> create_time  | timestamp without time zone | 
>> update_time  | timestamp without time zone | 
>> tool_shed| character varying(255)  | 
>> name | character varying(255)  | 
>> description  | text| 
>> owner| character varying(255)  | 
>> changeset_revision   | character varying(255)  | 
>> deleted  | boolean | 
>> metadata | bytea   | 
>> includes_datatypes   | boolean | 
>> update_available | boolean | 
>> installed_changeset_revision | character varying(255)  | 
>> uninstalled  | boolean | 
>> dist_to_shed | boolean | 
>> Indexes:
>>   "tool_shed_repository_pkey" PRIMARY KEY, btree (id)
>>   "ix_tool_shed_repository_changeset_revision" btree (changeset_revision)
>>   "ix_tool_shed_repository_deleted" btree (deleted)
>>   "ix_tool_shed_repository_name" btree (name)
>>   "ix_tool_shed_repository_owner" btree (owner)
>>   "ix_tool_shed_repository_tool_shed" btree (tool_shed)
>> 
>> 
>> The table is empty though:
>> 
>> galaxydev=> select * from tool_shed_repository;
>> id | create_time | update_time | tool_shed |

Re: [galaxy-dev] Problem with latest database upgrade

2012-02-07 Thread Anthonius deBoer

Hi Greg,I tried to upgrade the GALAXY-CENTRAL version of my installation to fix the 90-91 upgrade bug, but I don't seem to see revision 271ccd09bc23 there...Is this issue fixed in galaxy central or only in the main installation?ThanksThonOn Feb 07, 2012, at 08:11 AM, Greg Von Kuster  wrote:Hi Thon,The fix for this issue in now available in the central repo in change set  revision 271ccd09bc23.  Even though you have a working version of Galaxy, you should update your instance to at least this revision, and then do the following as soon as possible.%sh manage_db.sh downgrade 9091 -> 90... done%sh manage_db.sh upgrade90 -> 91... The problem was caused because you are running postgres version 9 and the original version of my migration script did not handle hex bytea for that version of postgres.Thanks for finding this problem!GregOn Feb 6, 2012, at 10:08 PM, Anthonius deBoer wrote:Oh, and whatever you did in the little patch, it worked...At least I have a working version of galaxy again and the database seems to be all intact..Thanks!ThonOn Feb 06, 2012, at 03:52 PM, Greg Von Kuster  wrote:Can you replace lines 75 - 86 of the file ~/lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py with the following for loop and send me your paster log?Thanks!    for row in result:        tool_shed_repository_id = row[0]        try:            repository_metadata = from_json_string( str( row[1] ) )            # Create a new row in the tool table for each tool included in repository.  We will NOT            # handle tool_version_associaions because we do not have the information we need to do so.            tools = repository_metadata.get( 'tools', [] )            for tool_dict in tools:                cmd = "INSERT INTO tool_version VALUES (%s, %s, %s, '%s', %s)" % \                    ( nextval( 'tool_version' ), localtimestamp(), localtimestamp(), tool_dict[ 'guid' ], tool_shed_repository_id )                db_session.execute( cmd )                count += 1        except Exception, e:            print "Repository metadata: ", str( row[1] )            print "Exception setting tool versions for tools in tool_shed_repository with id: ", str( tool_shed_repository_id )            print str( e )The lines of code you are replacing look like this:    for row in result:        if row[1]:            tool_shed_repository_id = row[0]            repository_metadata = from_json_string( str( row[1] ) )            # Create a new row in the tool table for each tool included in repository.  We will NOT            # handle tool_version_associaions because we do not have the information we need to do so.            tools = repository_metadata.get( 'tools', [] )            for tool_dict in tools:                cmd = "INSERT INTO tool_version VALUES (%s, %s, %s, '%s', %s)" % \                    ( nextval( 'tool_version' ), localtimestamp(), localtimestamp(), tool_dict[ 'guid' ], tool_shed_repository_id )                db_session.execute( cmd )                count += 1On Feb 6, 2012, at 5:23 PM, Anthonius deBoer wrote:I manually dropped the tool_version and tool_association table in the database and I am left with the JSON error as shown below:90 -> 91...Migration script to create the tool_version and tool_version_association tables and drop the tool_id_guid_map table.Traceback (most recent call last):  File "./scripts/manage_db.py", line 63, in     main( repository=repo, url="" )  File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py", line 150, in main    ret = command_func(**kwargs)  File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 221, in upgrade    return _migrate(url, repository, version, upgrade=True, err=err, **opts)  File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 349, in _migrate    schema.runchange(ver, change, changeset.step)  File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py", line 184, in runchange    change.run(self.engine, step)  File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py", line 101, in run    func()  File "lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py", line 78, in upgrade    repository_metadata = from_json_string( str( row[1] ) )  File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/__init__.py", line 384, in loads  File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/decoder.py", line 402, in decode  File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/decoder.py", line 420, in raw_decodesimplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)On Feb 06, 2012, at 02:08 PM, Anthonius deBoe

Re: [galaxy-dev] Problem with latest database upgrade

2012-02-07 Thread Anthonius deBoer

Never mind...I plucked the edit from a pristine instance of galaxy-central...It's fixed for me nowOn Feb 07, 2012, at 02:42 PM, Anthonius deBoer  wrote:Hi Greg,I tried to upgrade the GALAXY-CENTRAL version of my installation to fix the 90-91 upgrade bug, but I don't seem to see revision 271ccd09bc23 there...Is this issue fixed in galaxy central or only in the main installation?ThanksThonOn Feb 07, 2012, at 08:11 AM, Greg Von Kuster  wrote:Hi Thon,The fix for this issue in now available in the central repo in change set  revision 271ccd09bc23.  Even though you have a working version of Galaxy, you should update your instance to at least this revision, and then do the following as soon as possible.%sh manage_db.sh downgrade 9091 -> 90... done%sh manage_db.sh upgrade90 -> 91... The problem was caused because you are running postgres version 9 and the original version of my migration script did not handle hex bytea for that version of postgres.Thanks for finding this problem!GregOn Feb 6, 2012, at 10:08 PM, Anthonius deBoer wrote:Oh, and whatever you did in the little patch, it worked...At least I have a working version of galaxy again and the database seems to be all intact..Thanks!ThonOn Feb 06, 2012, at 03:52 PM, Greg Von Kuster  wrote:Can you replace lines 75 - 86 of the file ~/lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py with the following for loop and send me your paster log?Thanks!    for row in result:        tool_shed_repository_id = row[0]        try:            repository_metadata = from_json_string( str( row[1] ) )            # Create a new row in the tool table for each tool included in repository.  We will NOT            # handle tool_version_associaions because we do not have the information we need to do so.            tools = repository_metadata.get( 'tools', [] )            for tool_dict in tools:                cmd = "INSERT INTO tool_version VALUES (%s, %s, %s, '%s', %s)" % \                    ( nextval( 'tool_version' ), localtimestamp(), localtimestamp(), tool_dict[ 'guid' ], tool_shed_repository_id )                db_session.execute( cmd )                count += 1        except Exception, e:            print "Repository metadata: ", str( row[1] )            print "Exception setting tool versions for tools in tool_shed_repository with id: ", str( tool_shed_repository_id )            print str( e )The lines of code you are replacing look like this:    for row in result:        if row[1]:            tool_shed_repository_id = row[0]            repository_metadata = from_json_string( str( row[1] ) )            # Create a new row in the tool table for each tool included in repository.  We will NOT            # handle tool_version_associaions because we do not have the information we need to do so.            tools = repository_metadata.get( 'tools', [] )            for tool_dict in tools:                cmd = "INSERT INTO tool_version VALUES (%s, %s, %s, '%s', %s)" % \                    ( nextval( 'tool_version' ), localtimestamp(), localtimestamp(), tool_dict[ 'guid' ], tool_shed_repository_id )                db_session.execute( cmd )                count += 1On Feb 6, 2012, at 5:23 PM, Anthonius deBoer wrote:I manually dropped the tool_version and tool_association table in the database and I am left with the JSON error as shown below:90 -> 91...Migration script to create the tool_version and tool_version_association tables and drop the tool_id_guid_map table.Traceback (most recent call last):  File "./scripts/manage_db.py", line 63, in     main( repository=repo, url="" )  File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py", line 150, in main    ret = command_func(**kwargs)  File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 221, in upgrade    return _migrate(url, repository, version, upgrade=True, err=err, **opts)  File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 349, in _migrate    schema.runchange(ver, change, changeset.step)  File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py", line 184, in runchange    change.run(self.engine, step)  File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py", line 101, in run    func()  File "lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py", line 78, in upgrade    repository_metadata = from_json_string( str( row[1] ) )  File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/__init__.py", line 384, in loads  File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/decoder.py", line 402, in decode  File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/decoder.py", line 420,

Re: [galaxy-dev] Problem with latest database upgrade

2012-02-07 Thread Greg Von Kuster

Thon,

Since you are using the features enabling communication between your Galaxy 
instance and the tool shed, I recommend that you keep your Galaxy instance code 
base current with the tip from our Galaxy central repo, at least until the tool 
shed features become more stable.  I've committed several new features and 
fixes today related to tool shed repository installs, so you should pick them 
up.  Of course, you run the risk of getting updates that may not be thoroughly 
tested.

Greg

On Feb 7, 2012, at 6:00 PM, Anthonius deBoer wrote:

> Never mind...I plucked the edit from a pristine instance of galaxy-central...
> 
> It's fixed for me now
> 
> On Feb 07, 2012, at 02:42 PM, Anthonius deBoer  wrote:
> 
>> Hi Greg,
>> 
>> I tried to upgrade the GALAXY-CENTRAL version of my installation to fix the 
>> 90-91 upgrade bug, but I don't seem to see revision 271ccd09bc23 there...Is 
>> this issue fixed in galaxy central or only in the main installation?
>> 
>> Thanks
>> 
>> Thon
>> 
>> On Feb 07, 2012, at 08:11 AM, Greg Von Kuster  wrote:
>> 
>>> Hi Thon,
>>> 
>>> The fix for this issue in now available in the central repo in change set  
>>> revision 271ccd09bc23.  Even though you have a working version of Galaxy, 
>>> you should update your instance to at least this revision, and then do the 
>>> following as soon as possible.
>>> 
>>> %sh manage_db.sh downgrade 90
>>> 91 -> 90... 
>>> done
>>> 
>>> %sh manage_db.sh upgrade
>>> 90 -> 91... 
>>> 
>>> The problem was caused because you are running postgres version 9 and the 
>>> original version of my migration script did not handle hex bytea for that 
>>> version of postgres.
>>> 
>>> Thanks for finding this problem!
>>> 
>>> Greg
>>> 
>>> On Feb 6, 2012, at 10:08 PM, Anthonius deBoer wrote:
>>> 
 Oh, and whatever you did in the little patch, it worked...At least I have 
 a working version of galaxy again and the database seems to be all intact..
 
 Thanks!
 
 Thon
 
 On Feb 06, 2012, at 03:52 PM, Greg Von Kuster  wrote:
 
> Can you replace lines 75 - 86 of the file 
> ~/lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py with 
> the following for loop and send me your paster log?
> 
> Thanks!
> 
> for row in result:
> tool_shed_repository_id = row[0]
> try:
> repository_metadata = from_json_string( str( row[1] ) )
> # Create a new row in the tool table for each tool included 
> in repository.  We will NOT
> # handle tool_version_associaions because we do not have the 
> information we need to do so.
> tools = repository_metadata.get( 'tools', [] )
> for tool_dict in tools:
> cmd = "INSERT INTO tool_version VALUES (%s, %s, %s, '%s', 
> %s)" % \
> ( nextval( 'tool_version' ), localtimestamp(), 
> localtimestamp(), tool_dict[ 'guid' ], tool_shed_repository_id )
> db_session.execute( cmd )
> count += 1
> except Exception, e:
> print "Repository metadata: ", str( row[1] )
> print "Exception setting tool versions for tools in 
> tool_shed_repository with id: ", str( tool_shed_repository_id )
> print str( e )
> 
> The lines of code you are replacing look like this:
> 
> for row in result:
> if row[1]:
> tool_shed_repository_id = row[0]
> repository_metadata = from_json_string( str( row[1] ) )
> # Create a new row in the tool table for each tool included 
> in repository.  We will NOT
> # handle tool_version_associaions because we do not have the 
> information we need to do so.
> tools = repository_metadata.get( 'tools', [] )
> for tool_dict in tools:
> cmd = "INSERT INTO tool_version VALUES (%s, %s, %s, '%s', 
> %s)" % \
> ( nextval( 'tool_version' ), localtimestamp(), 
> localtimestamp(), tool_dict[ 'guid' ], tool_shed_repository_id )
> db_session.execute( cmd )
> count += 1
> 
> 
> On Feb 6, 2012, at 5:23 PM, Anthonius deBoer wrote:
> 
>> I manually dropped the tool_version and tool_association table in the 
>> database and I am left with the JSON error as shown below:
>> 
>> 90 -> 91...
>> 
>> Migration script to create the tool_version and tool_version_association 
>> tables and drop the tool_id_guid_map table.
>> 
>> Traceback (most recent call last):
>>   File "./scripts/manage_db.py", line 63, in 
>> main( repository=repo, url=db_url )
>>   File 
>> "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py",
>>  line 150, in main
>> ret = command_func(**kwargs)

Re: [galaxy-dev] Problems with custom data types and sniffers in shed tools

2012-02-07 Thread Ira Cooke

Hi Greg, 

Thanks for that fix ... I'll check it out.

I think the issue of sniff order is a pretty important one for shed tools that 
require proprietary datatypes.  The problem is that galaxy's default sniffers 
include some extremely generic sniffers (eg text,xml) which will catch pretty 
much anything so it's hard to imagine how a tool writer will ever have their 
sniffers used.

Since the potential for conflicts with galaxy's defaults is an issue how about 
changing sniffer behaviour so that datatypes which subclass always have 
priority over their parent class ... then among those subclasses the order in 
which sniffers are listed in datatypes_conf is used as the second priority.  I 
imagine that this would require that the sniff order code be changed so that 
all datatypes and sniffers are first loaded .. then reshuffled to get the 
correct order according to the class hierarchy.  I believe this option makes 
sense since a subclass of a datatype should always override its parent .. and I 
think it would also avoid the potential for conflicts with galaxy's defaults. 

What do you think? Is that a system that would work?

Cheers
ira




On 08/02/2012, at 3:25 AM, Greg Von Kuster wrote:

> Hello Ira,
> 
> Very sorry for the back-and-forth on this.  The behavior you encountered was 
> accurate - the environment in which I was testing was not pristine.  I have 
> committed a fix for this issue in revision b4ba8b20d78d, which is now 
> available from our central repo.  
> 
> I started looking at changing things so that proprietary sniffers are loaded 
> before the sniffers defined in the Galaxy distribution, and determined that 
> this will be a bit non-trivial.  Problems arise due to potential conflicts 
> between a proprietary datatype that has the same extension as a datatype 
> defined in the Galaxy distribution.  The approach I've taken to deal with 
> this is that datatypes in the Galaxy distribution get loaded first, and then 
> any conflicting proprietary datatypes will be ignored.  Loading proprietary 
> datatypes first will not allow for this rule.  
> 
> I'll gladly listen to advice from the community on this, but in the meantime, 
> proprietary datatypes and sniffers are loaded after those in the distribution.
> 
> Thanks for your patience and help on this issue.
> 
> Greg
> 
> On Feb 6, 2012, at 11:37 PM, Ira Cooke wrote:
> 
>> Hi Greg, 
>> 
>> As far as I know there is nothing in my local setup that affects this.  As I 
>> test I did the following; 
>> 
>> Checked out a clean copy of galaxy-central 
>> Edited my universe_wsgi.ini as follows;
>>  - Changed the port to 8300
>>  - Added an admin user
>>  - Uncommented this line
>>  tool_config_file = tool_conf.xml,shed_tool_conf.xml
>> 
>> Edited shed_tool_conf.xml to point to ../shed_tools_galaxy_central (which is 
>> an empty directory)
>> 
>> Started up galaxy 
>> ./run.sh --reload
>> 
>> I then went straight to admin -> "Search and browse tool sheds" and selected 
>> the main galaxy toolshed. 
>> I then selected the gmap tool and told it to install.  My paster log is at 
>> the bottom of this mail.
>> 
>> My system is Mac OSX 10.7.2 and I'm running python 2.7.1
>> 
>> I then went to the relevant bit of code and inserted some print statements.  
>> I found that; 
>> 
>> On line 191 in registry.py there is;
>> 
>> module = __import__( datatype_module )
>> 
>> and the value of datatype_module is galaxy.datatypes.gmap
>> 
>> this line seems to be causing the exception ... I'm not a python programmer 
>> so I don't really know how module importing works ... could this be related 
>> to the version of python I'm running?  
>> alternatively should the "imported_modules" variable be used here .. since 
>> wouldn't galaxy.datatypes.gmap suggest that the module needs to be located 
>> inside the main galaxy lib rather than in the shed_tools.
>> 
>> Strange that I'm the only one who gets these errors?  Any ideas what it 
>> could be?
>> 
>> Ira
>> 
>> 
>> -- paster log --
>> 
>> Cloning http://toolshed.g2.bx.psu.edu/repos/jjohnson/gmap
>> destination directory: gmap
>> requesting all changes
>> adding changesets
>> adding manifests
>> adding file changes
>> added 11 changesets with 32 changes to 17 files
>> updating to branch default
>> resolving manifests
>> getting README
>> getting gmap.xml
>> getting gmap_build.xml
>> getting gsnap.xml
>> getting iit_store.xml
>> getting lib/galaxy/datatypes/gmap.py
>> getting snpindex.xml
>> getting tool-data/datatypes_conf.xml
>> getting tool-data/gmap_indices.loc.sample
>> 9 files updated, 0 files merged, 0 files removed, 0 files unresolved
>> galaxy.util.shed_util DEBUG 2012-02-07 14:25:35,100 Updating cloned 
>> repository to revision "93911bac43da"
>> resolving manifests
>> 0 files updated, 0 files merged, 0 files removed, 0 files unresolved
>> docutils WARNING 2012-02-07 14:25:35,512 :10: (WARNING/2) Explicit 
>> markup ends without a blank line; unexpected unindent.
>> 
>> galaxy.util.shed_

Re: [galaxy-dev] Problems with custom data types and sniffers in shed tools

2012-02-07 Thread Greg Von Kuster

Ira,

Thanks for the feedback - I'll come up with something that allows for 
proprietary sniffers to be loaded first and let you know when it's available.  
It shouldn't take me too long.  I will probably continue to keep the rule of 
ignoring conflicts in proprietary datatypes, so sniffers for conflicting 
datatypes will probably be ignored as well.  However, hopefully conflicts will 
not exist, or if one is found, it will get fixed.

Greg

On Feb 7, 2012, at 6:16 PM, Ira Cooke wrote:

> Hi Greg, 
> 
> Thanks for that fix ... I'll check it out.
> 
> I think the issue of sniff order is a pretty important one for shed tools 
> that require proprietary datatypes.  The problem is that galaxy's default 
> sniffers include some extremely generic sniffers (eg text,xml) which will 
> catch pretty much anything so it's hard to imagine how a tool writer will 
> ever have their sniffers used.
> 
> Since the potential for conflicts with galaxy's defaults is an issue how 
> about changing sniffer behaviour so that datatypes which subclass always have 
> priority over their parent class ... then among those subclasses the order in 
> which sniffers are listed in datatypes_conf is used as the second priority.  
> I imagine that this would require that the sniff order code be changed so 
> that all datatypes and sniffers are first loaded .. then reshuffled to get 
> the correct order according to the class hierarchy.  I believe this option 
> makes sense since a subclass of a datatype should always override its parent 
> .. and I think it would also avoid the potential for conflicts with galaxy's 
> defaults. 
> 
> What do you think? Is that a system that would work?
> 
> Cheers
> ira
> 
> 
> 
> 
> On 08/02/2012, at 3:25 AM, Greg Von Kuster wrote:
> 
>> Hello Ira,
>> 
>> Very sorry for the back-and-forth on this.  The behavior you encountered was 
>> accurate - the environment in which I was testing was not pristine.  I have 
>> committed a fix for this issue in revision b4ba8b20d78d, which is now 
>> available from our central repo.  
>> 
>> I started looking at changing things so that proprietary sniffers are loaded 
>> before the sniffers defined in the Galaxy distribution, and determined that 
>> this will be a bit non-trivial.  Problems arise due to potential conflicts 
>> between a proprietary datatype that has the same extension as a datatype 
>> defined in the Galaxy distribution.  The approach I've taken to deal with 
>> this is that datatypes in the Galaxy distribution get loaded first, and then 
>> any conflicting proprietary datatypes will be ignored.  Loading proprietary 
>> datatypes first will not allow for this rule.  
>> 
>> I'll gladly listen to advice from the community on this, but in the 
>> meantime, proprietary datatypes and sniffers are loaded after those in the 
>> distribution.
>> 
>> Thanks for your patience and help on this issue.
>> 
>> Greg
>> 
>> On Feb 6, 2012, at 11:37 PM, Ira Cooke wrote:
>> 
>>> Hi Greg, 
>>> 
>>> As far as I know there is nothing in my local setup that affects this.  As 
>>> I test I did the following; 
>>> 
>>> Checked out a clean copy of galaxy-central 
>>> Edited my universe_wsgi.ini as follows;
>>> - Changed the port to 8300
>>> - Added an admin user
>>> - Uncommented this line
>>> tool_config_file = tool_conf.xml,shed_tool_conf.xml
>>> 
>>> Edited shed_tool_conf.xml to point to ../shed_tools_galaxy_central (which 
>>> is an empty directory)
>>> 
>>> Started up galaxy 
>>> ./run.sh --reload
>>> 
>>> I then went straight to admin -> "Search and browse tool sheds" and 
>>> selected the main galaxy toolshed. 
>>> I then selected the gmap tool and told it to install.  My paster log is at 
>>> the bottom of this mail.
>>> 
>>> My system is Mac OSX 10.7.2 and I'm running python 2.7.1
>>> 
>>> I then went to the relevant bit of code and inserted some print statements. 
>>>  I found that; 
>>> 
>>> On line 191 in registry.py there is;
>>> 
>>> module = __import__( datatype_module )
>>> 
>>> and the value of datatype_module is galaxy.datatypes.gmap
>>> 
>>> this line seems to be causing the exception ... I'm not a python programmer 
>>> so I don't really know how module importing works ... could this be related 
>>> to the version of python I'm running?  
>>> alternatively should the "imported_modules" variable be used here .. since 
>>> wouldn't galaxy.datatypes.gmap suggest that the module needs to be located 
>>> inside the main galaxy lib rather than in the shed_tools.
>>> 
>>> Strange that I'm the only one who gets these errors?  Any ideas what it 
>>> could be?
>>> 
>>> Ira
>>> 
>>> 
>>> -- paster log --
>>> 
>>> Cloning http://toolshed.g2.bx.psu.edu/repos/jjohnson/gmap
>>> destination directory: gmap
>>> requesting all changes
>>> adding changesets
>>> adding manifests
>>> adding file changes
>>> added 11 changesets with 32 changes to 17 files
>>> updating to branch default
>>> resolving manifests
>>> getting README
>>> getting gmap.xml
>>> getting gm

Re: [galaxy-dev] Problems with custom data types and sniffers in shed tools

2012-02-07 Thread Ira Cooke

Hi Greg, 

Thanks .. that sounds like a good solution.  I'll be happy to test it when its 
ready.

Ira

On 08/02/2012, at 10:22 AM, Greg Von Kuster wrote:

> Ira,
> 
> Thanks for the feedback - I'll come up with something that allows for 
> proprietary sniffers to be loaded first and let you know when it's available. 
>  It shouldn't take me too long.  I will probably continue to keep the rule of 
> ignoring conflicts in proprietary datatypes, so sniffers for conflicting 
> datatypes will probably be ignored as well.  However, hopefully conflicts 
> will not exist, or if one is found, it will get fixed.
> 
> Greg
> 
> On Feb 7, 2012, at 6:16 PM, Ira Cooke wrote:
> 
>> Hi Greg, 
>> 
>> Thanks for that fix ... I'll check it out.
>> 
>> I think the issue of sniff order is a pretty important one for shed tools 
>> that require proprietary datatypes.  The problem is that galaxy's default 
>> sniffers include some extremely generic sniffers (eg text,xml) which will 
>> catch pretty much anything so it's hard to imagine how a tool writer will 
>> ever have their sniffers used.
>> 
>> Since the potential for conflicts with galaxy's defaults is an issue how 
>> about changing sniffer behaviour so that datatypes which subclass always 
>> have priority over their parent class ... then among those subclasses the 
>> order in which sniffers are listed in datatypes_conf is used as the second 
>> priority.  I imagine that this would require that the sniff order code be 
>> changed so that all datatypes and sniffers are first loaded .. then 
>> reshuffled to get the correct order according to the class hierarchy.  I 
>> believe this option makes sense since a subclass of a datatype should always 
>> override its parent .. and I think it would also avoid the potential for 
>> conflicts with galaxy's defaults. 
>> 
>> What do you think? Is that a system that would work?
>> 
>> Cheers
>> ira
>> 
>> 
>> 
>> 
>> On 08/02/2012, at 3:25 AM, Greg Von Kuster wrote:
>> 
>>> Hello Ira,
>>> 
>>> Very sorry for the back-and-forth on this.  The behavior you encountered 
>>> was accurate - the environment in which I was testing was not pristine.  I 
>>> have committed a fix for this issue in revision b4ba8b20d78d, which is now 
>>> available from our central repo.  
>>> 
>>> I started looking at changing things so that proprietary sniffers are 
>>> loaded before the sniffers defined in the Galaxy distribution, and 
>>> determined that this will be a bit non-trivial.  Problems arise due to 
>>> potential conflicts between a proprietary datatype that has the same 
>>> extension as a datatype defined in the Galaxy distribution.  The approach 
>>> I've taken to deal with this is that datatypes in the Galaxy distribution 
>>> get loaded first, and then any conflicting proprietary datatypes will be 
>>> ignored.  Loading proprietary datatypes first will not allow for this rule. 
>>>  
>>> 
>>> I'll gladly listen to advice from the community on this, but in the 
>>> meantime, proprietary datatypes and sniffers are loaded after those in the 
>>> distribution.
>>> 
>>> Thanks for your patience and help on this issue.
>>> 
>>> Greg
>>> 
>>> On Feb 6, 2012, at 11:37 PM, Ira Cooke wrote:
>>> 
 Hi Greg, 
 
 As far as I know there is nothing in my local setup that affects this.  As 
 I test I did the following; 
 
 Checked out a clean copy of galaxy-central 
 Edited my universe_wsgi.ini as follows;
- Changed the port to 8300
- Added an admin user
- Uncommented this line
tool_config_file = tool_conf.xml,shed_tool_conf.xml
 
 Edited shed_tool_conf.xml to point to ../shed_tools_galaxy_central (which 
 is an empty directory)
 
 Started up galaxy 
 ./run.sh --reload
 
 I then went straight to admin -> "Search and browse tool sheds" and 
 selected the main galaxy toolshed. 
 I then selected the gmap tool and told it to install.  My paster log is at 
 the bottom of this mail.
 
 My system is Mac OSX 10.7.2 and I'm running python 2.7.1
 
 I then went to the relevant bit of code and inserted some print 
 statements.  I found that; 
 
 On line 191 in registry.py there is;
 
 module = __import__( datatype_module )
 
 and the value of datatype_module is galaxy.datatypes.gmap
 
 this line seems to be causing the exception ... I'm not a python 
 programmer so I don't really know how module importing works ... could 
 this be related to the version of python I'm running?  
 alternatively should the "imported_modules" variable be used here .. since 
 wouldn't galaxy.datatypes.gmap suggest that the module needs to be located 
 inside the main galaxy lib rather than in the shed_tools.
 
 Strange that I'm the only one who gets these errors?  Any ideas what it 
 could be?
 
 Ira
 
 
 -- paster log --
 
 Cloning http://toolshed.g2.bx.psu.edu/repos/jjohnson/gm

[galaxy-dev] Error when attempting to get_updates from toolshed

2012-02-07 Thread Ira Cooke

Dear Greg, 

In the latest version of the galaxy-central code there seems to be a bug 
preventing me from updating my tools.  When I go to update a tool I get the 
following error 

URL: 
http://cookie.latrobe.edu.au:8300/admin_toolshed/browse_repositories?sort=name&f-deleted=False&operation=Get+updates&id=a3329e0bcb13f2c9
File 
'/var/www/galaxy/galaxyapp/eggs/Paste-1.6-py2.7.egg/paste/exceptions/errormiddleware.py',
 line 143 in __call__
  app_iter = self.application(environ, start_response)
File '/var/www/galaxy/galaxyapp/eggs/Paste-1.6-py2.7.egg/paste/recursive.py', 
line 80 in __call__
  return self.application(environ, start_response)
File 
'/var/www/galaxy/galaxyapp/eggs/Paste-1.6-py2.7.egg/paste/httpexceptions.py', 
line 632 in __call__
  return self.application(environ, start_response)
File '/var/www/galaxy/galaxyapp/lib/galaxy/web/framework/base.py', line 160 in 
__call__
  body = method( trans, **kwargs )
File '/var/www/galaxy/galaxyapp/lib/galaxy/web/framework/__init__.py', line 173 
in decorator
  return func( self, trans, *args, **kwargs )
File '/var/www/galaxy/galaxyapp/lib/galaxy/web/controllers/admin_toolshed.py', 
line 96 in browse_repositories
  return self.check_for_updates( trans, **kwd )
File '/var/www/galaxy/galaxyapp/lib/galaxy/web/framework/__init__.py', line 173 
in decorator
  return func( self, trans, *args, **kwargs )
File '/var/www/galaxy/galaxyapp/lib/galaxy/web/controllers/admin_toolshed.py', 
line 654 in check_for_updates
  ( tool_shed_url, url_for( '/', qualified=True ), repository.name, 
repository.owner, repository.changeset_revision )
ValueError: unsupported format character '/' (0x2f) at index 1

Looking at the code there seems to be a typo on line 702 of admin_toolshed.py 

It reads 
url = 
'%/srepository/check_for_updates?galaxy_url=%s&name=%s&owner=%s&changeset_revision=%s&webapp=galaxy'
 % \

But I think it should be
url = 
'%s/repository/check_for_updates?galaxy_url=%s&name=%s&owner=%s&changeset_revision=%s&webapp=galaxy'
 % \


When I change this though I still get further errors ... seems like a query to 
the database is not finding my toolshedrepository

The new error is

URL: 
http://127.0.0.1:8300/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://127.0.0.1:9009/&name=protk&owner=iracooke&changeset_revision=21e996f8c650&latest_changeset_revision=21e996f8c650
File 
'/Users/iracooke/Sources/galaxy-proteomics/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py',
 line 364 in respond
  app_iter = self.application(environ, detect_start_response)
File 
'/Users/iracooke/Sources/galaxy-proteomics/eggs/Paste-1.6-py2.7.egg/paste/debug/prints.py',
 line 98 in __call__
  environ, self.app)
File 
'/Users/iracooke/Sources/galaxy-proteomics/eggs/Paste-1.6-py2.7.egg/paste/wsgilib.py',
 line 539 in intercept_output
  app_iter = application(environ, replacement_start_response)
File 
'/Users/iracooke/Sources/galaxy-proteomics/eggs/Paste-1.6-py2.7.egg/paste/recursive.py',
 line 80 in __call__
  return self.application(environ, start_response)
File 
'/Users/iracooke/Sources/galaxy-proteomics/eggs/Paste-1.6-py2.7.egg/paste/httpexceptions.py',
 line 632 in __call__
  return self.application(environ, start_response)
File 
'/Users/iracooke/Sources/galaxy-proteomics/lib/galaxy/web/framework/base.py', 
line 160 in __call__
  body = method( trans, **kwargs )
File 
'/Users/iracooke/Sources/galaxy-proteomics/lib/galaxy/web/framework/__init__.py',
 line 173 in decorator
  return func( self, trans, *args, **kwargs )
File 
'/Users/iracooke/Sources/galaxy-proteomics/lib/galaxy/web/controllers/admin_toolshed.py',
 line 772 in update_to_changeset_revision
  id=trans.security.encode_id( repository.id ),
AttributeError: 'NoneType' object has no attribute 'id'

If I browse my database I can find an entry in the tool_shed_repositories table 
which seems to have all the right data.


I'm getting these errors on the latest version of the galaxy-central code. 

Ira





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Error when attempting to get_updates from toolshed

2012-02-07 Thread Greg Von Kuster

Thanks Ira, I'll have this fixed tomorrow and let you know.  Sorry for these 
glitches, we'll get these things ironed out rapidly.  Thanks for helping!

On Feb 7, 2012, at 7:40 PM, Ira Cooke wrote:

> Dear Greg, 
> 
> In the latest version of the galaxy-central code there seems to be a bug 
> preventing me from updating my tools.  When I go to update a tool I get the 
> following error 
> 
> URL: 
> http://cookie.latrobe.edu.au:8300/admin_toolshed/browse_repositories?sort=name&f-deleted=False&operation=Get+updates&id=a3329e0bcb13f2c9
> File 
> '/var/www/galaxy/galaxyapp/eggs/Paste-1.6-py2.7.egg/paste/exceptions/errormiddleware.py',
>  line 143 in __call__
>   app_iter = self.application(environ, start_response)
> File '/var/www/galaxy/galaxyapp/eggs/Paste-1.6-py2.7.egg/paste/recursive.py', 
> line 80 in __call__
>   return self.application(environ, start_response)
> File 
> '/var/www/galaxy/galaxyapp/eggs/Paste-1.6-py2.7.egg/paste/httpexceptions.py', 
> line 632 in __call__
>   return self.application(environ, start_response)
> File '/var/www/galaxy/galaxyapp/lib/galaxy/web/framework/base.py', line 160 
> in __call__
>   body = method( trans, **kwargs )
> File '/var/www/galaxy/galaxyapp/lib/galaxy/web/framework/__init__.py', line 
> 173 in decorator
>   return func( self, trans, *args, **kwargs )
> File 
> '/var/www/galaxy/galaxyapp/lib/galaxy/web/controllers/admin_toolshed.py', 
> line 96 in browse_repositories
>   return self.check_for_updates( trans, **kwd )
> File '/var/www/galaxy/galaxyapp/lib/galaxy/web/framework/__init__.py', line 
> 173 in decorator
>   return func( self, trans, *args, **kwargs )
> File 
> '/var/www/galaxy/galaxyapp/lib/galaxy/web/controllers/admin_toolshed.py', 
> line 654 in check_for_updates
>   ( tool_shed_url, url_for( '/', qualified=True ), repository.name, 
> repository.owner, repository.changeset_revision )
> ValueError: unsupported format character '/' (0x2f) at index 1
> 
> Looking at the code there seems to be a typo on line 702 of admin_toolshed.py 
> 
> It reads 
> url = 
> '%/srepository/check_for_updates?galaxy_url=%s&name=%s&owner=%s&changeset_revision=%s&webapp=galaxy'
>  % \
> 
> But I think it should be
> url = 
> '%s/repository/check_for_updates?galaxy_url=%s&name=%s&owner=%s&changeset_revision=%s&webapp=galaxy'
>  % \
> 
> 
> When I change this though I still get further errors ... seems like a query 
> to the database is not finding my toolshedrepository
> 
> The new error is
> 
> URL: 
> http://127.0.0.1:8300/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://127.0.0.1:9009/&name=protk&owner=iracooke&changeset_revision=21e996f8c650&latest_changeset_revision=21e996f8c650
> File 
> '/Users/iracooke/Sources/galaxy-proteomics/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py',
>  line 364 in respond
>   app_iter = self.application(environ, detect_start_response)
> File 
> '/Users/iracooke/Sources/galaxy-proteomics/eggs/Paste-1.6-py2.7.egg/paste/debug/prints.py',
>  line 98 in __call__
>   environ, self.app)
> File 
> '/Users/iracooke/Sources/galaxy-proteomics/eggs/Paste-1.6-py2.7.egg/paste/wsgilib.py',
>  line 539 in intercept_output
>   app_iter = application(environ, replacement_start_response)
> File 
> '/Users/iracooke/Sources/galaxy-proteomics/eggs/Paste-1.6-py2.7.egg/paste/recursive.py',
>  line 80 in __call__
>   return self.application(environ, start_response)
> File 
> '/Users/iracooke/Sources/galaxy-proteomics/eggs/Paste-1.6-py2.7.egg/paste/httpexceptions.py',
>  line 632 in __call__
>   return self.application(environ, start_response)
> File 
> '/Users/iracooke/Sources/galaxy-proteomics/lib/galaxy/web/framework/base.py', 
> line 160 in __call__
>   body = method( trans, **kwargs )
> File 
> '/Users/iracooke/Sources/galaxy-proteomics/lib/galaxy/web/framework/__init__.py',
>  line 173 in decorator
>   return func( self, trans, *args, **kwargs )
> File 
> '/Users/iracooke/Sources/galaxy-proteomics/lib/galaxy/web/controllers/admin_toolshed.py',
>  line 772 in update_to_changeset_revision
>   id=trans.security.encode_id( repository.id ),
> AttributeError: 'NoneType' object has no attribute 'id'
> 
> If I browse my database I can find an entry in the tool_shed_repositories 
> table which seems to have all the right data.
> 
> 
> I'm getting these errors on the latest version of the galaxy-central code. 
> 
> Ira
> 
> 
> 
> 
> 

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] Galaxy server configuration question

2012-02-07 Thread Huayan Gao

Dear Sir or Madam, 

I am installing a galaxy server on CentOS with UCSC Genome Browser mirror site. 
The mirror site works well. I installed the galaxy in the same server. 
Now my question is: how to set up httpd.conf file so I can access both 
websites(UCSC Genome Browser, and Galaxy) remotely? 



Best, 
Huayan
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

37 matches

Mail list logo