Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Greg Caporaso]

OK, let's plan on a Skype call at 1pm MT/3pm ET this Thursday (9 Feb
2012). I will initiate the call - my Skype ID is gregcaporaso. Please
let me know if you'd like to join the call, and send my your skype id.

Looking forward to talking about this!

Greg

2012/2/7 Rob Knight :
> I can't make the call at that time (am in Dhaka) but am very enthusiastic
> about that effort; please keep me in the loop. I am cc:ing a couple of the
> people in my lab who also indicated interest in the qiime/galaxy integration
> effort (though Antonio won't be able to make it either, for the same
> reason). Thanks!
>
> Rob
>
> On Feb 7, 2012, at 8:48 AM, Jeffrey Long wrote:
>
> Hello Amanda,
> I was just about to embark on EXACTLY this process, so I would certainly be
> very interested in saving myself some work.
> Would there be any issue (that you're aware of, of course) with using QIIME
> 1.4.0 instead of 1.3?
>
> -Jeff
>
> On Tue, Feb 7, 2012 at 2:32 AM, Florent Angly 
> wrote:
>>
>> Hi Amanda,
>> I would certainly be interested in using your helpful QIIME wrappers if
>> you put them on the Toolshed.
>> Best,
>> Florent
>>
>> On 06/02/12 06:22, Amanda Zuzolo wrote:
>>>
>>> Hello, all.
>>>
>>> I have been working on getting the Qiime scripts into Galaxy as
>>> mentioned before, and they are working with Qiime 1.3.0. I have edited
>>> the wrapper file that Jim Johnson wrote to create more flexibility,
>>> especially in cases where the tool looks for a specific file type
>>> extension (for example, a .fna file), or where the tool normally
>>> outputs something to the command line that is not normally picked up
>>> in Galaxy.
>>>
>>> So far, I have completely finished fixing the XML files to the latest
>>> documentation for the entire Pick OTU process, Alpha Diversity, and
>>> Beta Diversity, as well as other miscellaneous functions. Currently, I
>>> am working on making scripts for jack-knifing functional. I determined
>>> that it would be easier to get individual scripts functional, rather
>>> than workflow scripts, since that allows the end-user to have more
>>> control. Additionally, the workflow scripts can easily be recreated by
>>> using Galaxy's workflows.
>>>
>>> As far as the toolshed goes, I don't believe I know the ins and outs
>>> yet, but I would be more than willing to learn if people would benefit
>>> from having these versions in that repository.
>>>
>>> 2012/1/29 Jim Johnson:

 Pat,

 That sounds great.   Do one of you want to take ownership of the
 toolshed
 repository?
 At minimum, we should add developers to the list that can push changes.

 Thanks,

 JJ

 On 1/28/12 9:37 AM, Gillevet Patrick wrote:

 Jim et al

 Amanda has most of the scripts working now and will be putting them up
 on
 the toolshed.
 She will be in touch as soon as the scripts are validated a couple of
 times
 with different datasets.

 cheers...
 Pat



 On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:


 It is easiest to generate tools for galaxy when the applications or
 scripts
 can take arbitrarily named input files and generate output to given path
 names.
 Input directories, output directories are very convenient on the command
 line, but more of a challenge when crafting a galaxy tool.
 That said, many applications require a wrapper script to work with in
 galaxy.
 Thank you for the consistent script_info[] help/usage syntax in the
 qiime
 scripts,  which enabled me to generate a skeleton galaxy tool_config
 file
 for each qiime script.

 I had some time last spring to work on integrating qiime into galaxy.
 Unfortunately, I haven't had any time since to work on this.
 I put those partial results  on the Galaxy Tool Shed:
 http://toolshed.g2.bx.psu.edu/
 There's a continuing effort at George Mason University to incorporate
 qiime
 into galaxy tools, so you may want to ask them what they need.


 I started by generating galaxy tool_config files, e.g. align_seqs.xml,
  by
 using python to get the script_info[] from the qiime script:

 $ cat generate_tool_config.bash
 #!/usr/bin/env bash
 python $1>  ${1%.*}.help
 cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h>
 ${1%.*}.log

 (I'll attach tool_template.txt )

 This generated skeleton tool_config .xml files that I could then edit as
 needed.
 ( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax )

 I originally was calling all qiime scripts from a tool wrapper:
 qiime_wrapper.py
 But, if a script can be called with any input filepaths and write its
 results to any filepaths, and only writes to STDERR when it fails, then
 you
 could call that script directly.


 When should you use a tool_wrapper or call the qiime script directly?
   Many

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Rob Knight]

That's great news! Thanks for checking.

Rob

On Feb 7, 2012, at 10:59 AM, Amanda Zuzolo wrote:

> Rob,
> 
> I am not aware that there would be any issue, as I've verified all the
> options with the Qiime documentation that is up now (and I've
> eliminated those being deprecated).
> 
> Amanda Zuzolo
> 
> On 2/7/12, Jeffrey Long  wrote:
>> Hello Amanda,
>> I was just about to embark on EXACTLY this process, so I would certainly be
>> very interested in saving myself some work.
>> Would there be any issue (that you're aware of, of course) with using QIIME
>> 1.4.0 instead of 1.3?
>> 
>> -Jeff
>> 
>> On Tue, Feb 7, 2012 at 2:32 AM, Florent Angly
>> wrote:
>> 
>>> Hi Amanda,
>>> I would certainly be interested in using your helpful QIIME wrappers if
>>> you put them on the Toolshed.
>>> Best,
>>> Florent
>>> 
>>> On 06/02/12 06:22, Amanda Zuzolo wrote:
>>> 
 Hello, all.
 
 I have been working on getting the Qiime scripts into Galaxy as
 mentioned before, and they are working with Qiime 1.3.0. I have edited
 the wrapper file that Jim Johnson wrote to create more flexibility,
 especially in cases where the tool looks for a specific file type
 extension (for example, a .fna file), or where the tool normally
 outputs something to the command line that is not normally picked up
 in Galaxy.
 
 So far, I have completely finished fixing the XML files to the latest
 documentation for the entire Pick OTU process, Alpha Diversity, and
 Beta Diversity, as well as other miscellaneous functions. Currently, I
 am working on making scripts for jack-knifing functional. I determined
 that it would be easier to get individual scripts functional, rather
 than workflow scripts, since that allows the end-user to have more
 control. Additionally, the workflow scripts can easily be recreated by
 using Galaxy's workflows.
 
 As far as the toolshed goes, I don't believe I know the ins and outs
 yet, but I would be more than willing to learn if people would benefit
 from having these versions in that repository.
 
 2012/1/29 Jim Johnson:
 
> Pat,
> 
> That sounds great.   Do one of you want to take ownership of the
> toolshed
> repository?
> At minimum, we should add developers to the list that can push changes.
> 
> Thanks,
> 
> JJ
> 
> On 1/28/12 9:37 AM, Gillevet Patrick wrote:
> 
> Jim et al
> 
> Amanda has most of the scripts working now and will be putting them up
> on
> the toolshed.
> She will be in touch as soon as the scripts are validated a couple of
> times
> with different datasets.
> 
> cheers...
> Pat
> 
> 
> 
> On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:
> 
> 
> It is easiest to generate tools for galaxy when the applications or
> scripts
> can take arbitrarily named input files and generate output to given
> path
> names.
> Input directories, output directories are very convenient on the
> command
> line, but more of a challenge when crafting a galaxy tool.
> That said, many applications require a wrapper script to work with in
> galaxy.
> Thank you for the consistent script_info[] help/usage syntax in the
> qiime
> scripts,  which enabled me to generate a skeleton galaxy tool_config
> file
> for each qiime script.
> 
> I had some time last spring to work on integrating qiime into galaxy.
> Unfortunately, I haven't had any time since to work on this.
> I put those partial results  on the Galaxy Tool Shed:
> http://toolshed.g2.bx.psu.edu/
> There's a continuing effort at George Mason University to incorporate
> qiime
> into galaxy tools, so you may want to ask them what they need.
> 
> 
> I started by generating galaxy tool_config files, e.g. align_seqs.xml,
> by
> using python to get the script_info[] from the qiime script:
> 
> $ cat generate_tool_config.bash
> #!/usr/bin/env bash
> python $1>  ${1%.*}.help
> cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1
> -h>
> ${1%.*}.log
> 
> (I'll attach tool_template.txt )
> 
> This generated skeleton tool_config .xml files that I could then edit
> as
> needed.
> (
> http://wiki.g2.bx.psu.edu/**Admin/Tools/Tool%20Config%**20Syntax)
> 
> I originally was calling all qiime scripts from a tool wrapper:
> qiime_wrapper.py
> But, if a script can be called with any input filepaths and write its
> results to any filepaths, and only writes to STDERR when it fails, then
> you
> could call that script directly.
> 
> 
> When should you use a tool_wrapper or call the qiime script directly?
>  Many of the qiime scripts could probably be called directly,
> especially if
>

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Daniel McDonald]

I'll be there
Daniel



On Feb 7, 2012, at 10:30, Greg Caporaso  wrote:

> OK, let's plan on a Skype call at 1pm MT/3pm ET this Thursday (9 Feb
> 2012). I will initiate the call - my Skype ID is gregcaporaso. Please
> let me know if you'd like to join the call, and send my your skype id.
>
> Looking forward to talking about this!
>
> Greg
>
> 2012/2/7 Rob Knight :
>> I can't make the call at that time (am in Dhaka) but am very enthusiastic
>> about that effort; please keep me in the loop. I am cc:ing a couple of the
>> people in my lab who also indicated interest in the qiime/galaxy integration
>> effort (though Antonio won't be able to make it either, for the same
>> reason). Thanks!
>>
>> Rob
>>
>> On Feb 7, 2012, at 8:48 AM, Jeffrey Long wrote:
>>
>> Hello Amanda,
>> I was just about to embark on EXACTLY this process, so I would certainly be
>> very interested in saving myself some work.
>> Would there be any issue (that you're aware of, of course) with using QIIME
>> 1.4.0 instead of 1.3?
>>
>> -Jeff
>>
>> On Tue, Feb 7, 2012 at 2:32 AM, Florent Angly 
>> wrote:
>>>
>>> Hi Amanda,
>>> I would certainly be interested in using your helpful QIIME wrappers if
>>> you put them on the Toolshed.
>>> Best,
>>> Florent
>>>
>>> On 06/02/12 06:22, Amanda Zuzolo wrote:

 Hello, all.

 I have been working on getting the Qiime scripts into Galaxy as
 mentioned before, and they are working with Qiime 1.3.0. I have edited
 the wrapper file that Jim Johnson wrote to create more flexibility,
 especially in cases where the tool looks for a specific file type
 extension (for example, a .fna file), or where the tool normally
 outputs something to the command line that is not normally picked up
 in Galaxy.

 So far, I have completely finished fixing the XML files to the latest
 documentation for the entire Pick OTU process, Alpha Diversity, and
 Beta Diversity, as well as other miscellaneous functions. Currently, I
 am working on making scripts for jack-knifing functional. I determined
 that it would be easier to get individual scripts functional, rather
 than workflow scripts, since that allows the end-user to have more
 control. Additionally, the workflow scripts can easily be recreated by
 using Galaxy's workflows.

 As far as the toolshed goes, I don't believe I know the ins and outs
 yet, but I would be more than willing to learn if people would benefit
 from having these versions in that repository.

 2012/1/29 Jim Johnson:
>
> Pat,
>
> That sounds great.   Do one of you want to take ownership of the
> toolshed
> repository?
> At minimum, we should add developers to the list that can push changes.
>
> Thanks,
>
> JJ
>
> On 1/28/12 9:37 AM, Gillevet Patrick wrote:
>
> Jim et al
>
> Amanda has most of the scripts working now and will be putting them up
> on
> the toolshed.
> She will be in touch as soon as the scripts are validated a couple of
> times
> with different datasets.
>
> cheers...
> Pat
>
>
>
> On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:
>
>
> It is easiest to generate tools for galaxy when the applications or
> scripts
> can take arbitrarily named input files and generate output to given path
> names.
> Input directories, output directories are very convenient on the command
> line, but more of a challenge when crafting a galaxy tool.
> That said, many applications require a wrapper script to work with in
> galaxy.
> Thank you for the consistent script_info[] help/usage syntax in the
> qiime
> scripts,  which enabled me to generate a skeleton galaxy tool_config
> file
> for each qiime script.
>
> I had some time last spring to work on integrating qiime into galaxy.
> Unfortunately, I haven't had any time since to work on this.
> I put those partial results  on the Galaxy Tool Shed:
> http://toolshed.g2.bx.psu.edu/
> There's a continuing effort at George Mason University to incorporate
> qiime
> into galaxy tools, so you may want to ask them what they need.
>
>
> I started by generating galaxy tool_config files, e.g. align_seqs.xml,
>  by
> using python to get the script_info[] from the qiime script:
>
> $ cat generate_tool_config.bash
> #!/usr/bin/env bash
> python $1>  ${1%.*}.help
> cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h>
> ${1%.*}.log
>
> (I'll attach tool_template.txt )
>
> This generated skeleton tool_config .xml files that I could then edit as
> needed.
> ( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax )
>
> I originally was calling all qiime scripts from a tool wrapper:
> qiime_wrapper.py
> But, if a script can be called with any input filepaths and write its
> 

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Gillevet Patrick]

Skype ID : patrickgillevet


On Feb 7, 2012, at 12:33 PM, Daniel McDonald wrote:

> I'll be there
> Daniel
> 
> 
> 
> On Feb 7, 2012, at 10:30, Greg Caporaso  wrote:
> 
>> OK, let's plan on a Skype call at 1pm MT/3pm ET this Thursday (9 Feb
>> 2012). I will initiate the call - my Skype ID is gregcaporaso. Please
>> let me know if you'd like to join the call, and send my your skype id.
>> 
>> Looking forward to talking about this!
>> 
>> Greg
>> 
>> 2012/2/7 Rob Knight :
>>> I can't make the call at that time (am in Dhaka) but am very enthusiastic
>>> about that effort; please keep me in the loop. I am cc:ing a couple of the
>>> people in my lab who also indicated interest in the qiime/galaxy integration
>>> effort (though Antonio won't be able to make it either, for the same
>>> reason). Thanks!
>>> 
>>> Rob
>>> 
>>> On Feb 7, 2012, at 8:48 AM, Jeffrey Long wrote:
>>> 
>>> Hello Amanda,
>>> I was just about to embark on EXACTLY this process, so I would certainly be
>>> very interested in saving myself some work.
>>> Would there be any issue (that you're aware of, of course) with using QIIME
>>> 1.4.0 instead of 1.3?
>>> 
>>> -Jeff
>>> 
>>> On Tue, Feb 7, 2012 at 2:32 AM, Florent Angly 
>>> wrote:
 
 Hi Amanda,
 I would certainly be interested in using your helpful QIIME wrappers if
 you put them on the Toolshed.
 Best,
 Florent
 
 On 06/02/12 06:22, Amanda Zuzolo wrote:
> 
> Hello, all.
> 
> I have been working on getting the Qiime scripts into Galaxy as
> mentioned before, and they are working with Qiime 1.3.0. I have edited
> the wrapper file that Jim Johnson wrote to create more flexibility,
> especially in cases where the tool looks for a specific file type
> extension (for example, a .fna file), or where the tool normally
> outputs something to the command line that is not normally picked up
> in Galaxy.
> 
> So far, I have completely finished fixing the XML files to the latest
> documentation for the entire Pick OTU process, Alpha Diversity, and
> Beta Diversity, as well as other miscellaneous functions. Currently, I
> am working on making scripts for jack-knifing functional. I determined
> that it would be easier to get individual scripts functional, rather
> than workflow scripts, since that allows the end-user to have more
> control. Additionally, the workflow scripts can easily be recreated by
> using Galaxy's workflows.
> 
> As far as the toolshed goes, I don't believe I know the ins and outs
> yet, but I would be more than willing to learn if people would benefit
> from having these versions in that repository.
> 
> 2012/1/29 Jim Johnson:
>> 
>> Pat,
>> 
>> That sounds great.   Do one of you want to take ownership of the
>> toolshed
>> repository?
>> At minimum, we should add developers to the list that can push changes.
>> 
>> Thanks,
>> 
>> JJ
>> 
>> On 1/28/12 9:37 AM, Gillevet Patrick wrote:
>> 
>> Jim et al
>> 
>> Amanda has most of the scripts working now and will be putting them up
>> on
>> the toolshed.
>> She will be in touch as soon as the scripts are validated a couple of
>> times
>> with different datasets.
>> 
>> cheers...
>> Pat
>> 
>> 
>> 
>> On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:
>> 
>> 
>> It is easiest to generate tools for galaxy when the applications or
>> scripts
>> can take arbitrarily named input files and generate output to given path
>> names.
>> Input directories, output directories are very convenient on the command
>> line, but more of a challenge when crafting a galaxy tool.
>> That said, many applications require a wrapper script to work with in
>> galaxy.
>> Thank you for the consistent script_info[] help/usage syntax in the
>> qiime
>> scripts,  which enabled me to generate a skeleton galaxy tool_config
>> file
>> for each qiime script.
>> 
>> I had some time last spring to work on integrating qiime into galaxy.
>> Unfortunately, I haven't had any time since to work on this.
>> I put those partial results  on the Galaxy Tool Shed:
>> http://toolshed.g2.bx.psu.edu/
>> There's a continuing effort at George Mason University to incorporate
>> qiime
>> into galaxy tools, so you may want to ask them what they need.
>> 
>> 
>> I started by generating galaxy tool_config files, e.g. align_seqs.xml,
>> by
>> using python to get the script_info[] from the qiime script:
>> 
>> $ cat generate_tool_config.bash
>> #!/usr/bin/env bash
>> python $1>  ${1%.*}.help
>> cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h>
>> ${1%.*}.log
>> 
>> (I'll attach tool_template.txt )
>> 
>> This generated skeleton tool_config .xml files that I could then edit as
>> n

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Rob Knight]

I can't make the call at that time (am in Dhaka) but am very enthusiastic about that effort; please keep me in the loop. I am cc:ing a couple of the people in my lab who also indicated interest in the qiime/galaxy integration effort (though Antonio won't be able to make it either, for the same reason). Thanks!RobOn Feb 7, 2012, at 8:48 AM, Jeffrey Long wrote:Hello Amanda,I was just about to embark on EXACTLY this process, so I would certainly be very interested in saving myself some work.  Would there be any issue (that you're aware of, of course) with using QIIME 1.4.0 instead of 1.3?
-JeffOn Tue, Feb 7, 2012 at 2:32 AM, Florent Angly  wrote:
Hi Amanda,
I would certainly be interested in using your helpful QIIME wrappers if you put them on the Toolshed.
Best,
Florent

On 06/02/12 06:22, Amanda Zuzolo wrote:

Hello, all.

I have been working on getting the Qiime scripts into Galaxy as
mentioned before, and they are working with Qiime 1.3.0. I have edited
the wrapper file that Jim Johnson wrote to create more flexibility,
especially in cases where the tool looks for a specific file type
extension (for example, a .fna file), or where the tool normally
outputs something to the command line that is not normally picked up
in Galaxy.

So far, I have completely finished fixing the XML files to the latest
documentation for the entire Pick OTU process, Alpha Diversity, and
Beta Diversity, as well as other miscellaneous functions. Currently, I
am working on making scripts for jack-knifing functional. I determined
that it would be easier to get individual scripts functional, rather
than workflow scripts, since that allows the end-user to have more
control. Additionally, the workflow scripts can easily be recreated by
using Galaxy's workflows.

As far as the toolshed goes, I don't believe I know the ins and outs
yet, but I would be more than willing to learn if people would benefit
from having these versions in that repository.

2012/1/29 Jim Johnson:

Pat,

That sounds great.   Do one of you want to take ownership of the toolshed
repository?
At minimum, we should add developers to the list that can push changes.

Thanks,

JJ

On 1/28/12 9:37 AM, Gillevet Patrick wrote:

Jim et al

Amanda has most of the scripts working now and will be putting them up on
the toolshed.
She will be in touch as soon as the scripts are validated a couple of times
with different datasets.

cheers...
Pat



On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:


It is easiest to generate tools for galaxy when the applications or scripts
can take arbitrarily named input files and generate output to given path
names.
Input directories, output directories are very convenient on the command
line, but more of a challenge when crafting a galaxy tool.
That said, many applications require a wrapper script to work with in
galaxy.
Thank you for the consistent script_info[] help/usage syntax in the qiime
scripts,  which enabled me to generate a skeleton galaxy tool_config file
for each qiime script.

I had some time last spring to work on integrating qiime into galaxy.
Unfortunately, I haven't had any time since to work on this.
I put those partial results  on the Galaxy Tool Shed:
http://toolshed.g2.bx.psu.edu/
There's a continuing effort at George Mason University to incorporate qiime
into galaxy tools, so you may want to ask them what they need.


I started by generating galaxy tool_config files, e.g. align_seqs.xml,  by
using python to get the script_info[] from the qiime script:

$ cat generate_tool_config.bash
#!/usr/bin/env bash
python $1>  ${1%.*}.help
cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h>
${1%.*}.log

(I'll attach tool_template.txt )

This generated skeleton tool_config .xml files that I could then edit as
needed.
( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax )

I originally was calling all qiime scripts from a tool wrapper:
qiime_wrapper.py
But, if a script can be called with any input filepaths and write its
results to any filepaths, and only writes to STDERR when it fails, then you
could call that script directly.


When should you use a tool_wrapper or call the qiime script directly?
   Many of the qiime scripts could probably be called directly, especially if
it can be called with arbitary input/output file pathnames.
   The reasons for using a tool wrapper may be if input/output needs to be
manipulated, moved, renamed in order to be used by the qiime script.
   You'll also need a tool wrapper if the names or number of the output files
can not be determined from the parameter settings.
   ( http://wiki.g2.bx.psu.edu/Admin/Tools/Multiple%20Output%20Files )
   If your tool relies on a file ext to determine a format, you'll have to
rename the input.
   ( Galaxy dataset pathnames will look something like:
//072/dataset_72931.dat )
   The format/type of a dataset is stored in its metadata, so the tool_config
can use that information, especially if a scrip

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Amanda Zuzolo]

Rob,

I am not aware that there would be any issue, as I've verified all the
options with the Qiime documentation that is up now (and I've
eliminated those being deprecated).

Amanda Zuzolo

On 2/7/12, Jeffrey Long  wrote:
> Hello Amanda,
> I was just about to embark on EXACTLY this process, so I would certainly be
> very interested in saving myself some work.
> Would there be any issue (that you're aware of, of course) with using QIIME
> 1.4.0 instead of 1.3?
>
> -Jeff
>
> On Tue, Feb 7, 2012 at 2:32 AM, Florent Angly
> wrote:
>
>> Hi Amanda,
>> I would certainly be interested in using your helpful QIIME wrappers if
>> you put them on the Toolshed.
>> Best,
>> Florent
>>
>> On 06/02/12 06:22, Amanda Zuzolo wrote:
>>
>>> Hello, all.
>>>
>>> I have been working on getting the Qiime scripts into Galaxy as
>>> mentioned before, and they are working with Qiime 1.3.0. I have edited
>>> the wrapper file that Jim Johnson wrote to create more flexibility,
>>> especially in cases where the tool looks for a specific file type
>>> extension (for example, a .fna file), or where the tool normally
>>> outputs something to the command line that is not normally picked up
>>> in Galaxy.
>>>
>>> So far, I have completely finished fixing the XML files to the latest
>>> documentation for the entire Pick OTU process, Alpha Diversity, and
>>> Beta Diversity, as well as other miscellaneous functions. Currently, I
>>> am working on making scripts for jack-knifing functional. I determined
>>> that it would be easier to get individual scripts functional, rather
>>> than workflow scripts, since that allows the end-user to have more
>>> control. Additionally, the workflow scripts can easily be recreated by
>>> using Galaxy's workflows.
>>>
>>> As far as the toolshed goes, I don't believe I know the ins and outs
>>> yet, but I would be more than willing to learn if people would benefit
>>> from having these versions in that repository.
>>>
>>> 2012/1/29 Jim Johnson:
>>>
 Pat,

 That sounds great.   Do one of you want to take ownership of the
 toolshed
 repository?
 At minimum, we should add developers to the list that can push changes.

 Thanks,

 JJ

 On 1/28/12 9:37 AM, Gillevet Patrick wrote:

 Jim et al

 Amanda has most of the scripts working now and will be putting them up
 on
 the toolshed.
 She will be in touch as soon as the scripts are validated a couple of
 times
 with different datasets.

 cheers...
 Pat



 On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:


 It is easiest to generate tools for galaxy when the applications or
 scripts
 can take arbitrarily named input files and generate output to given
 path
 names.
 Input directories, output directories are very convenient on the
 command
 line, but more of a challenge when crafting a galaxy tool.
 That said, many applications require a wrapper script to work with in
 galaxy.
 Thank you for the consistent script_info[] help/usage syntax in the
 qiime
 scripts,  which enabled me to generate a skeleton galaxy tool_config
 file
 for each qiime script.

 I had some time last spring to work on integrating qiime into galaxy.
 Unfortunately, I haven't had any time since to work on this.
 I put those partial results  on the Galaxy Tool Shed:
 http://toolshed.g2.bx.psu.edu/
 There's a continuing effort at George Mason University to incorporate
 qiime
 into galaxy tools, so you may want to ask them what they need.


 I started by generating galaxy tool_config files, e.g. align_seqs.xml,
  by
 using python to get the script_info[] from the qiime script:

 $ cat generate_tool_config.bash
 #!/usr/bin/env bash
 python $1>  ${1%.*}.help
 cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1
 -h>
 ${1%.*}.log

 (I'll attach tool_template.txt )

 This generated skeleton tool_config .xml files that I could then edit
 as
 needed.
 (
 http://wiki.g2.bx.psu.edu/**Admin/Tools/Tool%20Config%**20Syntax)

 I originally was calling all qiime scripts from a tool wrapper:
 qiime_wrapper.py
 But, if a script can be called with any input filepaths and write its
 results to any filepaths, and only writes to STDERR when it fails, then
 you
 could call that script directly.


 When should you use a tool_wrapper or call the qiime script directly?
   Many of the qiime scripts could probably be called directly,
 especially if
 it can be called with arbitary input/output file pathnames.
   The reasons for using a tool wrapper may be if input/output needs to
 be
 manipulated, moved, renamed in order to be used by the qiime script.
   You'll also need a tool wrapper if the names or 

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Jeffrey Long]

Hello Amanda,
I was just about to embark on EXACTLY this process, so I would certainly be
very interested in saving myself some work.
Would there be any issue (that you're aware of, of course) with using QIIME
1.4.0 instead of 1.3?

-Jeff

On Tue, Feb 7, 2012 at 2:32 AM, Florent Angly wrote:

> Hi Amanda,
> I would certainly be interested in using your helpful QIIME wrappers if
> you put them on the Toolshed.
> Best,
> Florent
>
> On 06/02/12 06:22, Amanda Zuzolo wrote:
>
>> Hello, all.
>>
>> I have been working on getting the Qiime scripts into Galaxy as
>> mentioned before, and they are working with Qiime 1.3.0. I have edited
>> the wrapper file that Jim Johnson wrote to create more flexibility,
>> especially in cases where the tool looks for a specific file type
>> extension (for example, a .fna file), or where the tool normally
>> outputs something to the command line that is not normally picked up
>> in Galaxy.
>>
>> So far, I have completely finished fixing the XML files to the latest
>> documentation for the entire Pick OTU process, Alpha Diversity, and
>> Beta Diversity, as well as other miscellaneous functions. Currently, I
>> am working on making scripts for jack-knifing functional. I determined
>> that it would be easier to get individual scripts functional, rather
>> than workflow scripts, since that allows the end-user to have more
>> control. Additionally, the workflow scripts can easily be recreated by
>> using Galaxy's workflows.
>>
>> As far as the toolshed goes, I don't believe I know the ins and outs
>> yet, but I would be more than willing to learn if people would benefit
>> from having these versions in that repository.
>>
>> 2012/1/29 Jim Johnson:
>>
>>> Pat,
>>>
>>> That sounds great.   Do one of you want to take ownership of the toolshed
>>> repository?
>>> At minimum, we should add developers to the list that can push changes.
>>>
>>> Thanks,
>>>
>>> JJ
>>>
>>> On 1/28/12 9:37 AM, Gillevet Patrick wrote:
>>>
>>> Jim et al
>>>
>>> Amanda has most of the scripts working now and will be putting them up on
>>> the toolshed.
>>> She will be in touch as soon as the scripts are validated a couple of
>>> times
>>> with different datasets.
>>>
>>> cheers...
>>> Pat
>>>
>>>
>>>
>>> On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:
>>>
>>>
>>> It is easiest to generate tools for galaxy when the applications or
>>> scripts
>>> can take arbitrarily named input files and generate output to given path
>>> names.
>>> Input directories, output directories are very convenient on the command
>>> line, but more of a challenge when crafting a galaxy tool.
>>> That said, many applications require a wrapper script to work with in
>>> galaxy.
>>> Thank you for the consistent script_info[] help/usage syntax in the qiime
>>> scripts,  which enabled me to generate a skeleton galaxy tool_config file
>>> for each qiime script.
>>>
>>> I had some time last spring to work on integrating qiime into galaxy.
>>> Unfortunately, I haven't had any time since to work on this.
>>> I put those partial results  on the Galaxy Tool Shed:
>>> http://toolshed.g2.bx.psu.edu/
>>> There's a continuing effort at George Mason University to incorporate
>>> qiime
>>> into galaxy tools, so you may want to ask them what they need.
>>>
>>>
>>> I started by generating galaxy tool_config files, e.g. align_seqs.xml,
>>>  by
>>> using python to get the script_info[] from the qiime script:
>>>
>>> $ cat generate_tool_config.bash
>>> #!/usr/bin/env bash
>>> python $1>  ${1%.*}.help
>>> cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h>
>>> ${1%.*}.log
>>>
>>> (I'll attach tool_template.txt )
>>>
>>> This generated skeleton tool_config .xml files that I could then edit as
>>> needed.
>>> ( 
>>> http://wiki.g2.bx.psu.edu/**Admin/Tools/Tool%20Config%**20Syntax)
>>>
>>> I originally was calling all qiime scripts from a tool wrapper:
>>> qiime_wrapper.py
>>> But, if a script can be called with any input filepaths and write its
>>> results to any filepaths, and only writes to STDERR when it fails, then
>>> you
>>> could call that script directly.
>>>
>>>
>>> When should you use a tool_wrapper or call the qiime script directly?
>>>   Many of the qiime scripts could probably be called directly,
>>> especially if
>>> it can be called with arbitary input/output file pathnames.
>>>   The reasons for using a tool wrapper may be if input/output needs to be
>>> manipulated, moved, renamed in order to be used by the qiime script.
>>>   You'll also need a tool wrapper if the names or number of the output
>>> files
>>> can not be determined from the parameter settings.
>>>   ( 
>>> http://wiki.g2.bx.psu.edu/**Admin/Tools/Multiple%20Output%**20Files)
>>>   If your tool relies on a file ext to determine a format, you'll have to
>>> rename the input.
>>>   ( Galaxy dataset pathnames will look something like:
>>> //07

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Florent Angly]

Hi Amanda,
I would certainly be interested in using your helpful QIIME wrappers if 
you put them on the Toolshed.

Best,
Florent

On 06/02/12 06:22, Amanda Zuzolo wrote:

Hello, all.

I have been working on getting the Qiime scripts into Galaxy as
mentioned before, and they are working with Qiime 1.3.0. I have edited
the wrapper file that Jim Johnson wrote to create more flexibility,
especially in cases where the tool looks for a specific file type
extension (for example, a .fna file), or where the tool normally
outputs something to the command line that is not normally picked up
in Galaxy.

So far, I have completely finished fixing the XML files to the latest
documentation for the entire Pick OTU process, Alpha Diversity, and
Beta Diversity, as well as other miscellaneous functions. Currently, I
am working on making scripts for jack-knifing functional. I determined
that it would be easier to get individual scripts functional, rather
than workflow scripts, since that allows the end-user to have more
control. Additionally, the workflow scripts can easily be recreated by
using Galaxy's workflows.

As far as the toolshed goes, I don't believe I know the ins and outs
yet, but I would be more than willing to learn if people would benefit
from having these versions in that repository.

2012/1/29 Jim Johnson:

Pat,

That sounds great.   Do one of you want to take ownership of the toolshed
repository?
At minimum, we should add developers to the list that can push changes.

Thanks,

JJ

On 1/28/12 9:37 AM, Gillevet Patrick wrote:

Jim et al

Amanda has most of the scripts working now and will be putting them up on
the toolshed.
She will be in touch as soon as the scripts are validated a couple of times
with different datasets.

cheers...
Pat



On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:


It is easiest to generate tools for galaxy when the applications or scripts
can take arbitrarily named input files and generate output to given path
names.
Input directories, output directories are very convenient on the command
line, but more of a challenge when crafting a galaxy tool.
That said, many applications require a wrapper script to work with in
galaxy.
Thank you for the consistent script_info[] help/usage syntax in the qiime
scripts,  which enabled me to generate a skeleton galaxy tool_config file
for each qiime script.

I had some time last spring to work on integrating qiime into galaxy.
Unfortunately, I haven't had any time since to work on this.
I put those partial results  on the Galaxy Tool Shed:
http://toolshed.g2.bx.psu.edu/
There's a continuing effort at George Mason University to incorporate qiime
into galaxy tools, so you may want to ask them what they need.


I started by generating galaxy tool_config files, e.g. align_seqs.xml,  by
using python to get the script_info[] from the qiime script:

$ cat generate_tool_config.bash
#!/usr/bin/env bash
python $1>  ${1%.*}.help
cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h>
${1%.*}.log

(I'll attach tool_template.txt )

This generated skeleton tool_config .xml files that I could then edit as
needed.
( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax )

I originally was calling all qiime scripts from a tool wrapper:
qiime_wrapper.py
But, if a script can be called with any input filepaths and write its
results to any filepaths, and only writes to STDERR when it fails, then you
could call that script directly.


When should you use a tool_wrapper or call the qiime script directly?
   Many of the qiime scripts could probably be called directly, especially if
it can be called with arbitary input/output file pathnames.
   The reasons for using a tool wrapper may be if input/output needs to be
manipulated, moved, renamed in order to be used by the qiime script.
   You'll also need a tool wrapper if the names or number of the output files
can not be determined from the parameter settings.
   ( http://wiki.g2.bx.psu.edu/Admin/Tools/Multiple%20Output%20Files )
   If your tool relies on a file ext to determine a format, you'll have to
rename the input.
   ( Galaxy dataset pathnames will look something like:
//072/dataset_72931.dat )
   The format/type of a dataset is stored in its metadata, so the tool_config
can use that information, especially if a script can take muliple
alternative input formats.
   A tool_wrapper can also be used to manage the stdout or stderr from a
tool.   Galaxy currently interprets any output on stderr as a failure.



A couple changes in galaxy should make somethings easier than when I first
attempted this:
   - galaxy now accepts dataset requests with sub directories. (
https://bitbucket.org/galaxy/galaxy-central/issue/494/support-sub-dirs-in-extra_files_path-patch
)
 That means that output HTML files with links into sub directories can be
left intact, with the html copied to the output dataset and the linked files
to its "extra_files_path".
   - if you know the pathname of an output relative to the wor

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Amanda Zuzolo]

Hello, all.

I have been working on getting the Qiime scripts into Galaxy as
mentioned before, and they are working with Qiime 1.3.0. I have edited
the wrapper file that Jim Johnson wrote to create more flexibility,
especially in cases where the tool looks for a specific file type
extension (for example, a .fna file), or where the tool normally
outputs something to the command line that is not normally picked up
in Galaxy.

So far, I have completely finished fixing the XML files to the latest
documentation for the entire Pick OTU process, Alpha Diversity, and
Beta Diversity, as well as other miscellaneous functions. Currently, I
am working on making scripts for jack-knifing functional. I determined
that it would be easier to get individual scripts functional, rather
than workflow scripts, since that allows the end-user to have more
control. Additionally, the workflow scripts can easily be recreated by
using Galaxy's workflows.

As far as the toolshed goes, I don't believe I know the ins and outs
yet, but I would be more than willing to learn if people would benefit
from having these versions in that repository.

2012/1/29 Jim Johnson :
> Pat,
>
> That sounds great.   Do one of you want to take ownership of the toolshed
> repository?
> At minimum, we should add developers to the list that can push changes.
>
> Thanks,
>
> JJ
>
> On 1/28/12 9:37 AM, Gillevet Patrick wrote:
>
> Jim et al
>
> Amanda has most of the scripts working now and will be putting them up on
> the toolshed.
> She will be in touch as soon as the scripts are validated a couple of times
> with different datasets.
>
> cheers...
> Pat
>
>
>
> On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:
>
>
> It is easiest to generate tools for galaxy when the applications or scripts
> can take arbitrarily named input files and generate output to given path
> names.
> Input directories, output directories are very convenient on the command
> line, but more of a challenge when crafting a galaxy tool.
> That said, many applications require a wrapper script to work with in
> galaxy.
> Thank you for the consistent script_info[] help/usage syntax in the qiime
> scripts,  which enabled me to generate a skeleton galaxy tool_config file
> for each qiime script.
>
> I had some time last spring to work on integrating qiime into galaxy.
> Unfortunately, I haven't had any time since to work on this.
> I put those partial results  on the Galaxy Tool Shed:
> http://toolshed.g2.bx.psu.edu/
> There's a continuing effort at George Mason University to incorporate qiime
> into galaxy tools, so you may want to ask them what they need.
>
>
> I started by generating galaxy tool_config files, e.g. align_seqs.xml,  by
> using python to get the script_info[] from the qiime script:
>
> $ cat generate_tool_config.bash
> #!/usr/bin/env bash
> python $1 > ${1%.*}.help
> cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h >
> ${1%.*}.log
>
> (I'll attach tool_template.txt )
>
> This generated skeleton tool_config .xml files that I could then edit as
> needed.
> ( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax )
>
> I originally was calling all qiime scripts from a tool wrapper:
> qiime_wrapper.py
> But, if a script can be called with any input filepaths and write its
> results to any filepaths, and only writes to STDERR when it fails, then you
> could call that script directly.
>
>
> When should you use a tool_wrapper or call the qiime script directly?
>   Many of the qiime scripts could probably be called directly, especially if
> it can be called with arbitary input/output file pathnames.
>   The reasons for using a tool wrapper may be if input/output needs to be
> manipulated, moved, renamed in order to be used by the qiime script.
>   You'll also need a tool wrapper if the names or number of the output files
> can not be determined from the parameter settings.
>   ( http://wiki.g2.bx.psu.edu/Admin/Tools/Multiple%20Output%20Files )
>   If your tool relies on a file ext to determine a format, you'll have to
> rename the input.
>   ( Galaxy dataset pathnames will look something like:
> //072/dataset_72931.dat )
>   The format/type of a dataset is stored in its metadata, so the tool_config
> can use that information, especially if a script can take muliple
> alternative input formats.
>   A tool_wrapper can also be used to manage the stdout or stderr from a
> tool.   Galaxy currently interprets any output on stderr as a failure.
>
>
>
> A couple changes in galaxy should make somethings easier than when I first
> attempted this:
>   - galaxy now accepts dataset requests with sub directories. (
> https://bitbucket.org/galaxy/galaxy-central/issue/494/support-sub-dirs-in-extra_files_path-patch
> )
>     That means that output HTML files with links into sub directories can be
> left intact, with the html copied to the output dataset and the linked files
> to its "extra_files_path".
>   - if you know the pathname of an output relative to the working directo

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Greg Caporaso]

Hi all,
Do you have a working tool definition file for QIIME's
beta_diversity_through_plots.py script? We're investigating whether
replacing the Fast UniFrac website with this would be feasible, and
I'd like to see if you have something together before I try to write
one myself.

Thanks!
Greg

On Sun, Jan 29, 2012 at 10:01 AM, Rob Knight  wrote:
> This is great news -- thanks for letting us know, and for your hard work on
> this!
>
> Rob
>
> On Jan 29, 2012, at 9:46 AM, Jim Johnson wrote:
>
> Pat,
>
> That sounds great.   Do one of you want to take ownership of the toolshed
> repository?
> At minimum, we should add developers to the list that can push changes.
>
> Thanks,
>
> JJ
>
> On 1/28/12 9:37 AM, Gillevet Patrick wrote:
>
> Jim et al
>
> Amanda has most of the scripts working now and will be putting them up on
> the toolshed.
> She will be in touch as soon as the scripts are validated a couple of times
> with different datasets.
>
> cheers...
> Pat
>
>
>
> On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:
>
>
> It is easiest to generate tools for galaxy when the applications or scripts
> can take arbitrarily named input files and generate output to given path
> names.
> Input directories, output directories are very convenient on the command
> line, but more of a challenge when crafting a galaxy tool.
> That said, many applications require a wrapper script to work with in
> galaxy.
> Thank you for the consistent script_info[] help/usage syntax in the qiime
> scripts,  which enabled me to generate a skeleton galaxy tool_config file
> for each qiime script.
>
> I had some time last spring to work on integrating qiime into galaxy.
> Unfortunately, I haven't had any time since to work on this.
> I put those partial results  on the Galaxy Tool Shed:
> http://toolshed.g2.bx.psu.edu/
> There's a continuing effort at George Mason University to incorporate qiime
> into galaxy tools, so you may want to ask them what they need.
>
>
> I started by generating galaxy tool_config files, e.g. align_seqs.xml,  by
> using python to get the script_info[] from the qiime script:
>
> $ cat generate_tool_config.bash
> #!/usr/bin/env bash
> python $1 > ${1%.*}.help
> cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h >
> ${1%.*}.log
>
> (I'll attach tool_template.txt )
>
> This generated skeleton tool_config .xml files that I could then edit as
> needed.
> ( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax )
>
> I originally was calling all qiime scripts from a tool wrapper:
> qiime_wrapper.py
> But, if a script can be called with any input filepaths and write its
> results to any filepaths, and only writes to STDERR when it fails, then you
> could call that script directly.
>
>
> When should you use a tool_wrapper or call the qiime script directly?
>   Many of the qiime scripts could probably be called directly, especially if
> it can be called with arbitary input/output file pathnames.
>   The reasons for using a tool wrapper may be if input/output needs to be
> manipulated, moved, renamed in order to be used by the qiime script.
>   You'll also need a tool wrapper if the names or number of the output files
> can not be determined from the parameter settings.
>   ( http://wiki.g2.bx.psu.edu/Admin/Tools/Multiple%20Output%20Files )
>   If your tool relies on a file ext to determine a format, you'll have to
> rename the input.
>   ( Galaxy dataset pathnames will look something like:
> //072/dataset_72931.dat )
>   The format/type of a dataset is stored in its metadata, so the tool_config
> can use that information, especially if a script can take muliple
> alternative input formats.
>   A tool_wrapper can also be used to manage the stdout or stderr from a
> tool.   Galaxy currently interprets any output on stderr as a failure.
>
>
>
> A couple changes in galaxy should make somethings easier than when I first
> attempted this:
>   - galaxy now accepts dataset requests with sub directories. (
> https://bitbucket.org/galaxy/galaxy-central/issue/494/support-sub-dirs-in-extra_files_path-patch
> )
>     That means that output HTML files with links into sub directories can be
> left intact, with the html copied to the output dataset and the linked files
> to its "extra_files_path".
>   - if you know the pathname of an output relative to the working directory,
> galaxy can copy it automatically to the output dataset using the
> from_work_dir attribute.
>     ( see example in:
> https://bitbucket.org/galaxy/galaxy-central/src/21b645303c02/tools/ngs_rna/tophat_wrapper.xml
> )
>
> Datatypes
>   You may want to create new datatypes to make it easier for the user to
> correctly select inputs to a tool from previous outputs.
>   For example, the qiime mapping file is a tabular file with specific
> requirements.  I put a 'qiimemapping' datatype in
> lib/galaxy/datatypes/metagenomics.py and datatypes_conf.xml
>   so an input could generate a select list containing only qiimemapping
> datasets rather than all t

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Rob Knight]

This is great news -- thanks for letting us know, and for your hard work on this!RobOn Jan 29, 2012, at 9:46 AM, Jim Johnson wrote:
  

  
  
Pat,

That sounds great.   Do one of you want to take ownership of the
toolshed repository?  
At minimum, we should add developers to the list that can push
changes.  

Thanks,

JJ  

On 1/28/12 9:37 AM, Gillevet Patrick wrote:

  Jim et al
  
  
  Amanda has most of the scripts working now and will be
putting them up on the toolshed.
  She will be in touch as soon as the scripts are validated a
couple of times with different datasets.
  
  
  cheers...
  Pat
  
  
  
  
  
  
On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:


  
   
It is easiest to generate tools for galaxy when the
applications or scripts can take arbitrarily named input
files and generate output to given path names.   
Input directories, output directories are very convenient on
the command line, but more of a challenge when crafting a
galaxy tool.  
That said, many applications require a wrapper script to
work with in galaxy.    
Thank you for the consistent script_info[] help/usage syntax
in the qiime scripts,  which enabled me to generate a
skeleton galaxy tool_config file for each qiime script.

I had some time last spring to work on integrating qiime
into galaxy.
Unfortunately, I haven't had any time since to work on this.
I put those partial results  on the Galaxy Tool Shed:    http://toolshed.g2.bx.psu.edu/
There's a continuing effort at George Mason University to
incorporate qiime into galaxy tools, so you may want to ask
them what they need.  


I started by generating galaxy tool_config files, e.g.
align_seqs.xml,  by using python to get the script_info[]
from the qiime script:

$ cat generate_tool_config.bash
#!/usr/bin/env bash
python $1 > ${1%.*}.help
cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" |
python -i $1 -h > ${1%.*}.log

(I'll attach tool_template.txt )

This generated skeleton tool_config .xml files that I could
then edit as needed.
( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax
)

I originally was calling all qiime scripts from a tool
wrapper:  qiime_wrapper.py
But, if a script can be called with any input filepaths and
write its results to any filepaths, and only writes to
STDERR when it fails, then you could call that script
directly.
   

When should you use a tool_wrapper or call the qiime script
directly?
  Many of the qiime scripts could probably be called
directly, especially if it can be called with arbitary
input/output file pathnames.
  The reasons for using a tool wrapper may be if
input/output needs to be manipulated, moved, renamed in
order to be used by the qiime script.
  You'll also need a tool wrapper if the names or number of
the output files can not be determined from the parameter
settings.
  ( http://wiki.g2.bx.psu.edu/Admin/Tools/Multiple%20Output%20Files
)
  If your tool relies on a file ext to determine a format,
you'll have to rename the input.
  ( Galaxy dataset pathnames will look something like: 
//072/dataset_72931.dat )
  The format/type of a dataset is stored in its metadata, so
the tool_config can use that information, especially if a
script can take muliple alternative input formats.
  A tool_wrapper can also be used to manage the stdout or
stderr from a tool.   Galaxy currently interprets any output
on stderr as a failure.



A couple changes in galaxy should make somethings easier
than when I first attempted this:   
  - galaxy now accepts dataset requests with sub
directories. ( https://bitbucket.org/galaxy/galaxy-central/issue/494/support-sub-dirs-in-extra_files_path-patch
)
    That means that output HTML files with links into sub
directories can be left intact, with the html copied to the
output dataset and the linked files to its
"extra_files_path".
  - if you know the pathname 

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Gillevet Patrick]

Jim et al

Amanda has most of the scripts working now and will be putting them up on the 
toolshed.
She will be in touch as soon as the scripts are validated a couple of times 
with different datasets.

cheers...
Pat



On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:

> 
> It is easiest to generate tools for galaxy when the applications or scripts 
> can take arbitrarily named input files and generate output to given path 
> names.   
> Input directories, output directories are very convenient on the command 
> line, but more of a challenge when crafting a galaxy tool.  
> That said, many applications require a wrapper script to work with in galaxy. 
>
> Thank you for the consistent script_info[] help/usage syntax in the qiime 
> scripts,  which enabled me to generate a skeleton galaxy tool_config file for 
> each qiime script.
> 
> I had some time last spring to work on integrating qiime into galaxy.
> Unfortunately, I haven't had any time since to work on this.
> I put those partial results  on the Galaxy Tool Shed:
> http://toolshed.g2.bx.psu.edu/
> There's a continuing effort at George Mason University to incorporate qiime 
> into galaxy tools, so you may want to ask them what they need.  
> 
> 
> I started by generating galaxy tool_config files, e.g. align_seqs.xml,  by 
> using python to get the script_info[] from the qiime script:
> 
> $ cat generate_tool_config.bash
> #!/usr/bin/env bash
> python $1 > ${1%.*}.help
> cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h > 
> ${1%.*}.log
> 
> (I'll attach tool_template.txt )
> 
> This generated skeleton tool_config .xml files that I could then edit as 
> needed.
> ( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax )
> 
> I originally was calling all qiime scripts from a tool wrapper:  
> qiime_wrapper.py
> But, if a script can be called with any input filepaths and write its results 
> to any filepaths, and only writes to STDERR when it fails, then you could 
> call that script directly.
>
> 
> When should you use a tool_wrapper or call the qiime script directly?
>   Many of the qiime scripts could probably be called directly, especially if 
> it can be called with arbitary input/output file pathnames.
>   The reasons for using a tool wrapper may be if input/output needs to be 
> manipulated, moved, renamed in order to be used by the qiime script.
>   You'll also need a tool wrapper if the names or number of the output files 
> can not be determined from the parameter settings.
>   ( http://wiki.g2.bx.psu.edu/Admin/Tools/Multiple%20Output%20Files )
>   If your tool relies on a file ext to determine a format, you'll have to 
> rename the input.
>   ( Galaxy dataset pathnames will look something like:  
> //072/dataset_72931.dat )
>   The format/type of a dataset is stored in its metadata, so the tool_config 
> can use that information, especially if a script can take muliple alternative 
> input formats.
>   A tool_wrapper can also be used to manage the stdout or stderr from a tool. 
>   Galaxy currently interprets any output on stderr as a failure.
> 
> 
> 
> A couple changes in galaxy should make somethings easier than when I first 
> attempted this:   
>   - galaxy now accepts dataset requests with sub directories. ( 
> https://bitbucket.org/galaxy/galaxy-central/issue/494/support-sub-dirs-in-extra_files_path-patch
>  )
> That means that output HTML files with links into sub directories can be 
> left intact, with the html copied to the output dataset and the linked files 
> to its "extra_files_path".
>   - if you know the pathname of an output relative to the working directory, 
> galaxy can copy it automatically to the output dataset using the 
> from_work_dir attribute.
> ( see example in:  
> https://bitbucket.org/galaxy/galaxy-central/src/21b645303c02/tools/ngs_rna/tophat_wrapper.xml
>  )
> 
> Datatypes
>   You may want to create new datatypes to make it easier for the user to 
> correctly select inputs to a tool from previous outputs.  
>   For example, the qiime mapping file is a tabular file with specific 
> requirements.  I put a 'qiimemapping' datatype in 
> lib/galaxy/datatypes/metagenomics.py and datatypes_conf.xml 
>   so an input could generate a select list containing only qiimemapping 
> datasets rather than all tabular ones.  
> 
> Generating a configfile
>   You can generate configfiles in the galaxy tool_config .xml file.   The 
> configfile is generated by the Cheetah interpreter just as the commandline is.
>   see:  alpha_rarefaction.xml
> 
> The qiime_wrapper.py was patterned after the mothur_wrapper.py   with some of 
> the same wrapper params to handle run time determined output (perhaps not 
> needed):
>   --galaxy_datasets
>  a comma separated list of regex:output_dataset the wrapper searches 
> the working_dir and copies the file that matches the regex to the outout 
> dataset
>  if the exact pathname is known, use the "from_work_dir" attribute 
> instead
>

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Jim Johnson]

Pat,

That sounds great.   Do one of you want to take ownership of the toolshed 
repository?
At minimum, we should add developers to the list that can push changes.

Thanks,

JJ

On 1/28/12 9:37 AM, Gillevet Patrick wrote:

Jim et al

Amanda has most of the scripts working now and will be putting them up on the 
toolshed.
She will be in touch as soon as the scripts are validated a couple of times 
with different datasets.

cheers...
Pat



On Dec 29, 2011, at 3:02 PM, Jim Johnson wrote:



It is easiest to generate tools for galaxy when the applications or scripts can 
take arbitrarily named input files and generate output to given path names.
Input directories, output directories are very convenient on the command line, 
but more of a challenge when crafting a galaxy tool.
That said, many applications require a wrapper script to work with in galaxy.
Thank you for the consistent script_info[] help/usage syntax in the qiime 
scripts,  which enabled me to generate a skeleton galaxy tool_config file for 
each qiime script.

I had some time last spring to work on integrating qiime into galaxy.
Unfortunately, I haven't had any time since to work on this.
I put those partial results  on the Galaxy Tool Shed: 
http://toolshed.g2.bx.psu.edu/
There's a continuing effort at George Mason University to incorporate qiime 
into galaxy tools, so you may want to ask them what they need.


I started by generating galaxy tool_config files, e.g. align_seqs.xml,  by 
using python to get the script_info[] from the qiime script:

$ cat generate_tool_config.bash
#!/usr/bin/env bash
python $1 > ${1%.*}.help
cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h > 
${1%.*}.log

(I'll attach tool_template.txt )

This generated skeleton tool_config .xml files that I could then edit as needed.
( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax )

I originally was calling all qiime scripts from a tool wrapper:  
qiime_wrapper.py
But, if a script can be called with any input filepaths and write its results 
to any filepaths, and only writes to STDERR when it fails, then you could call 
that script directly.


When should you use a tool_wrapper or call the qiime script directly?
  Many of the qiime scripts could probably be called directly, especially if it 
can be called with arbitary input/output file pathnames.
  The reasons for using a tool wrapper may be if input/output needs to be 
manipulated, moved, renamed in order to be used by the qiime script.
  You'll also need a tool wrapper if the names or number of the output files 
can not be determined from the parameter settings.
  ( http://wiki.g2.bx.psu.edu/Admin/Tools/Multiple%20Output%20Files )
  If your tool relies on a file ext to determine a format, you'll have to 
rename the input.
  ( Galaxy dataset pathnames will look something like:  
//072/dataset_72931.dat )
  The format/type of a dataset is stored in its metadata, so the tool_config 
can use that information, especially if a script can take muliple alternative 
input formats.
  A tool_wrapper can also be used to manage the stdout or stderr from a tool.   
Galaxy currently interprets any output on stderr as a failure.



A couple changes in galaxy should make somethings easier than when I first 
attempted this:
  - galaxy now accepts dataset requests with sub directories. ( 
https://bitbucket.org/galaxy/galaxy-central/issue/494/support-sub-dirs-in-extra_files_path-patch
 )
That means that output HTML files with links into sub directories can be left intact, 
with the html copied to the output dataset and the linked files to its 
"extra_files_path".
  - if you know the pathname of an output relative to the working directory, 
galaxy can copy it automatically to the output dataset using the from_work_dir 
attribute.
( see example in: 
https://bitbucket.org/galaxy/galaxy-central/src/21b645303c02/tools/ngs_rna/tophat_wrapper.xml
 )

Datatypes
  You may want to create new datatypes to make it easier for the user to 
correctly select inputs to a tool from previous outputs.
  For example, the qiime mapping file is a tabular file with specific 
requirements.  I put a 'qiimemapping' datatype in 
lib/galaxy/datatypes/metagenomics.py and datatypes_conf.xml
  so an input could generate a select list containing only qiimemapping 
datasets rather than all tabular ones.

Generating a configfile
  You can generate configfiles in the galaxy tool_config .xml file.   The 
configfile is generated by the Cheetah interpreter just as the commandline is.
  see:  alpha_rarefaction.xml

The qiime_wrapper.py was patterned after the mothur_wrapper.py   with some of 
the same wrapper params to handle run time determined output (perhaps not 
needed):
  --galaxy_datasets
 a comma separated list of regex:output_dataset the wrapper searches 
the working_dir and copies the file that matches the regex to the outout dataset
 if the exact pathname is known, use the "from_work_dir" attribute 
instead
  --g

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Hollander, Mattias de]

Thanks Jim for giving this extensive update on how to improve the Qiime 
wrappers. 

We had contact about the Qiime wrappers before, and I have been using them in 
Galaxy on a weekly basis. Therefore I can say they have been proven to be very 
useful. I mainly edited the parameter and output sections of some configuration 
files, but all still use the wrapper script. The most important scripts are the 
split libraries and the workflow scripts (pick_otus_through_otu_table, 
alpha/beta diversity workflows).

I use the Qiime scripts in a Galaxy Cloudman environment adjusted to a 
multicore server. Setup on EC2 shouldn't be a problem, although you need to 
place the requirement tag for all the needed dependecies on top of the xml file.

I will have a look at the suggestions and I am willing to help to improve Qiime 
support for galaxy either by working on the configuration files or testing them 
out. So please keep me in the loop...

Greetings,

Mattias


-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu on behalf of 
galaxy-dev-requ...@lists.bx.psu.edu
Sent: Sat 12/31/2011 6:00 PM
To: galaxy-dev@lists.bx.psu.edu
Subject: galaxy-dev Digest, Vol 66, Issue 26
 
Send galaxy-dev mailing list submissions to
galaxy-dev@lists.bx.psu.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://lists.bx.psu.edu/listinfo/galaxy-dev
or, via email, send a message with subject or body 'help' to
galaxy-dev-requ...@lists.bx.psu.edu

You can reach the person managing the list at
galaxy-dev-ow...@lists.bx.psu.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of galaxy-dev digest..."


HEY!  This is important!  If you reply to a thread in a digest, please
1. Change the subject of your response from "Galaxy-dev Digest Vol ..." to the 
original subject for the thread.
2. Strip out everything else in the digest that is not part of the thread you 
are responding to.

Why?
1. This will keep the subject meaningful.  People will have some idea from the 
subject line if they should read it or not.
2. Not doing this greatly increases the number of emails that match search 
queries, but that aren't actually informative.

Today's Topics:

   1. Re: Existing efforts to convert the QIIME pipeline to Galaxy?
  (Jim Johnson)


--

Message: 1
Date: Thu, 29 Dec 2011 14:02:04 -0600
From: Jim Johnson 
To: Jose Carlos Clemente ,
"galaxy-dev@lists.bx.psu.edu" 
Cc: Greg Caporaso , Rob Knight
,  Jesse Stombaugh ,
Zuzolo Amanda , Antonio Gonz?lez
,   Gillevet Patrick , 
j...@umn.edu,
    Daniel McDonald 
Subject: Re: [galaxy-dev] Existing efforts to convert the QIIME
pipeline to Galaxy?
Message-ID: <4efcc73c.7010...@umn.edu>
Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"


It is easiest to generate tools for galaxy when the applications or scripts can 
take arbitrarily named input files and generate output to given path names.
Input directories, output directories are very convenient on the command line, 
but more of a challenge when crafting a galaxy tool.
That said, many applications require a wrapper script to work with in galaxy.
Thank you for the consistent script_info[] help/usage syntax in the qiime 
scripts,  which enabled me to generate a skeleton galaxy tool_config file for 
each qiime script.

I had some time last spring to work on integrating qiime into galaxy.
Unfortunately, I haven't had any time since to work on this.
I put those partial results  on the Galaxy Tool Shed:
http://toolshed.g2.bx.psu.edu/
There's a continuing effort at George Mason University to incorporate qiime 
into galaxy tools, so you may want to ask them what they need.


I started by generating galaxy tool_config files, e.g. align_seqs.xml,  by 
using python to get the script_info[] from the qiime script:

$ cat generate_tool_config.bash
#!/usr/bin/env bash
python $1 > ${1%.*}.help
cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h > 
${1%.*}.log

(I'll attach tool_template.txt )

This generated skeleton tool_config .xml files that I could then edit as needed.
( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax )

I originally was calling all qiime scripts from a tool wrapper:  
qiime_wrapper.py
But, if a script can be called with any input filepaths and write its results 
to any filepaths, and only writes to STDERR when it fails, then you could call 
that script directly.


When should you use a tool_wrapper or call the qiime script directly?
   Many of the qiime scripts could probably be called directly, especially if 
it can be called with arbitary input/output file pathnames.
   The reasons for using a tool wrapper may be if input/output needs to be 
manipulated, moved, r

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Edward Kirton]

here's what i have; i started categorizing the tools under labels but hadn't
finished.  i'm also working on getting these tools working in galaxy (albeit
slowly as we won't need them until we get a miseq machine in oct), but maybe
we can be in communication with each other and jjohnson.






































































 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 




On Wed, Jul 27, 2011 at 7:01 AM, Amanda Zuzolo  wrote:

> Hello all,
>
> I'm working on a local instance of Galaxy at George Mason University. We'd
> been looking into integrating Qiime and I've found the toolkit very helpful,
> thanks! I did have one question, though: would you mind uploading the
> tool_conf file or chunk of text with the qiime functions in it? This would
> be helpful to myself, as well as others interested in the toolkit!
>
> Thank you,
>
> Amanda Zuzolo
> Bioengineering Major, George Mason University
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
>  http://lists.bx.psu.edu/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Amanda Zuzolo]

Hello all,

I'm working on a local instance of Galaxy at George Mason University. We'd
been looking into integrating Qiime and I've found the toolkit very helpful,
thanks! I did have one question, though: would you mind uploading the
tool_conf file or chunk of text with the qiime functions in it? This would
be helpful to myself, as well as others interested in the toolkit!

Thank you,

Amanda Zuzolo
Bioengineering Major, George Mason University
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Edward Kirton]

I took a look and it's a great start.  I can work on it 1d/wk.  I'll email
you again after I push my revisions to the toolshed.  Or email me anytime
you want me to push whatever I've got at that time.
Thanks,
Ed

On Sun, Jul 17, 2011 at 8:39 AM, Jim Johnson  wrote:

>
> Ed,
>
> I pushed what code I had to the toolshed repository "qiime"  and gave you
> push permissions.
> I'm not sure when I'll have time to work on it again.
>
> Thanks,
>
> JJ
>
>
>
> On 7/15/11 3:18 PM, Edward Kirton wrote:
>
> fantastic, thanks!
> i'm also available to help out, so perhaps you can find me something to
> work on.
> ed
>
> On Fri, Jul 15, 2011 at 1:09 PM, Jim Johnson  wrote:
>
>>  I haven't had time to work on it for a couple months.
>> I remember getting stuck on the output of some scripts being a HTML with a
>> file hierarchy underneath.
>> I proposed setting the galaxy routes to accept those, and that has since
>> been incorporated.
>>
>> I'll try to do a run through of what I had, and update what was working to
>> the current galaxy-central tip.
>> Then, maybe I'll bundle that and put it in the toolshed for common
>> development.
>>
>> Does that work for you?
>>
>> Thanks,
>>
>> JJ
>>
>>
>> On 7/15/11 1:58 PM, Edward Kirton wrote:
>>
>> Hello James, I was wondering if you're still tackling the task of adding
>> QIIME to galaxy?  I have some users in my group asking about this and I'd
>> rather not duplicate any efforts.  If you're still working on this, do you
>> have a ballpark estimate of when this will appear in the toolshed?
>> Many thanks,
>> Ed Kirton
>> US DOE JGI (http://galaxy.jgi-psf.org)
>>
>>  p.s. great job on the mothur suite; thanks!  i just found it in the
>> toolshed yesterday.
>>
>> On Wed, Nov 24, 2010 at 7:31 AM, Jim Johnson  wrote:
>>
>>>
>>> Tim,
>>>
>>> I hope to also look at incorporating QIIME into our local Galaxy instance
>>> at the University of Minnesota, but probably won't be able to start for a
>>> couple weeks.  It would be good to develop that in coordination with others.
>>>
>>> I just finished incorporating "Mothur" metagenomics suite
>>> http://www.mothur.org/ (Dr. Patrick Schloss,  Department of Microbiology
>>> & Immunology at The University of Michigan) into our Galaxy server at the
>>> University of Minnesota.   I hope to contribute that to
>>> http://community.g2.bx.psu.edu/ after some testing by our researchers.
>>>  If the Galaxy wrappings for Mothur are of any interest to you, I can send
>>> you a copy any time.
>>>
>>> Thanks,
>>>
>>> JJ
>>>
>>> James E Johnson
>>> Minnesota Supercomputing Institute
>>> University of Minnesota
>>>
>>>
>>> On Nov 23, 2010 at 07:22 AM, Tim te Beek wrote:
>>>
>>>  Hello all,

 Is anyone aware of any existing efforts to port the QIIME sequencing
 pipeline (http://qiime.sourceforge.net/) to Galaxy? I would like to run
 QIIME analyses through Galaxy to get better control of
 intermediate processing steps, but before I start to convert (a subset
 of)
 some 90 scripts, I'd first like to make sure this has not been done
 before
 by anyone willing to share their work.

 So: has anyone converted the QIIME pipeline to Galaxy before, and would
 they
 be willing to share their scripts?

 Best regards,
 Tim

>>>
>>> ___
>>> galaxy-dev mailing list
>>> galaxy-dev@lists.bx.psu.edu
>>> http://lists.bx.psu.edu/listinfo/galaxy-dev
>>>
>>
>>
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

[Jim Johnson]

Ed,

I pushed what code I had to the toolshed repository "qiime"  and gave you push 
permissions.
I'm not sure when I'll have time to work on it again.

Thanks,

JJ


On 7/15/11 3:18 PM, Edward Kirton wrote:

fantastic, thanks!
i'm also available to help out, so perhaps you can find me something to work on.
ed

On Fri, Jul 15, 2011 at 1:09 PM, Jim Johnson mailto:johns...@umn.edu>> wrote:

I haven't had time to work on it for a couple months.
I remember getting stuck on the output of some scripts being a HTML with a 
file hierarchy underneath.
I proposed setting the galaxy routes to accept those, and that has since 
been incorporated.

I'll try to do a run through of what I had, and update what was working to 
the current galaxy-central tip.
Then, maybe I'll bundle that and put it in the toolshed for common 
development.

Does that work for you?

Thanks,

JJ


On 7/15/11 1:58 PM, Edward Kirton wrote:

Hello James, I was wondering if you're still tackling the task of adding 
QIIME to galaxy?  I have some users in my group asking about this and I'd 
rather not duplicate any efforts.  If you're still working on this, do you have 
a ballpark estimate of when this will appear in the toolshed?
Many thanks,
Ed Kirton
US DOE JGI (http://galaxy.jgi-psf.org)

p.s. great job on the mothur suite; thanks!  i just found it in the 
toolshed yesterday.

On Wed, Nov 24, 2010 at 7:31 AM, Jim Johnson mailto:johns...@umn.edu>> wrote:


Tim,

I hope to also look at incorporating QIIME into our local Galaxy 
instance at the University of Minnesota, but probably won't be able to start 
for a couple weeks.  It would be good to develop that in coordination with 
others.

I just finished incorporating "Mothur" metagenomics suite 
http://www.mothur.org/ (Dr. Patrick Schloss,  Department of Microbiology & Immunology at 
The University of Michigan) into our Galaxy server at the University of Minnesota.   I hope 
to contribute that to http://community.g2.bx.psu.edu/ after some testing by our researchers.  
If the Galaxy wrappings for Mothur are of any interest to you, I can send you a copy any time.

Thanks,

JJ

James E Johnson
Minnesota Supercomputing Institute
University of Minnesota


On Nov 23, 2010 at 07:22 AM, Tim te Beek wrote:

Hello all,

Is anyone aware of any existing efforts to port the QIIME sequencing
pipeline (http://qiime.sourceforge.net/) to Galaxy? I would like to 
run
QIIME analyses through Galaxy to get better control of
intermediate processing steps, but before I start to convert (a 
subset of)
some 90 scripts, I'd first like to make sure this has not been done 
before
by anyone willing to share their work.

So: has anyone converted the QIIME pipeline to Galaxy before, and 
would they
be willing to share their scripts?

Best regards,
Tim


___
galaxy-dev mailing list
galaxy-dev@lists.bx.psu.edu 
http://lists.bx.psu.edu/listinfo/galaxy-dev







___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/