Re: [galaxy-dev] Comment on Training Documentation

2015-03-15 Thread Dooley, Damion
Darn, last week I wrote  I'll let trainees know that incrementing a tool 
version number leads to a separately selectable tool version to run.  To keep 
an up-to-date Galaxy install is to engage the admin to be bringing in these 
tool versions/choices more frequently even though they functionally don't do 
anything different. 

That second sentence was meant just to be about devs uploading cosmetic changes 
of their tool to the main toolshed.  Something to discourage.

Damion

Hsiao lab, BC Public Health Microbiology  Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Comment on Training Documentation

2015-03-13 Thread Dooley, Damion
Hi,

Ok, thanks for all the quick feedback folks, and your tool links John.  So much 
work rolling off everyone's presses!

John, on command and nixing interpreter= : Ok, I like that, to change it so 
that people can see command tag is a place for shell interaction. 

Re. Update / Revision distinction:
So looking at this in detail (Peter and Martin ;) ) - it turns out the update 
and revision terms are actually from Install latest revision vs Get 
updates options that occur in the Galaxy interface (you can see from snapshots 
I did of galaxy admin view in tool versioning tutorial) when downloading tool 
updates.  So my definitions were an attempt to explain the difference there.
Perhaps then we should have a ticket to have Galaxy force any toolshed update 
to have a version id that increments, then no opportunity for confusion, and no 
need for update phrase in galaxy UI? 

In tutorial I do want to show people the implications of not incrementing. ... 
I'll let trainees know that incrementing a tool version number leads to a 
separately selectable tool version to run.  To keep an up-to-date Galaxy 
install is to engage the admin to be bringing in these tool versions/choices 
more frequently even though they functionally don't do anything different.  
Advice is to encourage the dev to batch their changes into just a few revisions 
per year say for main Galaxy toolshed.  I've downgraded this whole update 
thing to just say its possible to upload a change without version increment but 
not advisable for main toolshed.  

Cheers,

Damion

P.s. for the ffp_phylogeny tool I wanted to do piping in the command tag  
e.g. commandprogx -a | progy -b -c | progz/command.  Easy enough unless 
one wants to catch errors when they happen within the piped processes.  I found 
that verbatim one would unfortunately never see any error info from earlier 
stages of the pipe on return to Galaxy, right?.  I took a stab at writing a 
check_output(command) call in ffp_phylogeny tool that sets up error reporting 
for each stage of pipe, which seems to do ok (though only tested manually), but 
wondering if anyone else has tackled the piping error reporting?


Hsiao lab, BC Public Health Microbiology  Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

From: John Chilton [jmchil...@gmail.com]
Sent: Friday, March 13, 2015 7:56 AM
To: Peter Cock
Cc: Martin Čech; galaxy-...@lists.bx.psu.edu; Hsiao, William; Dooley, Damion
Subject: Re: [galaxy-dev] Comment on Training Documentation

Awesome work Damion - thanks for publishing and sharing these! I hope
the workshop goes well.

I recently created a wiki page just with tool development resource links

https://wiki.galaxyproject.org/Develop/ResourcesTools

that now gets embedded in other more common landing pages including

https://wiki.galaxyproject.org/Admin/Tools/AddToolTutorial
https://wiki.galaxyproject.org/Develop

I have added links to your content from to this resource list.

One quick thought on the content. The interpreter=python idiom for
wrappers has been around for years and is used widely but it has some
serious drawbacks - in particular nothing can come before the wrapper
in the tool XML - no cheetah directives like #import or #set and no
linking in files (for instance the following setup idiom:

ln -s ${input_bam} temp_input.bam 
ln -s ${input_bam.metadata.bam_index} temp_input.bam.bai 
actual_command --args

In the planemo documentation I am not going to mention interpreter for
this reason - I think it is frustrating for people when it does not
work the way intuitively it should. The forthcoming 15.03 injects a
new variable called $__tool_directory__ that I think should be the new
best practice.

ln -s ${input_bam} temp_input.bam 
ln -s ${input_bam.metadata.bam_index} temp_input.bam.bai 
python $__tool_directory__/wrapper.py --args

I understand that it might be best to not base your tutorial on
features not even released yet :) - but I did want to take the
opportunity to mention this idiom.

As you mentioned I have also been working on updated tool development
documentation lately at (http://planemo.readthedocs.org/en/latest/). I
am trying to build a set of smaller resources that can be composed
into tailored resources (e.g. build tools without planemo, with
planemo, with planemo virtual appliance), (normal tutorial, verbose
tutorial, slide-based tutorial), (with and without Docker). I don't
know if rst is up to this task, but we will see.

Another effort along these lines worth checking out is Kyle's tool
tutorial - https://github.com/kellrott/galaxy-tooldev-docs/tree/master/docs.
This tutorial is focused on using the planemo appliance and
Dockerizing tools.

-John

On Fri, Mar 13, 2015 at 4:35 AM, Peter Cock p.j.a.c...@googlemail.com wrote:
 On Fri, Mar 13, 2015 at 12:23 AM, Martin Čech mar...@bx.psu.edu wrote:
 Your tutorial looks great Damion! Thank you

Re: [galaxy-dev] Comment on Training Documentation

2015-03-13 Thread Peter Cock
On Fri, Mar 13, 2015 at 12:23 AM, Martin Čech mar...@bx.psu.edu wrote:
 Your tutorial looks great Damion! Thank you for sharing.

 I am not sure about the vocabulary you are trying to estabilish though
 (update vs revision). I understand what you mean but I would just stick with
 'If the tool changed behavior you have to bump the version.' No need to
 specifically name the steps as it can cause confusion (revision has a
 different meaning in version control world).

I was also confused about your (Damion's) update vs revision terminology,
and agree with Martin that any behaviour change (especially bug fixes,
even minor ones) should come with a version change.

I've not had time to look at your slides, but thanks for sharing them.

Peter
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Comment on Training Documentation

2015-03-13 Thread John Chilton
Awesome work Damion - thanks for publishing and sharing these! I hope
the workshop goes well.

I recently created a wiki page just with tool development resource links

https://wiki.galaxyproject.org/Develop/ResourcesTools

that now gets embedded in other more common landing pages including

https://wiki.galaxyproject.org/Admin/Tools/AddToolTutorial
https://wiki.galaxyproject.org/Develop

I have added links to your content from to this resource list.

One quick thought on the content. The interpreter=python idiom for
wrappers has been around for years and is used widely but it has some
serious drawbacks - in particular nothing can come before the wrapper
in the tool XML - no cheetah directives like #import or #set and no
linking in files (for instance the following setup idiom:

ln -s ${input_bam} temp_input.bam 
ln -s ${input_bam.metadata.bam_index} temp_input.bam.bai 
actual_command --args

In the planemo documentation I am not going to mention interpreter for
this reason - I think it is frustrating for people when it does not
work the way intuitively it should. The forthcoming 15.03 injects a
new variable called $__tool_directory__ that I think should be the new
best practice.

ln -s ${input_bam} temp_input.bam 
ln -s ${input_bam.metadata.bam_index} temp_input.bam.bai 
python $__tool_directory__/wrapper.py --args

I understand that it might be best to not base your tutorial on
features not even released yet :) - but I did want to take the
opportunity to mention this idiom.

As you mentioned I have also been working on updated tool development
documentation lately at (http://planemo.readthedocs.org/en/latest/). I
am trying to build a set of smaller resources that can be composed
into tailored resources (e.g. build tools without planemo, with
planemo, with planemo virtual appliance), (normal tutorial, verbose
tutorial, slide-based tutorial), (with and without Docker). I don't
know if rst is up to this task, but we will see.

Another effort along these lines worth checking out is Kyle's tool
tutorial - https://github.com/kellrott/galaxy-tooldev-docs/tree/master/docs.
This tutorial is focused on using the planemo appliance and
Dockerizing tools.

-John

On Fri, Mar 13, 2015 at 4:35 AM, Peter Cock p.j.a.c...@googlemail.com wrote:
 On Fri, Mar 13, 2015 at 12:23 AM, Martin Čech mar...@bx.psu.edu wrote:
 Your tutorial looks great Damion! Thank you for sharing.

 I am not sure about the vocabulary you are trying to estabilish though
 (update vs revision). I understand what you mean but I would just stick with
 'If the tool changed behavior you have to bump the version.' No need to
 specifically name the steps as it can cause confusion (revision has a
 different meaning in version control world).

 I was also confused about your (Damion's) update vs revision terminology,
 and agree with Martin that any behaviour change (especially bug fixes,
 even minor ones) should come with a version change.

 I've not had time to look at your slides, but thanks for sharing them.

 Peter
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Comment on Training Documentation

2015-03-12 Thread Martin Čech
Your tutorial looks great Damion! Thank you for sharing.

I am not sure about the vocabulary you are trying to estabilish though
(update vs revision). I understand what you mean but I would just stick
with 'If the tool changed behavior you have to bump the version.' No need
to specifically name the steps as it can cause confusion (revision has a
different meaning in version control world).

The side fact is, that even if you do not raise the version of the tool
while uploading, the Tool Shed will try to detect whether there were any
behavior-altering changes. In case there were it will generate new
'installable revision' (not an update to the latest one).

Planemo is a nice tool for tooldev purposes and I am glad you found it.

Other good resources for tooldevs are the IUC and devteam tool
repositories: https://github.com/galaxyproject/tools-iuc
https://github.com/galaxyproject/tools-devteam that are full of high
quality tools.

Also developing tools openly in a GitHub repo is a recommended way and
deeper integration with Tool Shed is a feature to come.

Thank you very much for using Galaxy

Regards

Martin

On Thu, Mar 12, 2015 at 7:27 PM Dooley, Damion damion.doo...@bccdc.ca
wrote:

 I've prepared a few slide-decks for an in-house Galaxy tool-development
 workshop, available at https://github.com/Public-
 Health-Bioinformatics/galaxy-tool-tutorials in pdf and libre office odp
 format.  I thought others might appreciate them or have feedback.  I see
 now the https://planemo.readthedocs.org/en/latest/writing_standalone.html
 tutorial which will be a good resource for us too!

 In the Galaxy tool Versioning deck devoted to making people aware of
 what versioning means for tools, I said:

 An update: a cosmetic change or functional enhancement that doesn't
 affect the previous output behaviour of the tool.  (This can also include
 bug fixes that remedy previously useless output.)

 A revision: a functional enhancement that changes the current output of
 the tool, or marks significant enhancements.

 So is that agreeable?  I was trying to get at the fact that Galaxy will
 let you upload code changes without necessarily incrementing the tool's
 version #, so thinking of the allowable cases for that update situation.

 Damion

 Hsiao lab, BC Public Health Microbiology  Reference Laboratory, BC Centre
 for Disease Control
 655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/