Re: [galaxy-dev] Dynamically Generated Tools
I didn't mean to imply it was unclear. It definitely made sense. I completely agree with you. However, is there for a way (maybe via the toolshed api) for my tool to push these tools to a TS repo? Also, doesn't that add some more complexity for the user? Not everyone runs a local tool shed. Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotter...@gmail.com (mailto:mepcotter...@gmail.com) mepc...@uga.edu (mailto:mepc...@uga.edu) m...@cs.uga.edu (mailto:m...@cs.uga.edu) http://michaelcotterell.com/ On Wednesday, July 17, 2013 at 10:05 PM, Ross wrote: > Sorry if my post was rambling and unclear. > > Here's the executive summary for what it's worth: > IMHO, a Galaxy tool that generates tools should emit them as tool shed > compatible artefacts, ready for uploading to a TS repository for automated > installation to any target instance. > > > > > > On Thu, Jul 18, 2013 at 11:25 AM, Michael E. Cotterell > mailto:mepcotter...@gmail.com)> wrote: > > Ross, > > > > Thanks for the comments. > > > > I've already made my main tool "tool_shed" compatible. What I'm wondering > > about is what do I do with the tools that my tool creates? > > > > Sincerely, > > Michael E. Cotterell > > > > Ph.D. Student in Computer Science, University of Georgia > > Instructor of Record, Graduate RA & TA, University of Georgia > > Faculty Liaison, CS Graduate Student Association, University of Georgia > > mepcotter...@gmail.com (mailto:mepcotter...@gmail.com) > > (mailto:mepcotter...@gmail.com) > > mepc...@uga.edu (mailto:mepc...@uga.edu) (mailto:mepc...@uga.edu) > > m...@cs.uga.edu (mailto:m...@cs.uga.edu) (mailto:m...@cs.uga.edu) > > http://michaelcotterell.com/ > > > > > > On Wednesday, July 17, 2013 at 7:44 PM, Ross wrote: > > > > > Hi, Michael, > > > > > > As others have said, in the long term, tool wrappers and their > > > dependencies will be distributed and installed through tool sheds rather > > > than being distributed with Galaxy source, so you might want to plan to > > > generate and upload tool shed compatible archives from the get-go. The > > > test and main tool sheds contain plenty of examples and Björn's tools > > > exercise pretty much all of the functionality - you can browse the repo > > > structure which is identical to the gz archive structure you need to > > > upload to a new repo, or download the repo as a gz, unpack it and get > > > exactly the kind of directory structure and contents you need to emulate > > > for your tools. Once you have a working tool packaging the archive up is > > > straightforward. > > > > > > The format for tool shed repo uploads and syntax for the tags used to > > > define dependencies is very well documented in the tool shed section of > > > the wiki. As Björn points out, the tool factory python wrapper might be a > > > useful source of ideas and perhaps code. Your tool generator will need to > > > do something similar to write the content and generate complete tool shed > > > archives. When generating a new tool, the TF uses an ugly XML wrapper > > > generator (contributed improvements would be very welcome!) and (probably > > > more usefully) the few lines of code you need to package up the > > > functional test data and the XML and wrapper if you need one in toolshed > > > archive format. > > > > > > > > > On Thu, Jul 18, 2013 at 5:30 AM, Björn Grüning > > > > > (mailto:bjoern.gruen...@pharmazie.uni-freiburg.de) > > > (mailto:bjoern.gruen...@pharmazie.uni-freiburg.de)> wrote: > > > > Hi Michael, > > > > > > > > I think you will enter new ground with your tool. The closest tool that > > > > will do something similar is Ross toolfactory, I think: > > > > > > > > http://toolshed.g2.bx.psu.edu/view/fubar/toolfactory > > > > > > > > For me one question is, do you really want to offer that service to > > > > every user and is it save? > > > > > > > > > TL;DR I have a tool that generates other tools. Where should those > > > > > tools go, and how should I add them (without modifying Galaxy code)? > > > > > > > > > > I apologize, in advance, for the wall of text. I would like some > > > > > advice on what the best way to go about setting up a tool that > > > > > generates other tools... > > > > > > > > > > As some of you (especially those who saw my boss's talk at GCC) > > > > > already know, I'm working on a tool that lets galaxy users add web > > > > > service operations as tools to Galaxy. > > > > > We have a general purpose client for invoking web services, and we > > > > > have a tool that let's a user enter a WSDL/WADL URL, select the > > > > > operations they want to add, and generates tool config XML files for > > > > > Galaxy. > > > > > > > > > > Right now, I'm in the process of making our tool "tool shed" > > > >
Re: [galaxy-dev] Dynamically Generated Tools
Sorry if my post was rambling and unclear. Here's the executive summary for what it's worth: IMHO, a Galaxy tool that generates tools should emit them as tool shed compatible artefacts, ready for uploading to a TS repository for automated installation to any target instance. On Thu, Jul 18, 2013 at 11:25 AM, Michael E. Cotterell < mepcotter...@gmail.com> wrote: > Ross, > > Thanks for the comments. > > I've already made my main tool "tool_shed" compatible. What I'm wondering > about is what do I do with the tools that my tool creates? > > Sincerely, > Michael E. Cotterell > > Ph.D. Student in Computer Science, University of Georgia > Instructor of Record, Graduate RA & TA, University of Georgia > Faculty Liaison, CS Graduate Student Association, University of Georgia > mepcotter...@gmail.com (mailto:mepcotter...@gmail.com) > mepc...@uga.edu (mailto:mepc...@uga.edu) > m...@cs.uga.edu (mailto:m...@cs.uga.edu) > http://michaelcotterell.com/ > > > On Wednesday, July 17, 2013 at 7:44 PM, Ross wrote: > > > Hi, Michael, > > > > As others have said, in the long term, tool wrappers and their > dependencies will be distributed and installed through tool sheds rather > than being distributed with Galaxy source, so you might want to plan to > generate and upload tool shed compatible archives from the get-go. The test > and main tool sheds contain plenty of examples and Björn's tools exercise > pretty much all of the functionality - you can browse the repo structure > which is identical to the gz archive structure you need to upload to a new > repo, or download the repo as a gz, unpack it and get exactly the kind of > directory structure and contents you need to emulate for your tools. Once > you have a working tool packaging the archive up is straightforward. > > > > The format for tool shed repo uploads and syntax for the tags used to > define dependencies is very well documented in the tool shed section of the > wiki. As Björn points out, the tool factory python wrapper might be a > useful source of ideas and perhaps code. Your tool generator will need to > do something similar to write the content and generate complete tool shed > archives. When generating a new tool, the TF uses an ugly XML wrapper > generator (contributed improvements would be very welcome!) and (probably > more usefully) the few lines of code you need to package up the functional > test data and the XML and wrapper if you need one in toolshed archive > format. > > > > > > On Thu, Jul 18, 2013 at 5:30 AM, Björn Grüning < > bjoern.gruen...@pharmazie.uni-freiburg.de (mailto: > bjoern.gruen...@pharmazie.uni-freiburg.de)> wrote: > > > Hi Michael, > > > > > > I think you will enter new ground with your tool. The closest tool that > > > will do something similar is Ross toolfactory, I think: > > > > > > http://toolshed.g2.bx.psu.edu/view/fubar/toolfactory > > > > > > For me one question is, do you really want to offer that service to > > > every user and is it save? > > > > > > > TL;DR I have a tool that generates other tools. Where should those > tools go, and how should I add them (without modifying Galaxy code)? > > > > > > > > I apologize, in advance, for the wall of text. I would like some > advice on what the best way to go about setting up a tool that generates > other tools... > > > > > > > > As some of you (especially those who saw my boss's talk at GCC) > already know, I'm working on a tool that lets galaxy users add web service > operations as tools to Galaxy. > > > > We have a general purpose client for invoking web services, and we > have a tool that let's a user enter a WSDL/WADL URL, select the operations > they want to add, and generates tool config XML files for Galaxy. > > > > > > > > Right now, I'm in the process of making our tool "tool shed" > compatible, and that's going along great > > > > (I especially like the tool dependency installation stuff), but I'm > trying to determine the best course of action on the following: > > > > > > > > 1. Where is the best place to put the tool config (XML) files that > are generated by my tool? My predecessors just placed them in a > subdirectory under our tool's directory. > > > > I'm ok with this, but does that mean that all the generated files > will get removed if they upgrade our tool via the tool shed (haven't taken > the time to see if > > > > Galaxy just does an "hg pull" or if it's more complicated than that)? > > > > > > > > > It is mainly a hg pull. But if you deinstall that repository and > > > reinstall it, all your files are lost. > > > > > > > Would it be better to have my tool create a local tool shed > repository > > > > (just for these generated tools), and if so, can the tool shed API > let my tool install this repository? Having not looked at > install_tool_shed_repositories.py yet, > > > > it's unclear from the wiki documentation whether or not POST > /api/tool_shed_repositories/install_repository_revision requires that the > tool shed already registered in tool_sheds
Re: [galaxy-dev] Dynamically Generated Tools
Ross, Thanks for the comments. I've already made my main tool "tool_shed" compatible. What I'm wondering about is what do I do with the tools that my tool creates? Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Faculty Liaison, CS Graduate Student Association, University of Georgia mepcotter...@gmail.com (mailto:mepcotter...@gmail.com) mepc...@uga.edu (mailto:mepc...@uga.edu) m...@cs.uga.edu (mailto:m...@cs.uga.edu) http://michaelcotterell.com/ On Wednesday, July 17, 2013 at 7:44 PM, Ross wrote: > Hi, Michael, > > As others have said, in the long term, tool wrappers and their dependencies > will be distributed and installed through tool sheds rather than being > distributed with Galaxy source, so you might want to plan to generate and > upload tool shed compatible archives from the get-go. The test and main tool > sheds contain plenty of examples and Björn's tools exercise pretty much all > of the functionality - you can browse the repo structure which is identical > to the gz archive structure you need to upload to a new repo, or download the > repo as a gz, unpack it and get exactly the kind of directory structure and > contents you need to emulate for your tools. Once you have a working tool > packaging the archive up is straightforward. > > The format for tool shed repo uploads and syntax for the tags used to define > dependencies is very well documented in the tool shed section of the wiki. As > Björn points out, the tool factory python wrapper might be a useful source of > ideas and perhaps code. Your tool generator will need to do something similar > to write the content and generate complete tool shed archives. When > generating a new tool, the TF uses an ugly XML wrapper generator (contributed > improvements would be very welcome!) and (probably more usefully) the few > lines of code you need to package up the functional test data and the XML and > wrapper if you need one in toolshed archive format. > > > On Thu, Jul 18, 2013 at 5:30 AM, Björn Grüning > (mailto:bjoern.gruen...@pharmazie.uni-freiburg.de)> wrote: > > Hi Michael, > > > > I think you will enter new ground with your tool. The closest tool that > > will do something similar is Ross toolfactory, I think: > > > > http://toolshed.g2.bx.psu.edu/view/fubar/toolfactory > > > > For me one question is, do you really want to offer that service to > > every user and is it save? > > > > > TL;DR I have a tool that generates other tools. Where should those tools > > > go, and how should I add them (without modifying Galaxy code)? > > > > > > I apologize, in advance, for the wall of text. I would like some advice > > > on what the best way to go about setting up a tool that generates other > > > tools... > > > > > > As some of you (especially those who saw my boss's talk at GCC) already > > > know, I'm working on a tool that lets galaxy users add web service > > > operations as tools to Galaxy. > > > We have a general purpose client for invoking web services, and we have a > > > tool that let's a user enter a WSDL/WADL URL, select the operations they > > > want to add, and generates tool config XML files for Galaxy. > > > > > > Right now, I'm in the process of making our tool "tool shed" compatible, > > > and that's going along great > > > (I especially like the tool dependency installation stuff), but I'm > > > trying to determine the best course of action on the following: > > > > > > 1. Where is the best place to put the tool config (XML) files that are > > > generated by my tool? My predecessors just placed them in a subdirectory > > > under our tool's directory. > > > I'm ok with this, but does that mean that all the generated files will > > > get removed if they upgrade our tool via the tool shed (haven't taken the > > > time to see if > > > Galaxy just does an "hg pull" or if it's more complicated than that)? > > > > > > It is mainly a hg pull. But if you deinstall that repository and > > reinstall it, all your files are lost. > > > > > Would it be better to have my tool create a local tool shed repository > > > (just for these generated tools), and if so, can the tool shed API let my > > > tool install this repository? Having not looked at > > > install_tool_shed_repositories.py yet, > > > it's unclear from the wiki documentation whether or not POST > > > /api/tool_shed_repositories/install_repository_revision requires that the > > > tool shed already registered in tool_sheds_conf.xml)? > > > > > > I think that is a better way to do it. The API can be used to install > > your tools. But it need to be configured beforehand from the user, > > administrator. > > > > > 2. What's the best way to add the generated tools to Galaxy? My > > > predecessors modified the tool_config.xml directly and required users to > > > restart Galaxy. > > > Now, If these tools
Re: [galaxy-dev] Dynamically Generated Tools
Hi, Michael, As others have said, in the long term, tool wrappers and their dependencies will be distributed and installed through tool sheds rather than being distributed with Galaxy source, so you might want to plan to generate and upload tool shed compatible archives from the get-go. The test and main tool sheds contain plenty of examples and Björn's tools exercise pretty much all of the functionality - you can browse the repo structure which is identical to the gz archive structure you need to upload to a new repo, or download the repo as a gz, unpack it and get exactly the kind of directory structure and contents you need to emulate for your tools. Once you have a working tool packaging the archive up is straightforward. The format for tool shed repo uploads and syntax for the tags used to define dependencies is very well documented in the tool shed section of the wiki. As Björn points out, the tool factory python wrapper might be a useful source of ideas and perhaps code. Your tool generator will need to do something similar to write the content and generate complete tool shed archives. When generating a new tool, the TF uses an ugly XML wrapper generator (contributed improvements would be very welcome!) and (probably more usefully) the few lines of code you need to package up the functional test data and the XML and wrapper if you need one in toolshed archive format. On Thu, Jul 18, 2013 at 5:30 AM, Björn Grüning < bjoern.gruen...@pharmazie.uni-freiburg.de> wrote: > Hi Michael, > > I think you will enter new ground with your tool. The closest tool that > will do something similar is Ross toolfactory, I think: > > http://toolshed.g2.bx.psu.edu/view/fubar/toolfactory > > For me one question is, do you really want to offer that service to > every user and is it save? > > > TL;DR I have a tool that generates other tools. Where should those tools > go, and how should I add them (without modifying Galaxy code)? > > > > I apologize, in advance, for the wall of text. I would like some advice > on what the best way to go about setting up a tool that generates other > tools... > > > > As some of you (especially those who saw my boss's talk at GCC) already > know, I'm working on a tool that lets galaxy users add web service > operations as tools to Galaxy. > > We have a general purpose client for invoking web services, and we have > a tool that let's a user enter a WSDL/WADL URL, select the operations they > want to add, and generates tool config XML files for Galaxy. > > > > Right now, I'm in the process of making our tool "tool shed" compatible, > and that's going along great > > (I especially like the tool dependency installation stuff), but I'm > trying to determine the best course of action on the following: > > > > 1. Where is the best place to put the tool config (XML) files that are > generated by my tool? My predecessors just placed them in a subdirectory > under our tool's directory. > > I'm ok with this, but does that mean that all the generated files will > get removed if they upgrade our tool via the tool shed (haven't taken the > time to see if > > Galaxy just does an "hg pull" or if it's more complicated than that)? > > It is mainly a hg pull. But if you deinstall that repository and > reinstall it, all your files are lost. > > > Would it be better to have my tool create a local tool shed repository > > (just for these generated tools), and if so, can the tool shed API let > my tool install this repository? Having not looked at > install_tool_shed_repositories.py yet, > > it's unclear from the wiki documentation whether or not POST > /api/tool_shed_repositories/install_repository_revision requires that the > tool shed already registered in tool_sheds_conf.xml)? > > I think that is a better way to do it. The API can be used to install > your tools. But it need to be configured beforehand from the user, > administrator. > > > 2. What's the best way to add the generated tools to Galaxy? My > predecessors modified the tool_config.xml directly and required users to > restart Galaxy. > > Now, If these tools are in a tool shed repo, I know that I can use the > add_to_tool_panel() method in lib/tool_shed/util/tool_util.py if these > tools are in a tool shed repo, however, > > if that's not the recommended course of action, then I'd like some > advice on what to do? > > There is a feature request, that you can specify in your > toolshed-repository the preferred location in the tool panel. For now I > would ignore it and just install the tools somewhere and hope that > feature request is quiet high in Gregs ToDo list :) > > Cheers, > Bjoern > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Dynamically Generated Tools
Hi Michael, I think you will enter new ground with your tool. The closest tool that will do something similar is Ross toolfactory, I think: http://toolshed.g2.bx.psu.edu/view/fubar/toolfactory For me one question is, do you really want to offer that service to every user and is it save? > TL;DR I have a tool that generates other tools. Where should those tools go, > and how should I add them (without modifying Galaxy code)? > > I apologize, in advance, for the wall of text. I would like some advice on > what the best way to go about setting up a tool that generates other tools... > > As some of you (especially those who saw my boss's talk at GCC) already know, > I'm working on a tool that lets galaxy users add web service operations as > tools to Galaxy. > We have a general purpose client for invoking web services, and we have a > tool that let's a user enter a WSDL/WADL URL, select the operations they want > to add, and generates tool config XML files for Galaxy. > > Right now, I'm in the process of making our tool "tool shed" compatible, and > that's going along great > (I especially like the tool dependency installation stuff), but I'm trying to > determine the best course of action on the following: > > 1. Where is the best place to put the tool config (XML) files that are > generated by my tool? My predecessors just placed them in a subdirectory > under our tool's directory. > I'm ok with this, but does that mean that all the generated files will get > removed if they upgrade our tool via the tool shed (haven't taken the time to > see if > Galaxy just does an "hg pull" or if it's more complicated than that)? It is mainly a hg pull. But if you deinstall that repository and reinstall it, all your files are lost. > Would it be better to have my tool create a local tool shed repository > (just for these generated tools), and if so, can the tool shed API let my > tool install this repository? Having not looked at > install_tool_shed_repositories.py yet, > it's unclear from the wiki documentation whether or not POST > /api/tool_shed_repositories/install_repository_revision requires that the > tool shed already registered in tool_sheds_conf.xml)? I think that is a better way to do it. The API can be used to install your tools. But it need to be configured beforehand from the user, administrator. > 2. What's the best way to add the generated tools to Galaxy? My predecessors > modified the tool_config.xml directly and required users to restart Galaxy. > Now, If these tools are in a tool shed repo, I know that I can use the > add_to_tool_panel() method in lib/tool_shed/util/tool_util.py if these tools > are in a tool shed repo, however, > if that's not the recommended course of action, then I'd like some advice on > what to do? There is a feature request, that you can specify in your toolshed-repository the preferred location in the tool panel. For now I would ignore it and just install the tools somewhere and hope that feature request is quiet high in Gregs ToDo list :) Cheers, Bjoern > Thanks! > > Sincerely, > Michael E. Cotterell > > Ph.D. Student in Computer Science, University of Georgia > Instructor of Record, Graduate RA & TA, University of Georgia > Faculty Liaison, CS Graduate Student Association, University of Georgia > mepcotter...@gmail.com (mailto:mepcotter...@gmail.com) > mepc...@uga.edu (mailto:mepc...@uga.edu) > m...@cs.uga.edu (mailto:m...@cs.uga.edu) > http://michaelcotterell.com/ > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Dynamically Generated Tools
On Wed, Jul 17, 2013 at 4:58 PM, Michael E. Cotterell wrote: > TL;DR I have a tool that generates other tools. Where should those tools > go, and how should I add them (without modifying Galaxy code)? > > I apologize, in advance, for the wall of text. I would like some advice on > what the best way to go about setting up a tool that generates other > tools... > > As some of you (especially those who saw my boss's talk at GCC) already > know, I'm working on a tool that lets galaxy users add web service > operations as tools to Galaxy. We have a general purpose client for invoking > web services, and we have a tool that let's a user enter a WSDL/WADL URL, > select the operations they want to add, and generates tool config XML files > for Galaxy. For those not at GCC2013, this was the talk presented by Jessica C. Kissinger, "Extension of Galaxy to Utilize Web Services and A Semantic Suggestion Engine" Abstract: http://wiki.galaxyproject.org/Events/GCC2013/Abstracts#Events.2FGCC2013.2FAbstracts.2FTalks.Extension_of_Galaxy_to_Utilize_Web_Services_and_A_Semantic_Suggestion_Engine Slides: http://wiki.galaxyproject.org/Documents/Presentations/GCC2013?action=AttachFile&do=view&target=KissingerWebServices.pdf Regards, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
