Re: [galaxy-dev] environment variables and paths for toolshed tools
On Tue, Feb 19, 2013 at 12:06 PM, Peter Cock wrote: > On Tue, Feb 19, 2013 at 11:39 AM, Greg Von Kuster wrote: >> On Feb 19, 2013, at 5:10 AM, Peter Cock wrote: >> >> Yes, the complete directory hierarchy and content of the original >> repository revision is preserved when it installed into Galaxy. >> >>> Based on a sample of one, it seems the installation process just >>> unpacks the tool shed files (well, probably using hg rather than >>> unpacking a tar-ball), and preserves their relative path structure. >>> If so, I can just use relative paths to find a data file from the tool >>> executable's own location on disk. (This is what I was hoping >>> would be the case - I'm looking for explicit confirmation). >> >> This is correct, hg clone provides this behavior. >> >>> i.e. For the example use case, if motif.py and motif.dat are in the >>> same folder in the Tool Shed upload, they will be in the same folder >>> as each other once installed. That way motif.py can easily locate the >>> data file motif.dat by looking in the same folder as it itself is located. >>> This is actually very simple (provided I can assume the local folder >>> structure is maintained). >> >> Yes, this is correct. > > Excellent - that solves this use-case nicely. > The tool I was talking about is actually a re-implementation of predictNLS from the Rost Lab, which uses regular expressions to look for Nuclear Localization Signals (NLS): https://rostlab.org/owiki/index.php/PredictNLS My Galaxy version is now available here: http://toolshed.g2.bx.psu.edu/view/peterjc/predictnls Touch wood that will automatically install (and comes with a minimal test case too). Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
On Tue, Feb 19, 2013 at 11:39 AM, Greg Von Kuster wrote: > On Feb 19, 2013, at 5:10 AM, Peter Cock wrote: > > Yes, the complete directory hierarchy and content of the original > repository revision is preserved when it installed into Galaxy. > >> Based on a sample of one, it seems the installation process just >> unpacks the tool shed files (well, probably using hg rather than >> unpacking a tar-ball), and preserves their relative path structure. >> If so, I can just use relative paths to find a data file from the tool >> executable's own location on disk. (This is what I was hoping >> would be the case - I'm looking for explicit confirmation). > > This is correct, hg clone provides this behavior. > >> i.e. For the example use case, if motif.py and motif.dat are in the >> same folder in the Tool Shed upload, they will be in the same folder >> as each other once installed. That way motif.py can easily locate the >> data file motif.dat by looking in the same folder as it itself is located. >> This is actually very simple (provided I can assume the local folder >> structure is maintained). > > Yes, this is correct. Excellent - that solves this use-case nicely. >> (I think I was originally on the wrong track by assuming I should >> be using the tool-data folder, which is complicated by not knowing >> where that will existing on disk). > > I think your request is still valid though and we will add this > enhancement for the Galaxy tool component to handle > $REPOSITORY_INSTALL_DIR in tool configs. In the meantime, > it looks like you can work around this missing feature. Please let > me know if you bump into issues. I can still see potential uses for $REPOSITORY_INSTALL_DIR both in the tool XML language, and as an environment variable - but I don't need that for the time being. Thanks Greg, all sorted for now, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
Hi Peter, see below... On Feb 19, 2013, at 5:10 AM, Peter Cock wrote: > On Sat, Feb 16, 2013 at 9:41 PM, Greg Von Kuster wrote: >> Hi Peter, >> >> Thanks for your thoughts on this. I've created the following Trello card >> for this enhancement. >> >> https://trello.com/card/galaxy-tool-enhancement-to-accommodate-repository-install-dir/506338ce32ae458f6d15e4b3/636 >> >> The priority for the tool shed in the area of tool dependencies like this >> has been >> the implementation of features that enable sharing a single dependency across >> multiple tools in separate repositories, so your scenario is not yet >> supported. >> >> This enhancement would actually be as much (or perhaps more) on the Galaxy >> end than the tool shed, I think, so there may be more resources available to >> get >> to it sooner than I can right now. Of course, someone wil get to it as soon >> as >> possible. >> >> Thanks again for all of your feedback and insight on this. >> >> Greg Von Kuster > > Hi Greg, > > I've read http://wiki.galaxyproject.org/InstallingRepositoriesToGalaxy > which is targeted at Galaxy administrators rather than tools authors. > It explains how a revision specific folder is assigned to each installed > tool, but what (if anything) of the Tool's folder structure is preserved? The installation process uses the mercurial API, so the process to install a specific repository changeset_revision ultimately includes the following 2 hg commands. hg clone hg update -r changeset_revision This results in the structure of the installed repository being exactly the same of the specific revision from which the repository was cloned. > > e.g. One of the paths given is: > > .../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/emboss_datatypes/a89163f31369/emboss_datatypes/datatypes_conf.xml > > The prefix is the revision specific path for that tool for that Tool Shed, > > .../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/emboss_datatypes/a89163f31369/ > > Within that, it seems to preserve the emboss_datatypes folders. i.e. > goto http://toolshed.g2.bx.psu.edu/view/devteam/emboss_datatypes > and click browse repository tip files from the top left actions menu, > it has just one file: emboss_datatypes/datatypes_conf.xml Yes, the complete directory hierarchy and content of the original repository revision is preserved when it installed into Galaxy. > > Based on a sample of one, it seems the installation process just > unpacks the tool shed files (well, probably using hg rather than > unpacking a tar-ball), and preserves their relative path structure. > If so, I can just use relative paths to find a data file from the tool > executable's own location on disk. (This is what I was hoping > would be the case - I'm looking for explicit confirmation). This is correct, hg clone provides this behavior. > > i.e. For the example use case, if motif.py and motif.dat are in the > same folder in the Tool Shed upload, they will be in the same folder > as each other once installed. That way motif.py can easily locate the > data file motif.dat by looking in the same folder as it itself is located. > This is actually very simple (provided I can assume the local folder > structure is maintained). Yes, this is correct. > > (I think I was originally on the wrong track by assuming I should > be using the tool-data folder, which is complicated by not knowing > where that will existing on disk). I think your request is still valid though and we will add this enhancement for the Galaxy tool component to handle $REPOSITORY_INSTALL_DIR in tool configs. In the meantime, it looks like you can work around this missing feature. Please let me know if you bump into issues. > > Thanks, > > Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
On Sat, Feb 16, 2013 at 9:41 PM, Greg Von Kuster wrote: > Hi Peter, > > Thanks for your thoughts on this. I've created the following Trello card for > this enhancement. > > https://trello.com/card/galaxy-tool-enhancement-to-accommodate-repository-install-dir/506338ce32ae458f6d15e4b3/636 > > The priority for the tool shed in the area of tool dependencies like this has > been > the implementation of features that enable sharing a single dependency across > multiple tools in separate repositories, so your scenario is not yet > supported. > > This enhancement would actually be as much (or perhaps more) on the Galaxy > end than the tool shed, I think, so there may be more resources available to > get > to it sooner than I can right now. Of course, someone wil get to it as soon > as > possible. > > Thanks again for all of your feedback and insight on this. > > Greg Von Kuster Hi Greg, I've read http://wiki.galaxyproject.org/InstallingRepositoriesToGalaxy which is targeted at Galaxy administrators rather than tools authors. It explains how a revision specific folder is assigned to each installed tool, but what (if anything) of the Tool's folder structure is preserved? e.g. One of the paths given is: .../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/emboss_datatypes/a89163f31369/emboss_datatypes/datatypes_conf.xml The prefix is the revision specific path for that tool for that Tool Shed, .../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/emboss_datatypes/a89163f31369/ Within that, it seems to preserve the emboss_datatypes folders. i.e. goto http://toolshed.g2.bx.psu.edu/view/devteam/emboss_datatypes and click browse repository tip files from the top left actions menu, it has just one file: emboss_datatypes/datatypes_conf.xml Based on a sample of one, it seems the installation process just unpacks the tool shed files (well, probably using hg rather than unpacking a tar-ball), and preserves their relative path structure. If so, I can just use relative paths to find a data file from the tool executable's own location on disk. (This is what I was hoping would be the case - I'm looking for explicit confirmation). i.e. For the example use case, if motif.py and motif.dat are in the same folder in the Tool Shed upload, they will be in the same folder as each other once installed. That way motif.py can easily locate the data file motif.dat by looking in the same folder as it itself is located. This is actually very simple (provided I can assume the local folder structure is maintained). (I think I was originally on the wrong track by assuming I should be using the tool-data folder, which is complicated by not knowing where that will existing on disk). Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
Hi Peter, Thanks for your thoughts on this. I've created the following Trello card for this enhancement. https://trello.com/card/galaxy-tool-enhancement-to-accommodate-repository-install-dir/506338ce32ae458f6d15e4b3/636 The priority for the tool shed in the area of tool dependencies like this has been the implementation of features that enable sharing a single dependency across multiple tools in separate repositories, so your scenario is not yet supported. This enhancement would actually be as much (or perhaps more) on the Galaxy end than the tool shed, I think, so there may be more resources available to get to it sooner than I can right now. Of course, someone wil get to it as soon as possible. Thanks again for all of your feedback and insight on this. Greg Von Kuster On Feb 15, 2013, at 12:07 PM, Peter Cock wrote: > On Fri, Feb 15, 2013 at 4:57 PM, Greg Von Kuster wrote: >> >> >> In this case, your motif.xml file can be configured with the following: >> >> >> motif.py $fasta_file $tabular_file >> >> >> MOTIF_DAT_PATH >> >> >> and your repository will need to include the following tool_dependencies.xml >> file: >> >> >> >> >> > action="set_to">$REPOSITORY_INSTALL_DIR >> >> >> > > That seems overly complicated. Would I not be able to assume that > the motif.xml, motif.py, motif.dat etc are all in the same folder (then > I can just use a relative path to find the data file - nice and easy). > > If not, how about simply defining $REPOSITORY_INSTALL_DIR within > the XML language so I can do this: > > motif.py $fasta_file $tabular_file > $REPOSITORY_INSTALL_DIR/motif.dat > > And/or defining $REPOSITORY_INSTALL_DIR as an environment variable > which I can then use in the script (be it Python, Perl, shell, etc)? > > Thanks, > > Peter > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
On Fri, Feb 15, 2013 at 4:57 PM, Greg Von Kuster wrote: > > On Feb 15, 2013, at 11:45 AM, Peter Cock wrote: >> Actually for the example I am working on, the motif.py script would be >> the tool executable, called from motif.xml like this: >> >> motif.py $fasta_file $tabular_file >> >> i.e. A completely self contained Galaxy tool consisting of four files: >> >> motif.xml - Galaxy tool definition >> motif.txt - README file >> motif.py - Python script which is the tool >> motif.dat - Data file used by the Python script >> >> I am hoping that no further XML configuration files are needed in >> order to handle the data file. >> >> Peter > > In this case, your motif.xml file can be configured with the following: > > > motif.py $fasta_file $tabular_file > > > MOTIF_DAT_PATH > > > and your repository will need to include the following tool_dependencies.xml > file: > > > > > action="set_to">$REPOSITORY_INSTALL_DIR > > > That seems overly complicated. Would I not be able to assume that the motif.xml, motif.py, motif.dat etc are all in the same folder (then I can just use a relative path to find the data file - nice and easy). If not, how about simply defining $REPOSITORY_INSTALL_DIR within the XML language so I can do this: motif.py $fasta_file $tabular_file $REPOSITORY_INSTALL_DIR/motif.dat And/or defining $REPOSITORY_INSTALL_DIR as an environment variable which I can then use in the script (be it Python, Perl, shell, etc)? Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
On Feb 15, 2013, at 11:45 AM, Peter Cock wrote: > On Fri, Feb 15, 2013 at 4:32 PM, Greg Von Kuster wrote: > > Usecase: Tool comes with a simple datafile (e.g. defining a search motif, > say motif.dat) which is used by a script (say motif.py) via a normal > Galaxy tool XML file (say motif.xml). > > Perhaps I can just put my data file next to the script and XML file, in > which case it is easy for the script to locate? But I assumed that > Galaxy best practice would be to use the tool-data folder somehow... > > Thanks, > > Peter >>> >>> Is there a documented example I should be reading? In this use case >>> (without any explicit XML markup), can I assume/have the motif.dat >>> file be installed in the same folder next to the motif.py script and tool >>> defining motif.xml file? >> >> This scenario uses a tool dependency that is included in the repository >> which is discussed in the following section of the tool shed wiki. Using >> this approach a tool config (Cheetah template) would have to be >> configured with the proper tag set and an associated >> tag set that would ultimately have the path to the motif.py >> script and motif.dat file defined in the environment. Of course, this >> assumes the motify.py script is not the tool executable itself. > > Actually for the example I am working on, the motif.py script would be > the tool executable, called from motif.xml like this: > > motif.py $fasta_file $tabular_file > > i.e. A completely self contained Galaxy tool consisting of four files: > > motif.xml - Galaxy tool definition > motif.txt - README file > motif.py - Python script which is the tool > motif.dat - Data file used by the Python script > > I am hoping that no further XML configuration files are needed in > order to handle the data file. > > Peter In this case, your motif.xml file can be configured with the following: motif.py $fasta_file $tabular_file MOTIF_DAT_PATH and your repository will need to include the following tool_dependencies.xml file: $REPOSITORY_INSTALL_DIR ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
On Fri, Feb 15, 2013 at 4:32 PM, Greg Von Kuster wrote: Usecase: Tool comes with a simple datafile (e.g. defining a search motif, say motif.dat) which is used by a script (say motif.py) via a normal Galaxy tool XML file (say motif.xml). Perhaps I can just put my data file next to the script and XML file, in which case it is easy for the script to locate? But I assumed that Galaxy best practice would be to use the tool-data folder somehow... Thanks, Peter >> >> Is there a documented example I should be reading? In this use case >> (without any explicit XML markup), can I assume/have the motif.dat >> file be installed in the same folder next to the motif.py script and tool >> defining motif.xml file? > > This scenario uses a tool dependency that is included in the repository > which is discussed in the following section of the tool shed wiki. Using > this approach a tool config (Cheetah template) would have to be > configured with the proper tag set and an associated > tag set that would ultimately have the path to the motif.py > script and motif.dat file defined in the environment. Of course, this > assumes the motify.py script is not the tool executable itself. Actually for the example I am working on, the motif.py script would be the tool executable, called from motif.xml like this: motif.py $fasta_file $tabular_file i.e. A completely self contained Galaxy tool consisting of four files: motif.xml - Galaxy tool definition motif.txt - README file motif.py - Python script which is the tool motif.dat - Data file used by the Python script I am hoping that no further XML configuration files are needed in order to handle the data file. Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
Hi Peter, On Feb 15, 2013, at 11:14 AM, Peter Cock wrote: > On Fri, Feb 15, 2013 at 4:05 PM, Greg Von Kuster wrote: >>> >>> Hi guys, >>> >>> Reading the rest of the thread from December, Greg's new code did >>> get checked in and should be working (I've not tried this yet). >>> >>> However, I can't help feeling that defining the two environment variables >>> David suggested, $GALAXY_DATA_INDEX_DIR and/or $GALAXY_HOME, >>> would cover the majority of use cases and be far simpler for tool authors >>> to use. Am I overlooking something? >>> >>> Usecase: Tool comes with a simple datafile (e.g. defining a search motif, >>> say motif.dat) which is used by a script (say motif.py) via a normal >>> Galaxy tool XML file (say motif.xml). >>> >>> Perhaps I can just put my data file next to the script and XML file, in >>> which case it is easy for the script to locate? But I assumed that >>> Galaxy best practice would be to use the tool-data folder somehow... >>> >>> Thanks, >>> >>> Peter >> >> The current implementation places your motif.dat data file in the >> repository installation directory, and tools that are properly configured >> to locate dependencies that are included in the repository contents >> will find it. This approach allows for tool dependency discovery and >> easy maintenance when uninstalling repositories as all contents are >> kept where they were installed. Tools that are properly configured >> with tags will find dependencies because the >> associated env.sh file is sourced, providing the location of all >> dependencies to the environment. > > Is there a documented example I should be reading? In this use case > (without any explicit XML markup), can I assume/have the motif.dat > file be installed in the same folder next to the motif.py script and tool > defining motif.xml file? This scenario uses a tool dependency that is included in the repository which is discussed in the following section of the tool shed wiki. Using this approach a tool config (Cheetah template) would have to be configured with the proper tag set and an associated tag set that would ultimately have the path to the motif.py script and motif.dat file defined in the environment. Of course, this assumes the motify.py script is not the tool executable itself. http://wiki.galaxyproject.org/ToolShedToolFeatures#Finding_dependencies_included_in_an_installed_repository_at_tool_execution_time > >> If your motif.dat file is one that would be manually altered by the >> Galaxy administrator over time, perhaps automatically moving it >> to the GALAXY_DATA_INDEX_DIR is justified. However, in this >> case, uninstalling the repository would probably not uninstall the >> motif.dat file. Does this fit with what you are thinking? > > In this case no, I would not expect the motif.dat file to be edited. > > Does this mean GALAXY_DATA_INDEX_DIR and the galaxy > tool-data folder are intended for 'configuration' files the local admin > may need to edit, rather than static tool specific data file? Not necessarily, but with the exception of sample .loc files included in the repository, repository contents should not generally be required to be moved outside of their installation directory. > >> What kinds of files would require knowledge of the GALAXY_HOME >> directory? > > Knowing GALAXY_DATA_INDEX_DIR would probably cover most > cases, so knowing GALAXY_HOME is probably not needed. I believe that the current implementation supports locating all tool dependencies, whether included in the repository or 3-rd party, so I'm not sure introducing GALAXY_DATA_INDEX_DIR to the repository installation process will be beneficial. I can be swayed, however, if I see justification for supporting it. > > Regards, > > Peter > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
On Fri, Feb 15, 2013 at 4:05 PM, Greg Von Kuster wrote: >> >> Hi guys, >> >> Reading the rest of the thread from December, Greg's new code did >> get checked in and should be working (I've not tried this yet). >> >> However, I can't help feeling that defining the two environment variables >> David suggested, $GALAXY_DATA_INDEX_DIR and/or $GALAXY_HOME, >> would cover the majority of use cases and be far simpler for tool authors >> to use. Am I overlooking something? >> >> Usecase: Tool comes with a simple datafile (e.g. defining a search motif, >> say motif.dat) which is used by a script (say motif.py) via a normal >> Galaxy tool XML file (say motif.xml). >> >> Perhaps I can just put my data file next to the script and XML file, in >> which case it is easy for the script to locate? But I assumed that >> Galaxy best practice would be to use the tool-data folder somehow... >> >> Thanks, >> >> Peter > > The current implementation places your motif.dat data file in the > repository installation directory, and tools that are properly configured > to locate dependencies that are included in the repository contents > will find it. This approach allows for tool dependency discovery and > easy maintenance when uninstalling repositories as all contents are > kept where they were installed. Tools that are properly configured > with tags will find dependencies because the > associated env.sh file is sourced, providing the location of all > dependencies to the environment. Is there a documented example I should be reading? In this use case (without any explicit XML markup), can I assume/have the motif.dat file be installed in the same folder next to the motif.py script and tool defining motif.xml file? > If your motif.dat file is one that would be manually altered by the > Galaxy administrator over time, perhaps automatically moving it > to the GALAXY_DATA_INDEX_DIR is justified. However, in this > case, uninstalling the repository would probably not uninstall the > motif.dat file. Does this fit with what you are thinking? In this case no, I would not expect the motif.dat file to be edited. Does this mean GALAXY_DATA_INDEX_DIR and the galaxy tool-data folder are intended for 'configuration' files the local admin may need to edit, rather than static tool specific data file? > What kinds of files would require knowledge of the GALAXY_HOME > directory? Knowing GALAXY_DATA_INDEX_DIR would probably cover most cases, so knowing GALAXY_HOME is probably not needed. Regards, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
Hello Peter, please see my response at the end.
On Feb 15, 2013, at 6:47 AM, Peter Cock wrote:
>>> On Sep 12, 2012, at 1:00 PM, David Hoover wrote:
>>>
>>> Are there any environment variables that are honored by toolshed installed
>>> tools? I tried creating a tool that uses ${GALAXY_DATA_INDEX_DIR} or
>>> $GALAXY_HOME, then uploaded it the test toolshed, then installed it
>>> automatically. Neither of these resolved to what I expected. I don't want
>>> to hard-code the path in the xml tool file, but rather have a default
>>> location for other executables and jar files. How should this best be done?
>>>
>>> David Hoover
>>> Helix Systems Staff
>
> On Wed, Sep 12, 2012 at 6:15 PM, Greg Von Kuster wrote:
>> Hello David,
>>
>> This is not currently possible, but will be available in the next Galaxy
>> distribution release, currently scheduled for about 10 days from now. You
>> will include a tool_dependencies.xml file in your tool shed repository that
>> looks something like the following.
>>
>>
>>
>>
>>> action="set_to">$INSTALL_DIR
>>
>>
>>
>> Your tool will find the required files via the defined JAVA_JAR_PATH by
>> using a tag set in the tool config, something like this:
>>
>>
>>JAVA_JAR_PATH
>>
>>
>> The tag in the tool config would be something like this:
>>
>>
>> python_wrapper.py $JAVA_JAR_PATH/some_file.jar $param1 $param2...
>>
>>
>> I'm close to having this working, so if you are interested in testing before
>> the next Galaxy dist release, let me know and I'll tell you when the Galaxy
>> central repository has the new feature.
>>
>> Thanks!
>>
>> Greg Von Kuster
>
> Hi guys,
>
> Reading the rest of the thread from December, Greg's new code did
> get checked in and should be working (I've not tried this yet).
>
> However, I can't help feeling that defining the two environment variables
> David suggested, $GALAXY_DATA_INDEX_DIR and/or $GALAXY_HOME,
> would cover the majority of use cases and be far simpler for tool authors
> to use. Am I overlooking something?
>
> Usecase: Tool comes with a simple datafile (e.g. defining a search motif,
> say motif.dat) which is used by a script (say motif.py) via a normal
> Galaxy tool XML file (say motif.xml).
>
> Perhaps I can just put my data file next to the script and XML file, in
> which case it is easy for the script to locate? But I assumed that
> Galaxy best practice would be to use the tool-data folder somehow...
>
> Thanks,
>
> Peter
The current implementation places your motif.dat data file in the repository
installation directory, and tools that are properly configured to locate
dependencies that are included in the repository contents will find it. This
approach allows for tool dependency discovery and easy maintenance when
uninstalling repositories as all contents are kept where they were installed.
Tools that are properly configured with tags will find
dependencies because the associated env.sh file is sourced, providing the
location of all dependencies to the environment.
If your motif.dat file is one that would be manually altered by the Galaxy
administrator over time, perhaps automatically moving it to the
GALAXY_DATA_INDEX_DIR is justified. However, in this case, uninstalling the
repository would probably not uninstall the motif.dat file. Does this fit with
what you are thinking?
What kinds of files would require knowledge of the GALAXY_HOME directory?
Thanks Peter,
Greg Von Kuster
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
>> On Sep 12, 2012, at 1:00 PM, David Hoover wrote:
>>
>> Are there any environment variables that are honored by toolshed installed
>> tools? I tried creating a tool that uses ${GALAXY_DATA_INDEX_DIR} or
>> $GALAXY_HOME, then uploaded it the test toolshed, then installed it
>> automatically. Neither of these resolved to what I expected. I don't want
>> to hard-code the path in the xml tool file, but rather have a default
>> location for other executables and jar files. How should this best be done?
>>
>> David Hoover
>> Helix Systems Staff
On Wed, Sep 12, 2012 at 6:15 PM, Greg Von Kuster wrote:
> Hello David,
>
> This is not currently possible, but will be available in the next Galaxy
> distribution release, currently scheduled for about 10 days from now. You
> will include a tool_dependencies.xml file in your tool shed repository that
> looks something like the following.
>
>
>
>
> action="set_to">$INSTALL_DIR
>
>
>
> Your tool will find the required files via the defined JAVA_JAR_PATH by
> using a tag set in the tool config, something like this:
>
>
> JAVA_JAR_PATH
>
>
> The tag in the tool config would be something like this:
>
>
> python_wrapper.py $JAVA_JAR_PATH/some_file.jar $param1 $param2...
>
>
> I'm close to having this working, so if you are interested in testing before
> the next Galaxy dist release, let me know and I'll tell you when the Galaxy
> central repository has the new feature.
>
> Thanks!
>
> Greg Von Kuster
Hi guys,
Reading the rest of the thread from December, Greg's new code did
get checked in and should be working (I've not tried this yet).
However, I can't help feeling that defining the two environment variables
David suggested, $GALAXY_DATA_INDEX_DIR and/or $GALAXY_HOME,
would cover the majority of use cases and be far simpler for tool authors
to use. Am I overlooking something?
Usecase: Tool comes with a simple datafile (e.g. defining a search motif,
say motif.dat) which is used by a script (say motif.py) via a normal
Galaxy tool XML file (say motif.xml).
Perhaps I can just put my data file next to the script and XML file, in
which case it is easy for the script to locate? But I assumed that
Galaxy best practice would be to use the tool-data folder somehow...
Thanks,
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
> On 9/16/12 8:48 AM, Jim Johnson wrote:
>> Greg, Pablo,
>>
>> I'm using a SnpEffect repository 'snpeff_with_dep' in
>> http://testtoolshed.g2.bx.psu.edu/
>> to test using tool_dependencies.xml with the tag.
>> ( You both write access, if you want to use this for correcting my errors or
>> debugging. )
>>
>> I'm getting an error in prepare_for_install:
>>
>> Module galaxy.web.controllers.admin_toolshed:1174 in prepare_for_install
>> >>> tool_dependencies
>> {'set_environment': [{'type': 'environment variable', 'name':
>> 'JAVA_JAR_PATH'}], 'snpEff/3.0f': {'readme': '\n', 'version':
>> '3.0f', 'type': 'package', 'name': 'snpEff'}}
>> ...
>> Module _admin_tool_shed_repository_common_mako:242 in
>> render_render_tool_dependency_section
>> >> name = requirements_dict[ 'name' ]
>> TypeError: list indices must be integers, not str
>>
>> It appears that the requirements_dict for the set_environment tag is
>> enclosed in an array.
>> I haven't yet looked at the repository code to see how that might happen.
>>
>>
>> Pablo,
>>
>> The http://sourceforge.net/projects/snpeff/files/ repository should
>> probably have versions packaged for download named like:
>>
>> http://sourceforge.net/projects/snpeff/files/snpEff_3_0.tgz
>>
>> $ tar ztvf snpEff_3_0.tgz
>> drwxrwxrwx 0 0 0 0 Aug 23 12:17 snpEff_3_0/
>> -rwxrwxrwx 0 0 014298767 Aug 23 12:17 snpEff_3_0/SnpSift.jar
>> ...
>>
>> That will allow versioning within galaxy, and successive versions of
>> tool_dependencies.xml with only need a change to the action tag:
>> download_by_url
>>
>> Thanks,
>>
>> JJ
>>
>>
>>
>> I'm trying to test a repository that use in tool_depen
>>>
>>> Message: 18
>>> Date: Thu, 13 Sep 2012 11:46:32 -0400
>>> From: Greg Von Kuster
>>> To: David Hoover
>>> Cc: galaxy-dev@lists.bx.psu.edu
>>> Subject: Re: [galaxy-dev] environment variables and paths for toolshed
>>> tools
>>> Message-ID:
>>> Content-Type: text/plain; charset="us-ascii"
>>>
>>> Hello David,
>>>
>>> I've just committed change set 7656:6aadac8026cb to the Galaxy central
>>> repository that provides the ability to do what you need here. It would be
>>> great if you could try things out and let me know if you run into any
>>> problems.
>>>
>>> I've been working with nikhil-joshi's deseq_and_sam2counts repository in
>>> the main Galaxy tool shed because his tools require this new feature as
>>> well. This example should provide you with the information you'll need to
>>> tweak your tool shed repository so that your tool dependencies are located
>>> where they get installed rather than attempting to move them to
>>> ${GALAXY_DATA_INDEX_DIR} or some other location.
>>>
>>> Here is the tool_dependency.xml file entry for locating a directory
>>> referred to by an environment variable name R_SCRIPT_PATH.
>>>
>>>
>>>
>>>
>>>
>>> >> action="set_to">$REPOSITORY_INSTALL_DIR
>>>
>>>
>>>
>>>
>>> The tag, is still supported inside tag sets, but
>>> when defined at the xml root level, it will locate dependencies included in
>>> the installed tool shed repository. Of course, the above
>>> tool_dependencies.xml file could of course also include entries for tool
>>> dependencies that are packages, For example:
>>>
>>>
>>>
>>>
>>>
>>> >> action="set_to">$REPOSITORY_INSTALL_DIR
>>>
>>>
>>>
>>>
>>> >> type="download_by_url">http://CRAN.R-project.org/src/base/R-2/R-2.15.1.tar.gz
>>> ./configure
>>> --prefix=$INSTALL_DIR
>>> make
>>>
>>> >> action="prepend_to">$INSTALL_DIR/bin
>>>
>>>
>>>
>>>
>>> You need a FORTRAN compiler or perhaps f2c in additio
Re: [galaxy-dev] environment variables and paths for toolshed tools
Greg,
I think templates/admin/tool_shed_repository/common.mako
needs to bypass the 'set_environment' requirement in the following block.
Thanks,
JJ
$ hg diff templates/admin/tool_shed_repository/common.mako
diff -r 65ecf4e0ed28 templates/admin/tool_shed_repository/common.mako
--- a/templates/admin/tool_shed_repository/common.mako Mon Sep 17 16:24:27
2012 -0400
+++ b/templates/admin/tool_shed_repository/common.mako Mon Sep 17 16:06:52
2012 -0500
@@ -129,28 +129,30 @@
<% package_header_row_displayed = True %>
%endif
%for dependency_key, requirements_dict in
tool_dependencies.items():
-<%
-name = requirements_dict[ 'name' ]
-version = requirements_dict[ 'version' ]
-type = requirements_dict[ 'type' ]
-install_dir = os.path.join(
trans.app.config.tool_dependency_dir,
- name,
- version,
- repository_owner,
- repository_name,
- changeset_revision )
-tool_dependency_readme_text =
requirements_dict.get( 'readme', None )
-%>
-%if not os.path.exists( install_dir ):
-
- ${name}
- ${version}
- ${type}
- ${install_dir}
-
-%if tool_dependency_readme_text:
-${name}
${version} requirements and installation information
-${tool_dependency_readme_text}
+%if not dependency_key == 'set_environment':
+<%
+name = requirements_dict[ 'name' ]
+version = requirements_dict[ 'version'
]
+type = requirements_dict[ 'type' ]
+install_dir = os.path.join(
trans.app.config.tool_dependency_dir,
+ name,
+ version,
+ repository_owner,
+ repository_name,
+ changeset_revision )
+ tool_dependency_readme_text = requirements_dict.get( 'readme', None )
+%>
+%if not os.path.exists( install_dir ):
+
+ ${name}
+ ${version}
+ ${type}
+ ${install_dir}
+
+%if tool_dependency_readme_text:
+${name}
${version} requirements and installation information
+${tool_dependency_readme_text}
+%endif
%endif
%endif
%endfor
On 9/16/12 8:48 AM, Jim Johnson wrote:
Greg, Pablo,
I'm using a SnpEffect repository 'snpeff_with_dep' in
http://testtoolshed.g2.bx.psu.edu/
to test using tool_dependencies.xml with the tag.
( You both write access, if you want to use this for correcting my errors or
debugging. )
I'm getting an error in prepare_for_install:
Modulegalaxy.web.controllers.admin_toolshed:*1174*in|prepare_for_install|
|>>>|||tool_dependencies||
|{'set_environment': [{'type': 'environment variable', 'name':
'JAVA_JAR_PATH'}], 'snpEff/3.0f': {'readme': '\n', 'version': '3.0f',
'type': 'package', 'name': 'snpEff'}}|
...
Module_admin_tool_shed_repository_common_mako:*242*in|render_render_tool_dependency_section|
|>>
<http://localhost:18080/admin_toolshed/prepare_for_install?tool_shed_url=http://testtoolshed.g2.bx.psu.edu/&repository_ids=d01ace51ab1282a5&changeset_revisions=38b3a6b97bd1#>name*=*requirements_dict*[*'name'*]*|
*TypeError: list indices must be integers, not str*
It appears that the requirements_dict for the set_environment tag is enclosed
in an array.
I haven't yet looked at the repository code to see how that might happen.
Pablo,
The http://sourceforge.net/projects/snpeff/files/ repository should probably
have versions packaged for download named like:
http://sourceforge.net/projects/snpeff/files/snpEff_3_0.tgz
$ tar ztvf snpEff_3_0.tgz
drwxrwxrwx 0 0 0 0 Aug 23 12:17 snpEff_3_0/
-rwxrwxrwx 0 0 014298767 Aug 23 12:17 snpEff_3_0/SnpSift.jar
...
That will allow versioning within galaxy, and successive versions of
tool_dependencies.xml with only need a change to the action tag:
do
Re: [galaxy-dev] environment variables and paths for toolshed tools
Greg, Pablo,
I'm using a SnpEffect repository 'snpeff_with_dep' in
http://testtoolshed.g2.bx.psu.edu/
to test using tool_dependencies.xml with the tag.
( You both write access, if you want to use this for correcting my errors or
debugging. )
I'm getting an error in prepare_for_install:
Modulegalaxy.web.controllers.admin_toolshed:*1174*in|prepare_for_install|
|>>>|||tool_dependencies||
|{'set_environment': [{'type': 'environment variable', 'name':
'JAVA_JAR_PATH'}], 'snpEff/3.0f': {'readme': '\n', 'version': '3.0f',
'type': 'package', 'name': 'snpEff'}}|
...
Module_admin_tool_shed_repository_common_mako:*242*in|render_render_tool_dependency_section|
|>>
<http://localhost:18080/admin_toolshed/prepare_for_install?tool_shed_url=http://testtoolshed.g2.bx.psu.edu/&repository_ids=d01ace51ab1282a5&changeset_revisions=38b3a6b97bd1#>name*=*requirements_dict*[*'name'*]*|
*TypeError: list indices must be integers, not str*
It appears that the requirements_dict for the set_environment tag is enclosed
in an array.
I haven't yet looked at the repository code to see how that might happen.
Pablo,
The http://sourceforge.net/projects/snpeff/files/ repository should probably
have versions packaged for download named like:
http://sourceforge.net/projects/snpeff/files/snpEff_3_0.tgz
$ tar ztvf snpEff_3_0.tgz
drwxrwxrwx 0 0 0 0 Aug 23 12:17 snpEff_3_0/
-rwxrwxrwx 0 0 014298767 Aug 23 12:17 snpEff_3_0/SnpSift.jar
...
That will allow versioning within galaxy, and successive versions of
tool_dependencies.xml with only need a change to the action tag: download_by_url
Thanks,
JJ
I'm trying to test a repository that use in tool_depen
Message: 18
Date: Thu, 13 Sep 2012 11:46:32 -0400
From: Greg Von Kuster
To: David Hoover
Cc:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] environment variables and paths for toolshed
tools
Message-ID:
Content-Type: text/plain; charset="us-ascii"
Hello David,
I've just committed change set 7656:6aadac8026cb to the Galaxy central
repository that provides the ability to do what you need here. It would be
great if you could try things out and let me know if you run into any problems.
I've been working with nikhil-joshi's deseq_and_sam2counts repository in the
main Galaxy tool shed because his tools require this new feature as well. This
example should provide you with the information you'll need to tweak your tool
shed repository so that your tool dependencies are located where they get
installed rather than attempting to move them to ${GALAXY_DATA_INDEX_DIR} or
some other location.
Here is the tool_dependency.xml file entry for locating a directory referred to
by an environment variable name R_SCRIPT_PATH.
$REPOSITORY_INSTALL_DIR
The tag, is still supported inside tag sets, but
when defined at the xml root level, it will locate dependencies included in the installed
tool shed repository. Of course, the above tool_dependencies.xml file could of course also
include entries for tool dependencies that are packages, For example:
$REPOSITORY_INSTALL_DIR
http://CRAN.R-project.org/src/base/R-2/R-2.15.1.tar.gz
./configure
--prefix=$INSTALL_DIR
make
$INSTALL_DIR/bin
You need a FORTRAN compiler or perhaps f2c in addition to a C compiler to build
R.
These tool dependency definitions are handled as described in the following
section of the tool shed wiki:
http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_third-party_tool_dependency_installation_and_compilation_with_installed_repositories
So in order for the dependencies to be handled when the repository is installed, they must
be defined in at least one of the tag sets in at least 1 tool config in
the repository. So, the tag set in the deseq.xml tool config file
will look something like this (there are more requirement tags defined here than in the
tool_dependencies.xml fiel above, but you should get the idea).
R_SCRIPT_PATH
R
Bioconductor
DESeq
aroma.light
lattice
The tag set in the deseq.xml tool config is also slightly altered.
Here is the way it looks before using this new approach:
stderr_wrapper.py Rscript ${GALAXY_DATA_INDEX_DIR}/deseq.R $counts $column_types
$comparison $top_table $diagnostic_html "$diagnostic_html.files_path"
"$counts.name"
To use this new feature, the command string now uses the R_SCRIPT_PATH
environment variable ( notice the required ba
Re: [galaxy-dev] environment variables and paths for toolshed tools
Hello David,
I've just committed change set 7656:6aadac8026cb to the Galaxy central
repository that provides the ability to do what you need here. It would be
great if you could try things out and let me know if you run into any problems.
I've been working with nikhil-joshi's deseq_and_sam2counts repository in the
main Galaxy tool shed because his tools require this new feature as well. This
example should provide you with the information you'll need to tweak your tool
shed repository so that your tool dependencies are located where they get
installed rather than attempting to move them to ${GALAXY_DATA_INDEX_DIR} or
some other location.
Here is the tool_dependency.xml file entry for locating a directory referred to
by an environment variable name R_SCRIPT_PATH.
$REPOSITORY_INSTALL_DIR
The tag, is still supported inside tag sets, but
when defined at the xml root level, it will locate dependencies included in the
installed tool shed repository. Of course, the above tool_dependencies.xml
file could of course also include entries for tool dependencies that are
packages, For example:
$REPOSITORY_INSTALL_DIR
http://CRAN.R-project.org/src/base/R-2/R-2.15.1.tar.gz
./configure
--prefix=$INSTALL_DIR
make
$INSTALL_DIR/bin
You need a FORTRAN compiler or perhaps f2c in addition to a C compiler to build
R.
These tool dependency definitions are handled as described in the following
section of the tool shed wiki:
http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_third-party_tool_dependency_installation_and_compilation_with_installed_repositories
So in order for the dependencies to be handled when the repository is
installed, they must be defined in at least one of the tag sets
in at least 1 tool config in the repository. So, the tag set in
the deseq.xml tool config file will look something like this (there are more
requirement tags defined here than in the tool_dependencies.xml fiel above, but
you should get the idea).
R_SCRIPT_PATH
R
Bioconductor
DESeq
aroma.light
lattice
The tag set in the deseq.xml tool config is also slightly altered.
Here is the way it looks before using this new approach:
stderr_wrapper.py Rscript ${GALAXY_DATA_INDEX_DIR}/deseq.R $counts
$column_types $comparison $top_table $diagnostic_html
"$diagnostic_html.files_path" "$counts.name"
To use this new feature, the command string now uses the R_SCRIPT_PATH
environment variable ( notice the required backslash to escape the $ )
stderr_wrapper.py Rscript \$R_SCRIPT_PATH/deseq.R $counts $column_types
$comparison $top_table $diagnostic_html "$diagnostic_html.files_path"
"$counts.name"
When installed from the tool shed, a tool dependency object named R_SCRIPT_PATH
will be created and associated with the installed repository. The dependency
will have a pointer to the env.sh file that is created to set the value of the
R_SCRIPT_PATH environment variable.
Let me know if you bump into any issues in getting this working for your tools.
Thanks!
Greg Von Kuster
On Sep 12, 2012, at 1:00 PM, David Hoover wrote:
> Are there any environment variables that are honored by toolshed installed
> tools? I tried creating a tool that uses ${GALAXY_DATA_INDEX_DIR} or
> $GALAXY_HOME, then uploaded it the test toolshed, then installed it
> automatically. Neither of these resolved to what I expected. I don't want
> to hard-code the path in the xml tool file, but rather have a default
> location for other executables and jar files. How should this best be done?
>
> David Hoover
> Helix Systems Staff
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
> http://lists.bx.psu.edu/
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
Hi David,
With the exception of requiring versions strings in tags in the
tool config in order to enable versioned tool dependency installation (which is
documented in the tool shed wiki at
http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_third-party_tool_dependency_installation_and_compilation_with_installed_repositories),
this is the first change to the way Galaxy tool configs should be written.
I'll get a wiki put together for this new enhancement as soon as possible. I'm
hoping to have it as part of the next Galaxy dist release when the feature is
released.
Greg Von Kuster
On Sep 12, 2012, at 1:47 PM, David Hoover wrote:
> On the topic of toolshed tools, is there any documented guidance about how to
> create a tool for the toolshed? There is some syntactical help on the wiki,
> but no comprehensive guide.
>
> On Sep 12, 2012, at 1:41 PM, Greg Von Kuster wrote:
>
>> It'll be available in Galaxy central sooner than Galaxy dist. I'll send you
>> an email when I commit it.
>>
>> On Sep 12, 2012, at 1:31 PM, David Hoover wrote:
>>
>>> Will it be available through galaxy-dist or galaxy-central? I have started
>>> using the galaxy-central repository just to be on the bleeding edge.
>>>
>>> On Sep 12, 2012, at 1:15 PM, Greg Von Kuster wrote:
>>>
Hello David,
This is not currently possible, but will be available in the next Galaxy
distribution release, currently scheduled for about 10 days from now. You
will include a tool_dependencies.xml file in your tool shed repository
that looks something like the following.
>>> action="set_to">$INSTALL_DIR
Your tool will find the required files via the defined JAVA_JAR_PATH by
using a tag set in the tool config, something like this:
JAVA_JAR_PATH
The tag in the tool config would be something like this:
python_wrapper.py $JAVA_JAR_PATH/some_file.jar $param1 $param2...
I'm close to having this working, so if you are interested in testing
before the next Galaxy dist release, let me know and I'll tell you when
the Galaxy central repository has the new feature.
Thanks!
Greg Von Kuster
On Sep 12, 2012, at 1:00 PM, David Hoover wrote:
> Are there any environment variables that are honored by toolshed
> installed tools? I tried creating a tool that uses
> ${GALAXY_DATA_INDEX_DIR} or $GALAXY_HOME, then uploaded it the test
> toolshed, then installed it automatically. Neither of these resolved to
> what I expected. I don't want to hard-code the path in the xml tool
> file, but rather have a default location for other executables and jar
> files. How should this best be done?
>
> David Hoover
> Helix Systems Staff
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
> http://lists.bx.psu.edu/
>>>
>>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
Hello David,
This is not currently possible, but will be available in the next Galaxy
distribution release, currently scheduled for about 10 days from now. You will
include a tool_dependencies.xml file in your tool shed repository that looks
something like the following.
$INSTALL_DIR
Your tool will find the required files via the defined JAVA_JAR_PATH by using a
tag set in the tool config, something like this:
JAVA_JAR_PATH
The tag in the tool config would be something like this:
python_wrapper.py $JAVA_JAR_PATH/some_file.jar $param1 $param2...
I'm close to having this working, so if you are interested in testing before
the next Galaxy dist release, let me know and I'll tell you when the Galaxy
central repository has the new feature.
Thanks!
Greg Von Kuster
On Sep 12, 2012, at 1:00 PM, David Hoover wrote:
> Are there any environment variables that are honored by toolshed installed
> tools? I tried creating a tool that uses ${GALAXY_DATA_INDEX_DIR} or
> $GALAXY_HOME, then uploaded it the test toolshed, then installed it
> automatically. Neither of these resolved to what I expected. I don't want
> to hard-code the path in the xml tool file, but rather have a default
> location for other executables and jar files. How should this best be done?
>
> David Hoover
> Helix Systems Staff
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
> http://lists.bx.psu.edu/
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
