I wrote something in Python a while back (which I've attached) to parse a
shed_tool_conf.xml file and generate a shell script to install those tools
on a different server. The shell script uses Galaxy's
scripts/api/install_tool_shed_repositories.py to install the tools and ends
up looking something like this:
#!/bin/sh
set -eux
python ./scripts/api/install_tool_shed_repositories.py --local
$GALAXY_INSTALL_URL --api $GALAXY_INSTALL_KEY --tool-deps --repository-deps
--url https://toolshed.g2.bx.psu.edu --owner devteam --name
fasta_to_tabular --revision 9d189d08f2ad --panel-section-id convert
python ./scripts/api/install_tool_shed_repositories.py --local
$GALAXY_INSTALL_URL --api $GALAXY_INSTALL_KEY --tool-deps --repository-deps
--url https://toolshed.g2.bx.psu.edu --owner devteam --name
tabular_to_fasta --revision 0b4e36026794 --panel-section-id convert
... etc
All you have to do is set those two environment variables appropriately and
run the shell script from your Galaxy root directory. I did end up having
to modify our nginx config to increase the proxy read timeout for the /api
location to a much larger value (I used 3600s) to give the tool
dependencies enough time to build, though.
Hope you find it useful.
Cheers,
Brian
On Wed, Nov 23, 2016 at 7:32 AM, Peter Briggs <peter.bri...@manchester.ac.uk
> wrote:
> Hi Peter
>
> Thanks for your comments - at least it doesn't sound like I'm missing the
> "perfect" way to migrate the installed tools. In the longer term moving
> towards bioconda dependency resolution will probably be more stable.
>
> Also re breaking tool repos: even repos that aren't updated on the
> toolshed can break over time, for example if the URL for an executable
> moves, or if an implicit Python dependency is updated and breaks an install
> (these are both things that I'm seeing in my tests).
>
> Thanks again
>
> Best wishes
>
> Peter
>
>
> On 23/11/16 11:40, Peter Cock wrote:
>
>> On Wed, Nov 23, 2016 at 10:49 AM, Peter Briggs
>> <peter.bri...@manchester.ac.uk> wrote:
>>
>>> Hello Evan, Hans-Rudolf
>>>
>>> I'm just in the middle of doing a similar migration for our local
>>> production
>>> server, and Hans-Rudolf's advice seems sound to me. Definitely moving the
>>> core Galaxy server has been relatively straightforward.
>>>
>>> However: for the installed tools, I'm not sure that changing the paths in
>>> the env.sh files is sufficient - in our installation the absolute paths
>>> seemed to be baked into a lot of other files under the
>>> 'tool_dependencies'
>>> directory - including things like compiled files (e.g. static and shared
>>> libraries). So for many of the tools I wouldn't feel confident that they
>>> would still work after the move.
>>>
>>
>> Yes, I've seen that with various third party tool installations (out side
>> of
>> Galaxy), so sadly you cannot in general move the files to a different
>> path :(
>>
>> I don't know if we can do anything about this directly in Galaxy, or even
>> in BioConda?
>>
>> My plan has been to reinstall each of the tools from the toolshed (i.e.
>>> uninstall via the admin interface then reinstall the same tool repository
>>> revision(s) using the API), but I don't feel able to recommend this
>>> approach
>>> either as in my testing this has also had problems - ranging from some
>>> tool
>>> revisions no longer being available, through to more serious issues
>>> (such as
>>> tool dependencies which used to work but since become broken). I figured
>>> I'd
>>> just have to knuckle down and work through each problem as I encountered
>>> it.
>>>
>>
>> One simple reason for this is stale URLs, where Galaxy's cache can help
>> but is another step for tool wrapper authors to do when setting up a new
>> Galaxy dependency: https://github.com/galaxyproject/cargo-port
>>
>> However, even well intentioned Tool Shed updates could also break things
>> :(
>>
>> If anyone else has experiences with these kinds of migrations then I'm
>>> also
>>> very interested to know what worked (and what didn't)!
>>>
>>
>> Sorry - the closest I've been is setting up a new Galaxy server in
>> parallel with our old server, and manually installing "missing" tools
>> via the Tool Shed.
>>
>> Peter
>>
>>
> --
> Peter Briggs peter.bri...@manchester.ac.uk
> Bioinformatics Core Facility University of Manchester
> B.1083 Michael Smith Bldg Tel: (0161) 2751482
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/
>
--
Brian Claywell | programmer/analyst
Matsen Group | matsen.fredhutch.org <http://matsen.fhcrc.org/>
Fred Hutchinson Cancer Research Center
#!/usr/bin/env python
import xml.etree.ElementTree as ET
import argparse
#####
parser = argparse.ArgumentParser(description="""
Generate a tool install script from a shed_tool_conf.xml file.
""")
parser.add_argument('--latest', action='store_true',
help='install latest revisions instead of those specified')
parser.add_argument('tool_conf', help='tool config file')
args = parser.parse_args()
#####
main_sections = { "Get Data": 'getext',
"Send Data": 'send',
"Lift-Over": 'liftOver',
"Text Manipulation": 'textutil',
"Filter and Sort": 'filter',
"Join, Subtract and Group": 'group',
"Convert Formats": 'convert',
"Extract Features": 'features',
"Fetch Sequences": 'fetchSeq',
"Fetch Alignments": 'fetchAlign',
"Statistics": 'stats',
"Graph/Display Data": 'plots' }
cmd_tmpl = ("python ./scripts/api/install_tool_shed_repositories.py "
"--local $GALAXY_INSTALL_URL --api $GALAXY_INSTALL_KEY "
"--tool-deps --repository-deps ")
repo_tmpl = "--url https://{} --owner {} --name {} "
repos_seen = set()
#####
tree = ET.parse(args.tool_conf)
root = tree.getroot()
print "#!/bin/sh\nset -eux\n"
for section in root.iter('section'):
section_exists = False
section_name = section.get('name')
if section_name in main_sections:
section_exists = True
section_id = main_sections[section_name]
else:
# Use Galaxy's method of assigning section IDs from
# galaxy/lib/tool_shed/galaxy_install/tools/tool_panel_manager.py
section_id = str(section_name.lower().replace(' ', '_'))
for tool in section.iter('tool'):
if tool.find('tool_shed') is None:
continue
tool_shed = tool.find('tool_shed').text
owner = tool.find('repository_owner').text
name = tool.find('repository_name').text
revision = tool.find('installed_changeset_revision').text
repo_str = repo_tmpl.format(tool_shed, owner, name)
if not args.latest:
repo_str += "--revision {} ".format(revision)
# It's legal for a tool to be in multiple sections, so append
# the section ID to the repository string for the purpose of
# detecting duplicates.
repo_key = repo_str + section_id
if repo_key in repos_seen:
continue
else:
repos_seen.add(repo_key)
# The first tool in each section will be passed to the install
# script with the --panel-section-name argument to create the
# section. Remaining tools in the section will be passed with
# the --panel-section-id argument.
if section_exists:
repo_str += "--panel-section-id {}".format(section_id)
else:
repo_str += "--panel-section-name \"{}\"".format(section_name)
section_exists = True
print cmd_tmpl + repo_str
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
