Re: [galaxy-dev] Configurable toolshed package installation to support tool-dependency-resolver-plugins
I think that the best approach for this is an additional ToolDependency status as John proposed a while back as this approach allows for the flexibility needed to support just about anything users may want to do. I haven't had a chance to look at this yet, and I'm currently working on a couple of high priority items. But I can probably take a looke at this probably within the next couple of weeks. If there is an example tool dependency packafge that you have that uses modules, it would really help me with this implementation. Of course, if you want to tak ethis John, it would be much appreciated as well. Greg Von Kuster On Nov 5, 2013, at 12:03 AM, John Chilton chil...@msi.umn.edu wrote: Simon, As you have probably noticed a new stable galaxy was released. It includes 95% of what we discussed including this implicit check to see if tool shed packages are enabled. Your help implementing, testing, and driving these changes was greatly appreciated! I couldn't however pull the trigger and mark packages resolved via modules as Installed - so they will still appear to be in an error state (though your check is in there and they won't attempt to be installed, they will be just marked as errors). The upshot is you can change this one line of code in your Galaxy instance to get the behavior you desire (patch attached). The reason I don't want to mark these packages as Installed is that I am worried about Galaxy deployments that maybe want to use modules are first but transition to tool shed packages down the road. I am unsure what will happen if things are marked as Installed even if no files corresponding to the installation exist. I think the state NEVER_INSTALLED may be preferable - but I need to understand more about what that means. For your own instance, if you are certainly committed to using modules and not using the tool shed - it should be easy to apply the above patch. Is this a fair compromise for the time being? Until I can resolve this last issue, the Trello card remains open, but it should now be quite trivial to modify Galaxy to get the behavior you desire and hopefully this can serve as a model for how others can hook in other dependency resolution mechanisms. I would be eager to hear how this experiment progresses and how you feel about the implementation. Thanks for your contributions, -John On Mon, Oct 14, 2013 at 8:33 PM, Guest, Simon simon.gu...@agresearch.co.nz wrote: At Mon, 14 Oct 2013 20:22:06 -0500, John Chilton wrote: Simon, Very cool! I have two concerns. Rather than adding a new configuration option I think I would prefer to just check the configured dependency resolvers and then infer from them if the tool shed will be used. The configuration option strikes me as having to configure the same thing twice, and this change would make your setup slightly easier. Do you have any objection to me reworking your patch to do this? On the other hand, perhaps it is made more clear to the deployer that they are definitely disabling tool dependency installations if they have to add the explicit option this way. Hi John, I have no problem with you reworking it in that way. There are two reasons I didn't do that myself: 1. I would have had to change the interface to the dependency resolvers somehow to support this query, and I wasn't sure that was a good thing. 2. I wanted to make it explicit that toolshed package installation was disabled in this case, as I thought that would make it more likely this change gets accepted into the mainline. Whichever way you Greg and Dave are happy with is OK by me. Actually, I like your implicit approach better, so hope that's the one that gets agreed. cheers, Simon modules_installed.patch ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Configurable toolshed package installation to support tool-dependency-resolver-plugins
Simon, As you have probably noticed a new stable galaxy was released. It includes 95% of what we discussed including this implicit check to see if tool shed packages are enabled. Your help implementing, testing, and driving these changes was greatly appreciated! I couldn't however pull the trigger and mark packages resolved via modules as Installed - so they will still appear to be in an error state (though your check is in there and they won't attempt to be installed, they will be just marked as errors). The upshot is you can change this one line of code in your Galaxy instance to get the behavior you desire (patch attached). The reason I don't want to mark these packages as Installed is that I am worried about Galaxy deployments that maybe want to use modules are first but transition to tool shed packages down the road. I am unsure what will happen if things are marked as Installed even if no files corresponding to the installation exist. I think the state NEVER_INSTALLED may be preferable - but I need to understand more about what that means. For your own instance, if you are certainly committed to using modules and not using the tool shed - it should be easy to apply the above patch. Is this a fair compromise for the time being? Until I can resolve this last issue, the Trello card remains open, but it should now be quite trivial to modify Galaxy to get the behavior you desire and hopefully this can serve as a model for how others can hook in other dependency resolution mechanisms. I would be eager to hear how this experiment progresses and how you feel about the implementation. Thanks for your contributions, -John On Mon, Oct 14, 2013 at 8:33 PM, Guest, Simon simon.gu...@agresearch.co.nz wrote: At Mon, 14 Oct 2013 20:22:06 -0500, John Chilton wrote: Simon, Very cool! I have two concerns. Rather than adding a new configuration option I think I would prefer to just check the configured dependency resolvers and then infer from them if the tool shed will be used. The configuration option strikes me as having to configure the same thing twice, and this change would make your setup slightly easier. Do you have any objection to me reworking your patch to do this? On the other hand, perhaps it is made more clear to the deployer that they are definitely disabling tool dependency installations if they have to add the explicit option this way. Hi John, I have no problem with you reworking it in that way. There are two reasons I didn't do that myself: 1. I would have had to change the interface to the dependency resolvers somehow to support this query, and I wasn't sure that was a good thing. 2. I wanted to make it explicit that toolshed package installation was disabled in this case, as I thought that would make it more likely this change gets accepted into the mainline. Whichever way you Greg and Dave are happy with is OK by me. Actually, I like your implicit approach better, so hope that's the one that gets agreed. cheers, Simon # HG changeset patch # User John Chilton jmchil...@gmail.com # Date 1383627567 21600 # Mon Nov 04 22:59:27 2013 -0600 # Branch stable # Node ID a278fe29f7217feb3145e785de92287f76455636 # Parent 5c789ab4144ac9db6c91b5646032894cae016309 Patch latest galaxy stable to mark packages resolved via modules when tool shed resolution disabled to be marked as INSTALLED in admin interface instead of ERROR. diff -r 5c789ab4144a -r a278fe29f721 lib/tool_shed/util/common_install_util.py --- a/lib/tool_shed/util/common_install_util.py Mon Nov 04 15:04:42 2013 -0500 +++ b/lib/tool_shed/util/common_install_util.py Mon Nov 04 22:59:27 2013 -0600 @@ -467,7 +467,7 @@ if app.toolbox.dependency_manager.find_dep( package_name, package_version, type='package') != INDETERMINATE_DEPENDENCY: ## TODO: Do something here such as marking it installed or ## configured externally. -pass +status = app.model.ToolDependency.installation_status.INSTALLED tool_dependency.status = status else: tool_dependency = install_package( app, elem, tool_shed_repository, tool_dependencies=tool_dependencies ) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Configurable toolshed package installation to support tool-dependency-resolver-plugins
At Mon, 14 Oct 2013 20:22:06 -0500, John Chilton wrote: Simon, Very cool! I have two concerns. Rather than adding a new configuration option I think I would prefer to just check the configured dependency resolvers and then infer from them if the tool shed will be used. The configuration option strikes me as having to configure the same thing twice, and this change would make your setup slightly easier. Do you have any objection to me reworking your patch to do this? On the other hand, perhaps it is made more clear to the deployer that they are definitely disabling tool dependency installations if they have to add the explicit option this way. Hi John, I have no problem with you reworking it in that way. There are two reasons I didn't do that myself: 1. I would have had to change the interface to the dependency resolvers somehow to support this query, and I wasn't sure that was a good thing. 2. I wanted to make it explicit that toolshed package installation was disabled in this case, as I thought that would make it more likely this change gets accepted into the mainline. Whichever way you Greg and Dave are happy with is OK by me. Actually, I like your implicit approach better, so hope that's the one that gets agreed. cheers, Simon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Configurable toolshed package installation to support tool-dependency-resolver-plugins
At Fri, 4 Oct 2013 23:12:01 -0500, John Chilton wrote: I understand the desire to not want to try to execute the tool shed actions if tool_shed_packages are not specified in the dependency resolvers list. I have created a Trello card for it (https://trello.com/c/CPeU3VlR). It sounds like the new status quo will be functional it will just be kind of annoying to have those actions try and fail and then marked as errors, right? If that is right then for this reason I don't think implementing this behavior will be a high priority for the team, but if you send another brilliant patch or pull request I will be happy to test it and merge it. Hi John, I have now implemented this. Mercurial changeset attached. This adds another configuration item for universe_wsgi.ini as documented in the updated universe_wsgi.ini.sample: # This option may be used to disable installation of package # dependencies from tool sheds, which usually means downloading a # source tarball and compiling it locally. You would disable this if # you want to make use of system installed packages for example. In # that case, alternative tool dependency resolvers should be # configured in dependency_resolvers_conf.xml #enable_tool_shed_package_dependency_installation = True The default behaviour is per standard toolshed. If configured to False, then the package components of tool dependencies will never be installed from the toolshed. Rather, they will rely on the tool dependency manager to resolve the requirement. This is integrated with the missing_tool_dependencies status, so that if dependency resolution fails, the repository gets flagged with missing tool dependencies on the installed tool shed repositories page, and the paster.log file contains a warning of exactly what wasn't found. Of course, if the dependency can be satisfied, then it's green lights all the way, and everything should work fine. I've tested this interactively by installing the standard emboss_5 repository, with various combinations of environment module available and not. It looks to work fine. I'm afraid I haven't yet read up on the Galaxy unit testing framework, so no unit tests for now. What do you think? cheers, Simon configurable-toolshed-package-installation.patch Description: Binary data ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
