Re: [galaxy-dev] [Patch] ToolShed support for included tools
Hello Franco, These details for defining tool dependency installation for tools contained in tool shed repositories are discussed in the Galaxy tool shed wiki at http://wiki.g2.bx.psu.edu/ToolShedToolFeatures. You can also learn a lot by looking at example tool_dependency.xml files in repositories in the tool shed. These repositories in the main Galaxy tool shed provide some good examples: htseqcount, emboss_5, freebayes, picard Greg Von Kuster On Oct 26, 2012, at 1:49 AM, Franco Caramia wrote: Hi Greg, Sorry to ask you something unrelated. Where can I find documentation about usage and syntax of tags such as : install, package Is there any available? There has to be more than just : http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax Thanks, Franco On 26/10/12 1:58 AM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Bjorn, I've committed an slightly altered version of your contributed code for handling tool dependencies that are zip archives in change set 8107:b12140970208, which will be available in the next Galaxy release currently scheduled for the end of next week. Your code required Python 2.6+, so I made some changes to support Python 2.5+. Please let me know if you have additional questions regarding this. Thanks very much for your contributions! Greg Von Kuster On Oct 15, 2012, at 1:15 PM, Björn Grüning wrote: Hi, i'm writing a galaxy wrapper for bismark and trim-galore. Both are plain perl scripts, that wraps around other dependencies (e.g. Bowtie). The idea was to include the perl-scripts directly in the galaxy-wrapper and update the PATH to the REPOSITORY_INSTALL_DIR in the tool_dependency.xml file. package name=bismark version=0.7.7 install version=1.0 actions action type=set_environment environment_variable name=PATH action=prepend_to$REPOSITORY_INSTALL_DIR/environment_variable /action /actions /install readme bismark, bismark_genome_preparation and bismark_methylation_extractor are shipped with that wrapper /readme /package Unfortunately, that was not supported because the toolshed expected at least one action_type. The attached patch should add that feature. Furthermore, bowtie2 is only available as zip archive and afaik that was not handled in the toolshed. The attached patch also added check_zipfile(), extract_zip() and zip_extraction_directory() to fully support zip archives. Thanks! Bjoern toolshed_zip.patch_ __ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ This email (including any attachments or links) may contain confidential and/or legally privileged information and is intended only to be read or used by the addressee. If you are not the intended addressee, any use, distribution, disclosure or copying of this email is strictly prohibited. Confidentiality and legal privilege attached to this email (including any attachments) are not waived or lost by reason of its mistaken delivery to you. If you have received this email in error, please delete it and notify us immediately by telephone or email. Peter MacCallum Cancer Centre provides no guarantee that this transmission is free of virus or that it has not been intercepted or altered and will not be liable for any delay in its receipt. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [Patch] ToolShed support for included tools
Hi Bjorn, I've committed an slightly altered version of your contributed code for handling tool dependencies that are zip archives in change set 8107:b12140970208, which will be available in the next Galaxy release currently scheduled for the end of next week. Your code required Python 2.6+, so I made some changes to support Python 2.5+. Please let me know if you have additional questions regarding this. Thanks very much for your contributions! Greg Von Kuster On Oct 15, 2012, at 1:15 PM, Björn Grüning wrote: Hi, i'm writing a galaxy wrapper for bismark and trim-galore. Both are plain perl scripts, that wraps around other dependencies (e.g. Bowtie). The idea was to include the perl-scripts directly in the galaxy-wrapper and update the PATH to the REPOSITORY_INSTALL_DIR in the tool_dependency.xml file. package name=bismark version=0.7.7 install version=1.0 actions action type=set_environment environment_variable name=PATH action=prepend_to$REPOSITORY_INSTALL_DIR/environment_variable /action /actions /install readme bismark, bismark_genome_preparation and bismark_methylation_extractor are shipped with that wrapper /readme /package Unfortunately, that was not supported because the toolshed expected at least one action_type. The attached patch should add that feature. Furthermore, bowtie2 is only available as zip archive and afaik that was not handled in the toolshed. The attached patch also added check_zipfile(), extract_zip() and zip_extraction_directory() to fully support zip archives. Thanks! Bjoern toolshed_zip.patch___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [Patch] ToolShed support for included tools
Hello Bjorn, From your comments, it seems you are looking for a way to define a tool dependency that is included in the contents of the repository when it is installed from the tool shed. If this is the case, then your tool_dependencies.xml tag set needs to be changed to the following. tool_dependency set_environment version=1.0 environment_variable name=PATH action=prepend_to$REPOSITORY_INSTALL_DIR/environment_variable /set_environment /tool_deppendency All of the details for how to define your tool configs and dependencies included in your repository like this are available in the following tool shed wiki page. http://wiki.g2.bx.psu.edu/ToolShedToolFeatures#Finding_dependencies_included_in_an_installed_repository_at_tool_execution_time Thanks very much for contributing - I'll take a look at this patch and integrate part or all of it into the next release if appropriate. Greg Von Kuster On Oct 15, 2012, at 1:15 PM, Björn Grüning wrote: Hi, i'm writing a galaxy wrapper for bismark and trim-galore. Both are plain perl scripts, that wraps around other dependencies (e.g. Bowtie). The idea was to include the perl-scripts directly in the galaxy-wrapper and update the PATH to the REPOSITORY_INSTALL_DIR in the tool_dependency.xml file. package name=bismark version=0.7.7 install version=1.0 actions action type=set_environment environment_variable name=PATH action=prepend_to$REPOSITORY_INSTALL_DIR/environment_variable /action /actions /install readme bismark, bismark_genome_preparation and bismark_methylation_extractor are shipped with that wrapper /readme /package Unfortunately, that was not supported because the toolshed expected at least one action_type. The attached patch should add that feature. Furthermore, bowtie2 is only available as zip archive and afaik that was not handled in the toolshed. The attached patch also added check_zipfile(), extract_zip() and zip_extraction_directory() to fully support zip archives. Thanks! Bjoern toolshed_zip.patch___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] [Patch] ToolShed support for included tools
Hi,
i'm writing a galaxy wrapper for bismark and trim-galore. Both are plain
perl scripts, that wraps around other dependencies (e.g. Bowtie). The
idea was to include the perl-scripts directly in the galaxy-wrapper and
update the PATH to the REPOSITORY_INSTALL_DIR in the tool_dependency.xml
file.
package name=bismark version=0.7.7
install version=1.0
actions
action type=set_environment
environment_variable name=PATH
action=prepend_to$REPOSITORY_INSTALL_DIR/environment_variable
/action
/actions
/install
readme
bismark, bismark_genome_preparation and bismark_methylation_extractor
are shipped with that wrapper
/readme
/package
Unfortunately, that was not supported because the toolshed expected at
least one action_type. The attached patch should add that feature.
Furthermore, bowtie2 is only available as zip archive and afaik that was
not handled in the toolshed. The attached patch also added
check_zipfile(), extract_zip() and zip_extraction_directory() to fully
support zip archives.
Thanks!
Bjoern
diff -r cc45bdf57b66 lib/galaxy/tool_shed/tool_dependencies/common_util.py
--- a/lib/galaxy/tool_shed/tool_dependencies/common_util.py Fri Oct 12 17:22:51 2012 -0400
+++ b/lib/galaxy/tool_shed/tool_dependencies/common_util.py Mon Oct 15 18:30:58 2012 +0200
@@ -1,12 +1,13 @@
import os, shutil, tarfile, urllib2
from galaxy.datatypes.checkers import *
+from zipfile import ZipFile
def create_env_var_dict( elem, tool_dependency_install_dir=None, tool_shed_repository_install_dir=None ):
env_var_name = elem.get( 'name', 'PATH' )
env_var_action = elem.get( 'action', 'prepend_to' )
env_var_text = None
if elem.text and elem.text.find( 'REPOSITORY_INSTALL_DIR' ) = 0:
-if tool_shed_repository_install_dir:
+if tool_shed_repository_install_dir and elem.text.find( '$REPOSITORY_INSTALL_DIR' ) != -1:
env_var_text = elem.text.replace( '$REPOSITORY_INSTALL_DIR', tool_shed_repository_install_dir )
return dict( name=env_var_name, action=env_var_action, value=env_var_text )
else:
@@ -50,6 +51,27 @@
tar = tarfile.open( file_name )
tar.extractall( path=file_path )
tar.close()
+def check_zipfile( archive_name ):
+
+This function is a bit pedantic and not functionally necessary.
+It checks whether there is no file pointing outside of the extraction,
+because ZipFile.extractall() has some potential security holes.
+See python zipfile documentation for more details.
+
+basename = os.path.normcase( os.path.realpath( os.path.dirname( archive_name ) ) )
+with ZipFile( archive_name ) as zip:
+for member in ZipFile.namelist(zip):
+member_path = os.path.normcase( os.path.realpath( os.path.join(basename, member) ) )
+if not member_path.startswith( basename ):
+return False
+return True
+def extract_zip( archive_path, extraction_path ):
+if not check_zipfile( archive_path ):
+return False
+zip = ZipFile( archive_path )
+zip.extractall( extraction_path )
+zip.close()
+return True
def isbz2( file_path ):
return is_bz2( file_path )
def isgzip( file_path ):
@@ -87,6 +109,21 @@
if os.path.exists( os.path.abspath( os.path.join( file_path, file_name ) ) ):
return os.path.abspath( os.path.join( file_path, file_name ) )
raise ValueError( 'Could not find directory %s' % os.path.abspath( os.path.join( file_path, file_name[ :-len( extension ) ] ) ) )
+def zip_extraction_directory( file_path, file_name ):
+
+Try to guess the extraction directory.
+1. Is there one folder inside the working directory.
+2. Are there more than one folder/files return working directory.
+
+files = [filename for filename in os.listdir( file_path ) if not filename.endswith('.zip')]
+if len(files) 1:
+return os.path.abspath( file_path )
+elif len(files) == 1:
+# if there is only on file it should be a folder
+if os.path.isdir( os.path.join( file_path, files[0] ) ):
+return os.path.abspath( os.path.join( file_path, files[0] ) )
+# otherwise we assume that the download or something else went wrong
+raise ValueError( 'Could not find directory for the extracted file %s' % os.path.abspath( os.path.join( file_path, file_name ) ) )
def url_download( install_dir, downloaded_file_name, download_url ):
file_path = os.path.join( install_dir, downloaded_file_name )
src = None
diff -r cc45bdf57b66 lib/galaxy/tool_shed/tool_dependencies/fabric_util.py
--- a/lib/galaxy/tool_shed/tool_dependencies/fabric_util.py Fri Oct 12 17:22:51 2012 -0400
+++ b/lib/galaxy/tool_shed/tool_dependencies/fabric_util.py Mon Oct 15 18:30:58 2012 +0200
@@ -58,6 +58,9 @@
# The first action in the list of actions will be the one that defines the installation process. There
