Hi Everyone

I'm throwing this out there for some feedback and recommendations.

Objective: Facilitate transferring large files (> 2GB) from an HPC cluster (and 
its associated fast tier storage) to galaxy for my clients.  I enabled the FTP 
upload option in galaxy but it involves users learning to copy files over FTP.

So, I created a galaxy folder in each users' home directory on the HPC Cluster 
that symbolically links to the FTP upload folder for galaxy.  Hence, users can 
use either FTP to upload files (drag and drop in windows) or simply copy files 
into this folder from an ssh session on the cluster.  The problem with that 
strategy was that galaxy had to be the owner of the file (similar to the 
ProFTPd configuration that sets the UID and GID of uploads files to galaxy's 
UID/GID).  Otherwise, galaxy threw errors when it tried deleting the original 
file from the FTP upload folder.  I could have added the galaxy user to the 
same group as all user but this meant that users would have to ensure the 
correct permissions are set on files so that galaxy can read and delete the 
file thereafter.  The alternative involved modifying the upload.py tool to 
chown/chmod files that were being uploaded.  Upload.py now sudo executes an 
external script that sets ownership to the galaxy user and corrects the 
permissions if required (see attachment for code modification).  The galaxy 
user has sudo rights on this script and the script restricts chown/chmod to the 
ftp folder path for security reasons.

I was planning to clean up the code and make it production ready by adding an 
option in universe_wsgi.ini for this "feature", but I thought I would check 
with the galaxy devs first. Am I taking the wrong approach?  Is there a better 

As an alternative, I thought about locating the handler code for dataset.type 
== file and possibly making it support the SETGID sticky bits on folders.  In 
that case, the FTP upload folder would have the sticky bit set for UID and can 
assume the role of the user to upload that file.

Your input is much appreciated.

Iyad Kandalaft

Bioinformatics Application Developer
Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada
KW Neatby Bldg | éd. KW Neatby
960 Carling Ave| 960, avenue Carling
Ottawa, ON | Ottawa (ON) K1A 0C6
E-mail Address / Adresse courriel: 
Telephone | Téléphone 613- 759-1228
Facsimile | Télécopieur 613-759-1701
Government of Canada | Gouvernement du Canada

diff -r 29ce93a13ac7 tools/data_source/upload.py
--- a/tools/data_source/upload.py       Mon Feb 10 13:22:47 2014 -0500
+++ b/tools/data_source/upload.py       Tue May 27 10:54:42 2014 -0400
@@ -4,7 +4,7 @@
 # WARNING: Changes in this tool (particularly as related to parsing) may need
 # to be reflected in galaxy.web.controllers.tool_runner and galaxy.tools
-import urllib, sys, os, gzip, tempfile, shutil, re, gzip, zipfile, codecs, 
+import urllib, sys, os, gzip, tempfile, shutil, re, gzip, zipfile, codecs, 
binascii, subprocess
 from galaxy import eggs
 # need to import model before sniff to resolve a circular import dependency
 import galaxy.model
@@ -352,6 +352,19 @@
                      stdout = 'uploaded %s file' % dataset.file_type )
         json_file.write( to_json_string( info ) + "\n" )
+def _chown( dataset ):
+    if ( dataset.type != 'file' ):
+        return
+    try:
+        cmd = [ '/usr/bin/sudo', '-E', 
'/home/galaxy/server-conf/chown_script.pl', dataset.path ]
+        sys.stdout.write( 'Changing ownership of %s with: %s' % ( 
dataset.path, ' '.join( cmd ) ) )
+        p = subprocess.Popen( cmd, shell=False, stdout=subprocess.PIPE, 
stderr=subprocess.PIPE )
+        sys.stdout.write( p.stdout.read() )
+        sys.stderr.write( p.stderr.read() )
+    except Exception, e:
+        sys.stderr.write( 'Changing ownership of uploaded file %s failed: %s' 
% ( dataset.path, str( e ) ) )
 def __main__():
     if len( sys.argv ) < 4:
@@ -372,6 +385,9 @@
             print >>sys.stderr, 'Output path for dataset %s not found on 
command line' % dataset.dataset_id
             sys.exit( 1 )
+        _chown( dataset )   
         if dataset.type == 'composite':
             files_path = output_paths[int( dataset.dataset_id )][1]
             add_composite_file( dataset, registry, json_file, output_path, 
files_path )
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

To search Galaxy mailing lists use the unified search at:

Reply via email to