Hi John

I will review the galaxy code more closely and implement it as you suggested. 
It's been a while since I've implemented this 'fix', so I will have to dig up 
the code that tries to delete the file after upload.

Once it's simple and clean, I will do a pull request.

Thanks for your input.

Iyad Kandalaft
Bioinformatics Programmer
Microbial Biodiversity Bioinformatics
Science & Technology Branch
Agriculture & Agri-Food Canada
iyad.kandal...@agr.gc.ca | (613) 759-1228

________________________________________
From: John Chilton [jmchil...@gmail.com]
Sent: June 1, 2014 7:53 PM
To: Kandalaft, Iyad
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Uploading files to galaxy from a folder

Hello Iyad,

  Thanks for taking the time to describe your problem and share your
solution - I am sure other institutions have desires to harness the
ftp upload option the way you are doing it.

  Is the only problem that occurs if the ownership is incorrect that
Galaxy cannot delete the files? If so - I feel like as a deployer I
would prefer to have a sudo script that just deletes the files instead
of changing the ownership? It is a more... focused sudo command.

  This strategy could also allow a slightly more general
universe_wsgi.ini option - say a few options such as:

ftp_upload_delete_handling = __default__ # just assume ownership and delete
ftp_upload_delete_handling = __none__ # don't delete files - user is responsible
ftp_upload_delete_handling = sudo -E rm # command prefix to delete file

  I am not saying if you clean up what you have and opened a pull
request I would say no - I am just saying I would be more eager for
this more general option. If you can confirm the only reason the
permissions need to change is to delete the files and likewise prefer
this variant but cannot figure out how to modify upload.py - let me
know I can try to look into it (no promises that it is easy to
implement).

-John

On Tue, May 27, 2014 at 10:28 AM, Kandalaft, Iyad
<iyad.kandal...@agr.gc.ca> wrote:
> Hi Everyone
>
>
>
> I’m throwing this out there for some feedback and recommendations.
>
>
>
>
>
> Objective: Facilitate transferring large files (> 2GB) from an HPC cluster
> (and its associated fast tier storage) to galaxy for my clients.  I enabled
> the FTP upload option in galaxy but it involves users learning to copy files
> over FTP.
>
>
>
> So, I created a galaxy folder in each users’ home directory on the HPC
> Cluster that symbolically links to the FTP upload folder for galaxy.  Hence,
> users can use either FTP to upload files (drag and drop in windows) or
> simply copy files into this folder from an ssh session on the cluster.  The
> problem with that strategy was that galaxy had to be the owner of the file
> (similar to the ProFTPd configuration that sets the UID and GID of uploads
> files to galaxy’s UID/GID).  Otherwise, galaxy threw errors when it tried
> deleting the original file from the FTP upload folder.  I could have added
> the galaxy user to the same group as all user but this meant that users
> would have to ensure the correct permissions are set on files so that galaxy
> can read and delete the file thereafter.  The alternative involved modifying
> the upload.py tool to chown/chmod files that were being uploaded.  Upload.py
> now sudo executes an external script that sets ownership to the galaxy user
> and corrects the permissions if required (see attachment for code
> modification).  The galaxy user has sudo rights on this script and the
> script restricts chown/chmod to the ftp folder path for security reasons.
>
>
>
> I was planning to clean up the code and make it production ready by adding
> an option in universe_wsgi.ini for this “feature”, but I thought I would
> check with the galaxy devs first. Am I taking the wrong approach?  Is there
> a better alternative?
>
>
>
> As an alternative, I thought about locating the handler code for
> dataset.type == file and possibly making it support the SETGID sticky bits
> on folders.  In that case, the FTP upload folder would have the sticky bit
> set for UID and can assume the role of the user to upload that file.
>
>
>
> Your input is much appreciated.
>
>
>
> Iyad Kandalaft
>
>
>
> Bioinformatics Application Developer
>
> Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada
>
> KW Neatby Bldg | éd. KW Neatby
>
> 960 Carling Ave| 960, avenue Carling
>
> Ottawa, ON | Ottawa (ON) K1A 0C6
>
> E-mail Address / Adresse courriel: iyad.kandal...@agr.gc.ca
>
> Telephone | Téléphone 613- 759-1228
>
> Facsimile | Télécopieur 613-759-1701
>
> Government of Canada | Gouvernement du Canada
>
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to