Thank you for your email Peter.
We have implemented Galaxy to interface with our HPC cluster via PBS/Torque. 
Thanks to DRMAA (not PBS python) all user cpu usage can be accounted.The 
motivation is indeed what you describe besides managing cost/disk performance 
on user/project basis as we have a tiered storage.
Our filesystem is GPFS which as you might know has one (amongst many) nice 
feature called fileset: it's basically a data bucket that reports usage 
disregarding the Unix ownership. It works great for project type directory. 
The file name length is a legitimate one indeed for command line limitation 
(GPFS has same length name limit as ext3/4). The current filename can remain 
unmodified: the requested schema would only introduce the user database ID 
(usually 3-4 digits) in the path e.g.
~/galaxy-dist/database/files/000/dataset_0001.dat
~/galaxy-dist/database/files/001/dataset_0002.dat
~/galaxy-dist/database/files/002/dataset_0003.dat
~/galaxy-dist/database/files/000/dataset_0004.dat
~/galaxy-dist/database/files/000/dataset_0005.dat
~/galaxy-dist/database/files/000/dataset_0006.dat
~/galaxy-dist/database/files/002/dataset_0007.dat
[...]
Any thoughts?
Thanks again
JC

________________________________________
From: Peter Cock [p.j.a.c...@googlemail.com]
Sent: Tuesday, May 15, 2012 1:38 AM
To: Jean-Christophe Ducom
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] user data upload directory structure

On Mon, May 14, 2012 at 10:22 PM, Jean-Christophe Ducom
<jcdu...@scripps.edu> wrote:
> All-
> Is there a way to change the upload default directory structure
> (/database/files) to organize files per user_id instead?
> something along the following lines
> ~galaxy-dist/database/files/postgresql_user_id0
> ~galaxy-dist/database/files/postgresql_user_id1
>
> Thank you
> JC

I can see that being useful for a quick way to look at per user
disk usage - although you'd have problems counting with
shared data. Is that your motivation?

Another concern would be overly long filenames, which has
a direct impact on the command line lengths used to call the
tools - there are OS limits on this.

Peter

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Reply via email to