Hi Samuel
Have you looked at "Data Libraries" ?
(http://wiki.galaxyproject.org/Admin/Data%20Libraries/Libraries )
also look at the configurations in universe_wsgi.ini:
library_import_dir
user_library_import_dir
allow_library_path_paste
which gives you many option to access data out side of the Galaxy
database/ folder - without duplicating the data.
Regards, Hans-Rudolf
On 11/14/2012 03:23 PM, Samuel Lampa wrote:
Hi Jorrit, thanks for the input!
On 11/14/2012 03:12 PM, Jorrit Boekel wrote:
This may be implemented, I believe some people run Galaxy on the cluster,
Yes, but AFAIK, most cluster job runners use some kind of file staging
(copying) into local storage on the compute nodes OR, require you do
have the data uploaded in the galaxy file area, and to have this file
are availble NFS mounted on all compute nodes.
It would be very interesting if there is an existing solution for
running jobs on files without moving/copying them at all, though!
Best
// Samuel
Galaxy normally operates on files stored in the database/files folder.
If your Galaxy instance has access to the files you need on your
parallel fs, I guess you could start by writing a tool that creates
links to your files when given the path to the input file. Another
tool may then move the dataset into the desired place while creating a
link in the database folder.
Sounds like a hack to me, but may work.
cheers,
jorrit
On Wed, Nov 14, 2012 at 2:52 PM, Samuel Lampa <samuel.la...@gmail.com
<mailto:samuel.la...@gmail.com>> wrote:
Hi,
We are looking into ways to integrate galaxy into the workflows of
users at our cluster, with lots of NGS users running all and any
kind of analyses on their typically huge amounts of data. For this
we use a parallel file system, available on all compute nodes.
This file system, although approx 1PB in size, is constantly
filling up, and thus we are not very attracted by the idea of
copying files into/out of galaxy for each analysis.
Thus, we would be interested to know what are the options for
working with existing/external(to galaxy) file systems?
Eg. would it be possible to link files into some kind of galaxy
file system (I'm not totally clear about how galaxy stores it's
data, although I found out that stuff is created in
database/files), from outside?
... or is there any work going on for selecting any file system
path as input in galaxy workflows?
... or any other hints?
As said, I'm quite new to galaxy, trying to grok my head around
how we can use it, so all hints are welcome.
Best Regards
// Samuel
-- Developer at SNIC-UPPMAX www.uppmax.uu.se
<http://www.uppmax.uu.se>
Developer at Dept of Pharm Biosciences www.farmbio.uu.se
<http://www.farmbio.uu.se>
Twitter - twitter.com/samuellampa <http://twitter.com/samuellampa>
Blog - saml.rilspace.org <http://saml.rilspace.org>
G+ - gplus.to/saml <http://gplus.to/saml>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
