Dear list,

Our lab has been outputting data in multiple files that we currently merge (in galaxy) by tarring them. This works fine with the parallel processing that Galaxy offers.


The problem, see also below, was to create a user-friendly way to not having to create 50-200 datasets in Galaxy but one containing all the merged files. I do not know if this functionality is something that people want to use, or if it goes against Galaxy design principles, but I have implemented it for our lab.

I have enabled a <input type="file" multiple> (may not work in IE though) in a separate FileField subclass, and the list of files that subsequently uploads is persisted and passed to the upload tool, where they are merged according to a datatype (specified in the upload tool) merge-method. File type detection is done by using sniffers as file type is set to auto.

I don't have a very good view of the demand for this sort of function, but if anyone else would like to use/modify it, I can fork and issue a pull request.

cheers,
jorrit



On 03/06/2012 06:56 PM, Nate Coraor wrote:
On Feb 29, 2012, at 11:34 AM, Jorrit Boekel wrote:

Dear list,

Our lab's proteomics data is frequently outputted into>50 files containing 
different fractions of proteins. The files are locally stored and not present on 
the Galaxy server. We've planned to somehow (inside galaxy) merge these files and 
split them into tasks so they can be run on the cluster. We would either 
merge/split the files by concatenation, or untar/tar files at every job, depending 
on filetype and tool. No problems so far.

However, I have been looking around for a way to upload>50 files simultaneously to galaxy and 
convert to one dataset, and this does not seem to be supported. Before starting to create a hack to 
make this work, which doesn't seem especially trivial to me, I'd like to know if I should instead use 
libraries. From what I've seen, libraries are not treated as datasets in Galaxy but rather contain 
datasets. If there was a "tar all sets in library and import to history" I'd be using that, 
but I've only encountered "tar/zip sets and download locally" which would be a bit of a 
workaround.
Hi Jorrit,

It's not possible to do this all in one step, but you can definitely upload 
them all simultaneously and then concatenate them using the concatenate tool 
(or write a simple tool to tar them).

--nate

I haven't found much on this subject in the mailing list, has this 
functionality been requested before?

cheers,
jorrit boekel
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

 http://lists.bx.psu.edu/

Reply via email to