Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Peter Cock
On Wed, Feb 15, 2012 at 6:07 PM, Dannon Baker dannonba...@me.com wrote: Main still runs these jobs in the standard non-split fashion, and as a resource that is occasionally saturated (and thus doesn't necessarily have extra resources to parallelize to) will probably continue doing so as long

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Dannon Baker
On Feb 16, 2012, at 5:15 AM, Peter Cock wrote: On Wed, Feb 15, 2012 at 6:07 PM, Dannon Baker dannonba...@me.com wrote: Main still runs these jobs in the standard non-split fashion, and as a resource that is occasionally saturated (and thus doesn't necessarily have extra resources to

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Peter Cock
On Wed, Feb 15, 2012 at 6:07 PM, Dannon Baker dannonba...@me.com wrote: Good luck, let me know how it goes, and again - contributions are certainly welcome :) I think I found the first bug, method split in lib/galaxy/datatypes/sequence.py for class Sequence assumes four lines per sequence. This

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Peter Cock
On Thu, Feb 16, 2012 at 10:47 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Wed, Feb 15, 2012 at 6:07 PM, Dannon Baker dannonba...@me.com wrote: Good luck, let me know how it goes, and again - contributions are certainly welcome :) I think I found the first bug, method split in

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Fields, Christopher J
On Feb 16, 2012, at 4:47 AM, Peter Cock wrote: On Wed, Feb 15, 2012 at 6:07 PM, Dannon Baker dannonba...@me.com wrote: Good luck, let me know how it goes, and again - contributions are certainly welcome :) I think I found the first bug, method split in lib/galaxy/datatypes/sequence.py

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Peter Cock
On Thu, Feb 16, 2012 at 1:53 PM, Fields, Christopher J cjfie...@illinois.edu wrote: Makes sense from my perspective; splits have to be defined based on data type.  It could be as low-level as defining a simple iterator per record, then a wrapper that allows a specific chunk-size.  The split

[galaxy-dev] Tool that outputs html report

2012-02-16 Thread Daniel Sobral
Hello, I want to develop a galaxy wrapper for a tool that outputs an html report (with images inside specific subfolders). Do I need to do like the FastQC galaxy wrapper that reconstructs the html so that they are all in the same folder? Or is there a way to circumvent this so that I can keep

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Peter Cock
Hi Dan, I think I need a little more advice - what is the role of the script scripts/extract_dataset_part.py and the JSON files created when splitting FASTQ files in lib/galaxy/datatypes/sequence.py, and then used by the class' process_split_file method? Why is there no JSON file created by the

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Peter Cock
On Thu, Feb 16, 2012 at 4:28 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Dan, I think I need a little more advice - what is the role of the script scripts/extract_dataset_part.py and the JSON files created when splitting FASTQ files in lib/galaxy/datatypes/sequence.py, and then used

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Dannon Baker
Very cool, I'll check it out! The addition of the JSON files is indeed very new and was likely unfinished with respect to the base splitter. -Dannon On Feb 16, 2012, at 1:24 PM, Peter Cock wrote: On Thu, Feb 16, 2012 at 4:28 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Dan, I think

[galaxy-dev] Expanding galaxyTool volume?

2012-02-16 Thread Dave Lin
Hi All, What is the recommend process for expanding the galaxyTool volume for an existing galaxy instance (using EC2/cloudman)? I tried the following, but it didnt' work for me. 0) Terminate cluster. 1) Amazon EC2- create snapshot of current galaxyTools volume 2) Amazon EC2- create volume from

Re: [galaxy-dev] Expanding galaxyTool volume?

2012-02-16 Thread Dave Lin
Hi Enis, I installed a new test cluster earlier today and did notice that the new clusters magically now have galaxyTool volumes with 10GB. That is a good change. However, you are correct. I have an existing cluster (that had the old 2 GB volume size) that I'm trying to expand. With additional

Re: [galaxy-dev] Expanding galaxyTool volume?

2012-02-16 Thread Enis Afgan
No problem; this should get you there: 1. With CloudMan running, go to the Admin console and stop Galaxy and PostgreSQL services (in that order) 2. From instance's CLI: sudo umount /mnt/galaxyTools 3. From the AWS console, detach the tools volume (but remember as which device it was attached) 4.

[galaxy-dev] transferring sample datasets stuck in 'In queue' status

2012-02-16 Thread Luobin Yang
Hi all, I configured Sample Tracking System in Galaxy to transfer datasets from a sequencer to data libraries, however, after I selected the datasets to be transferred on the sequencer and clicked Transfer button, the transfer status has been in queue forever. I didn't find any error message in

[galaxy-dev] producing HTML output with images

2012-02-16 Thread Nikhil Joshi
Hi all, I am having trouble producing HTML output with images. In the past I have been able to produce HTML files with no images and it seemed to work fine. However, now I am writing a script that produces diagnostic images and I want to display all of the images on one page using HTML. I am

Re: [galaxy-dev] producing HTML output with images

2012-02-16 Thread Nikhil Joshi
Well, I guess I figured it out... it turns out that galaxy does some internal magic to automatically add the path of the output directory to the filename when it is rendering the html. So what I needed to do was to create the file in the temporary job directory and then just use the file name