Re: [galaxy-dev] Galaxy Cloudman - How to analyse > 1TB data ?

2012-02-16 Thread Wetzels, Yves [JRDBE Extern]
Hi Brad I did not restart the master CloudMan node. I only restarted the services (Galaxy, PostgreSQL and SGE). I do not have these problems without creating the logical volume. Kind Regards Yves Yves; I'm hoping Enis can jump in here since he is more familiar with the internals of CloudMan an

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Peter Cock
On Wed, Feb 15, 2012 at 6:07 PM, Dannon Baker wrote: > > Main still runs these jobs in the standard non-split fashion, and as a > resource that is occasionally saturated (and thus doesn't necessarily have > extra resources to parallelize to) will probably continue doing so as long > as there's sig

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Dannon Baker
On Feb 16, 2012, at 5:15 AM, Peter Cock wrote: > On Wed, Feb 15, 2012 at 6:07 PM, Dannon Baker wrote: >> >> Main still runs these jobs in the standard non-split fashion, and as a >> resource that is occasionally saturated (and thus doesn't necessarily have >> extra resources to parallelize to)

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Peter Cock
On Wed, Feb 15, 2012 at 6:07 PM, Dannon Baker wrote: > Good luck, let me know how it goes, and again - contributions are certainly > welcome :) I think I found the first bug, method split in lib/galaxy/datatypes/sequence.py for class Sequence assumes four lines per sequence. This would make sense

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Peter Cock
On Thu, Feb 16, 2012 at 10:47 AM, Peter Cock wrote: > On Wed, Feb 15, 2012 at 6:07 PM, Dannon Baker wrote: >> Good luck, let me know how it goes, and again - contributions are certainly >> welcome :) > > I think I found the first bug, method split in > lib/galaxy/datatypes/sequence.py > for clas

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Fields, Christopher J
On Feb 16, 2012, at 4:47 AM, Peter Cock wrote: > On Wed, Feb 15, 2012 at 6:07 PM, Dannon Baker wrote: >> Good luck, let me know how it goes, and again - contributions are certainly >> welcome :) > > I think I found the first bug, method split in > lib/galaxy/datatypes/sequence.py > for class Se

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Peter Cock
On Thu, Feb 16, 2012 at 1:53 PM, Fields, Christopher J wrote: > > Makes sense from my perspective; splits have to be defined based on > data type.  It could be as low-level as defining a simple iterator per > record, then a wrapper that allows a specific chunk-size.  The split > file creation coul

[galaxy-dev] Tool that outputs html report

2012-02-16 Thread Daniel Sobral
Hello, I want to develop a galaxy wrapper for a tool that outputs an html report (with images inside specific subfolders). Do I need to do like the FastQC galaxy wrapper that reconstructs the html so that they are all in the same folder? Or is there a way to circumvent this so that I can keep the

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Peter Cock
Hi Dan, I think I need a little more advice - what is the role of the script scripts/extract_dataset_part.py and the JSON files created when splitting FASTQ files in lib/galaxy/datatypes/sequence.py, and then used by the class' process_split_file method? Why is there no JSON file created by the b

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Peter Cock
On Thu, Feb 16, 2012 at 4:28 PM, Peter Cock wrote: > Hi Dan, > > I think I need a little more advice - what is the role of the script > scripts/extract_dataset_part.py and the JSON files created > when splitting FASTQ files in lib/galaxy/datatypes/sequence.py, > and then used by the class' process

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Dannon Baker
Very cool, I'll check it out! The addition of the JSON files is indeed very new and was likely unfinished with respect to the base splitter. -Dannon On Feb 16, 2012, at 1:24 PM, Peter Cock wrote: > On Thu, Feb 16, 2012 at 4:28 PM, Peter Cock wrote: >> Hi Dan, >> >> I think I need a little mo

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Fields, Christopher J
On Feb 16, 2012, at 12:24 PM, Peter Cock wrote: > On Thu, Feb 16, 2012 at 4:28 PM, Peter Cock wrote: >> Hi Dan, >> >> I think I need a little more advice - what is the role of the script >> scripts/extract_dataset_part.py and the JSON files created >> when splitting FASTQ files in lib/galaxy/dat

[galaxy-dev] Expanding galaxyTool volume?

2012-02-16 Thread Dave Lin
Hi All, What is the recommend process for expanding the galaxyTool volume for an existing galaxy instance (using EC2/cloudman)? I tried the following, but it didnt' work for me. 0) Terminate cluster. 1) Amazon EC2- create snapshot of current galaxyTools volume 2) Amazon EC2- create volume from

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Peter Cock
On Thu, Feb 16, 2012 at 6:42 PM, Fields, Christopher J wrote: > On Feb 16, 2012, at 12:24 PM, Peter Cock wrote: >> I've checked in my FASTA splitting, which now seems to be >> working OK with my BLAST tests. (If this was unclear, I mean checked into my branch - I don't have commit privileges to t

Re: [galaxy-dev] Expanding galaxyTool volume?

2012-02-16 Thread Enis Afgan
Hi Dave, Are you trying to modify the size of the tools volume for a cluster that's been around for a while and you customized already or could this be a new cluster? The reason I'm asking is because as of Tuesday (3 days ago), the default tools volume for any new cluster will be 10GB (vs 2GB previ

Re: [galaxy-dev] Expanding galaxyTool volume?

2012-02-16 Thread Dave Lin
Hi Enis, I installed a new test cluster earlier today and did notice that the new clusters magically now have galaxyTool volumes with 10GB. That is a good change. However, you are correct. I have an existing cluster (that had the old 2 GB volume size) that I'm trying to expand. With additional to

Re: [galaxy-dev] Expanding galaxyTool volume?

2012-02-16 Thread Enis Afgan
No problem; this should get you there: 1. With CloudMan running, go to the Admin console and stop Galaxy and PostgreSQL services (in that order) 2. From instance's CLI: sudo umount /mnt/galaxyTools 3. From the AWS console, detach the tools volume (but remember as which device it was attached) 4.

[galaxy-dev] transferring sample datasets stuck in 'In queue' status

2012-02-16 Thread Luobin Yang
Hi all, I configured Sample Tracking System in Galaxy to transfer datasets from a sequencer to data libraries, however, after I selected the datasets to be transferred on the sequencer and clicked "Transfer" button, the transfer status has been "in queue" forever. I didn't find any error message

[galaxy-dev] producing HTML output with images

2012-02-16 Thread Nikhil Joshi
Hi all, I am having trouble producing HTML output with images. In the past I have been able to produce HTML files with no images and it seemed to work fine. However, now I am writing a script that produces diagnostic images and I want to display all of the images on one page using HTML. I am us

Re: [galaxy-dev] producing HTML output with images

2012-02-16 Thread Nikhil Joshi
Well, I guess I figured it out... it turns out that galaxy does some internal magic to automatically add the path of the output directory to the filename when it is rendering the html. So what I needed to do was to create the file in the temporary job directory and then just use the file name with