Re: [galaxy-dev] Tool shed and datatypes

2012-01-06 Thread Greg Von Kuster
Hello Jim, Thanks for sending your converter. I've committed change set 6484:4fdceec512f5 to our central repository. I now have things working for properly handling proprietary datatype converters and indexers. I've also added the following paragraph to the tool shed wiki. It doesn't apply

Re: [galaxy-dev] Tool shed and datatypes

2012-01-05 Thread Greg Von Kuster
Of course, your approach of prepending the repository name would probably eliminate any future issue in this regard. Whatever you feel is best... ;) On Jan 5, 2012, at 4:49 PM, Greg Von Kuster wrote: > Yes, this is certainly important, but I think the hope is that proprietary > data types wi

Re: [galaxy-dev] Tool shed and datatypes

2012-01-05 Thread Greg Von Kuster
Yes, this is certainly important, but I think the hope is that proprietary data types will not become so prevalent that name-spacing the extensions is necessary. On Jan 5, 2012, at 4:15 PM, Jim Johnson wrote: > > Big Question? > When I started creating all those datatype classes for mothu

Re: [galaxy-dev] Tool shed and datatypes

2012-01-05 Thread Greg Von Kuster
Of course, this assume that there is not more than one datatypes class module in your repository with the same name. This would definitely pose problems, so care should be taken that it is not done. On Jan 5, 2012, at 3:29 PM, Greg Von Kuster wrote: > However, your datatype class module files

Re: [galaxy-dev] Tool shed and datatypes

2012-01-05 Thread Greg Von Kuster
Your approach is great since it models the Galaxy distribution, and as you say, make sit clear to those downloading your repository. However, your datatype class module files will be found no matter where they are located within your repository hierarchy. On Jan 5, 2012, at 3:25 PM, Jim Johns

Re: [galaxy-dev] Tool shed and datatypes

2012-01-05 Thread Greg Von Kuster
Hi Jim, Here are the changes you'll need to make to your mothur tool suite. CHANGE 1 Add the following datatypes.conf.xml file to your repository.

Re: [galaxy-dev] Tool shed and datatypes

2012-01-05 Thread Greg Von Kuster
Hello Jim, I've implemented support for proprietary datatypes that use class modules included in tool shed repositories. To see how this works, you'll need at least change set revision 6479:4d131422777f, which is currently available only from our central repo at https://bitbucket.org/galaxy/ga

Re: [galaxy-dev] Tool shed and datatypes

2011-11-08 Thread Duddy, John
y-dev@lists.bx.psu.edu; Nate Coraor Subject: Re: [galaxy-dev] Tool shed and datatypes On Tue, Nov 8, 2011 at 11:45 PM, Duddy, John wrote: > It's not public yet, and it involves a little conundrum - we want > it so we can support large amounts of data efficiently on a variety > of aligners, inclu

Re: [galaxy-dev] Tool shed and datatypes

2011-11-08 Thread Peter Cock
On Tue, Nov 8, 2011 at 11:45 PM, Duddy, John wrote: > It's not public yet, and it involves a little conundrum - we want > it so we can support large amounts of data efficiently on a variety > of aligners, including our ELAND from CASAVA. However, ELAND > does not support unaligned BAM inputs yet,

Re: [galaxy-dev] Tool shed and datatypes

2011-11-08 Thread Duddy, John
- From: Peter Cock [mailto:p.j.a.c...@googlemail.com] Sent: Tuesday, November 08, 2011 3:29 PM To: Duddy, John Cc: Greg Von Kuster; galaxy-dev@lists.bx.psu.edu; Nate Coraor Subject: Re: [galaxy-dev] Tool shed and datatypes On Thu, Oct 6, 2011 at 5:45 PM, Duddy, John wrote: > GZIP files are definit

Re: [galaxy-dev] Tool shed and datatypes

2011-11-08 Thread Duddy, John
to:p.j.a.c...@googlemail.com] Sent: Tuesday, November 08, 2011 3:29 PM To: Duddy, John Cc: Greg Von Kuster; galaxy-dev@lists.bx.psu.edu; Nate Coraor Subject: Re: [galaxy-dev] Tool shed and datatypes On Thu, Oct 6, 2011 at 5:45 PM, Duddy, John wrote: > GZIP files are definitely our plan. I just finished tes

Re: [galaxy-dev] Tool shed and datatypes

2011-11-08 Thread Peter Cock
On Thu, Oct 6, 2011 at 5:45 PM, Duddy, John wrote: > GZIP files are definitely our plan. I just finished testing the code > that distributes the processing of a FASTQ (or pair for PE) to an > arbitrary number of tasks, where each subtask extracts just the > data it needs without reading any of the

Re: [galaxy-dev] Tool shed and datatypes

2011-10-21 Thread Jim Johnson
On 10/21/11 12:29 PM, James Taylor wrote: Excerpts from Jim Johnson's message of 2011-10-21 17:13:02 +: I put the gmap tool suite in the galaxy Tool Shed, let me know if there is more I should do. Awesome! I added a requirement tag for the datatypes to the tool-configs: % grep 're

Re: [galaxy-dev] Tool shed and datatypes

2011-10-21 Thread Greg Von Kuster
alaxy tool shed tools are automatically installed. There >>>>>> are some complexities to consider in attempting this. One of the issues >>>>>> to consider is that the work for adding support for a new datatype to >>>>>> Galaxy lies outside of the in

Re: [galaxy-dev] Tool shed and datatypes

2011-10-21 Thread James Taylor
Excerpts from Jim Johnson's message of 2011-10-21 17:13:02 +: > I put the gmap tool suite in the galaxy Tool Shed, let me know if there is > more I should do. Awesome! > I added a requirement tag for the datatypes to the tool-configs: > > % grep 'requirement.*datatype' *.xml > gmap

Re: [galaxy-dev] Tool shed and datatypes

2011-10-21 Thread Jim Johnson
should be separate from the aligner, but that would require that Galaxy be made aware of the custom file formats – we’d have to add a datatype. John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jduddy at illumina.com

Re: [galaxy-dev] Tool shed and datatypes

2011-10-18 Thread Greg Von Kuster
>>>> >>>> >>>> On Oct 5, 2011, at 11:48 PM, Duddy, John wrote: >>>> >>>>> One of the things we’re facing is the sheer size of a whole human genome >>>>> at 30x coverage. An effective way to deal with that is by compressi

Re: [galaxy-dev] Tool shed and datatypes

2011-10-18 Thread Jim Johnson
equire that Galaxy be made aware of the custom file formats – we’d have to add a datatype. John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jduddy at illumina.com From: Greg Von Kuster [mailto:greg at bx.psu.edu] Sent: Wed

Re: [galaxy-dev] Tool shed and datatypes

2011-10-17 Thread Greg Von Kuster
h worlds – >>> efficient storage and use by all existing tools. >>> >>> Another example would be adding the CASAVA tools to Galaxy. Some of the >>> statistics generation tools use custom file formats. To be able to make the >>> use of those

Re: [galaxy-dev] Tool shed and datatypes

2011-10-10 Thread Greg Von Kuster
drastically overestimate the frequency with which submissions might > be made (which is entirely possible), that poor team's operations could wind > up looking not unlike the USPTO. >> >> Anyway, my general point is that there are many non-trivial factors to >> consider in the question of creating a TypeSh

Re: [galaxy-dev] Tool shed and datatypes

2011-10-10 Thread Greg Von Kuster
Peter has the right idea here - we will add support for appropriate data types to the Galaxy distribution. Of course, the key word here is "appropriate", but any industry-standard data format should fall under this category. On Oct 10, 2011, at 12:46 PM, Peter Cock wrote: > On Mon, Oct 10, 201

Re: [galaxy-dev] Tool shed and datatypes

2011-10-10 Thread Jim Johnson
du...@illumina.com -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Paniagua, Eric Sent: Friday, October 07, 2011 5:53 PM To: j...@umn.edu; galaxy-dev@lists.bx.psu.edu Cc: Greg Von Kuster Subject: Re: [galaxy-dev] T

Re: [galaxy-dev] Tool shed and datatypes

2011-10-10 Thread Peter Cock
On Mon, Oct 10, 2011 at 5:09 PM, Duddy, John wrote: > I agree with the risks you cited. > > There is a risk in the other direction that I think is even scarier - > without the ability to add data types, tool authors may be forced > to use a "typeless" system, declaring all inputs/outputs as "data"

Re: [galaxy-dev] Tool shed and datatypes

2011-10-10 Thread Duddy, John
...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Paniagua, Eric Sent: Friday, October 07, 2011 5:53 PM To: j...@umn.edu; galaxy-dev@lists.bx.psu.edu Cc: Greg Von Kuster Subject: Re: [galaxy-dev] Tool shed and datatypes Hi all, Just my 2 cents. This is a really great idea to have

Re: [galaxy-dev] Tool shed and datatypes

2011-10-07 Thread Paniagua, Eric
to me yet? What would be the benefits and potential pitfalls? Best, Eric From: galaxy-dev-boun...@lists.bx.psu.edu [galaxy-dev-boun...@lists.bx.psu.edu] on behalf of Jim Johnson [j...@umn.edu] Sent: Friday, October 07, 2011 2:06 PM To: galaxy-dev@lists.bx.psu.edu

Re: [galaxy-dev] Tool shed and datatypes

2011-10-07 Thread Jim Johnson
Tel: 858-736-3584 E-mail: jduddy at illumina.com From: Greg Von Kuster [mailto:greg at bx.psu.edu] Sent: Wednesday, October 05, 2011 6:25 PM To: Duddy, John Cc: galaxy-dev at lists.bx.psu.edu Subject: Re: [galaxy-dev] Tool shed and datatypes Hello John, The Galaxy tool shed currently is not enabled to a

Re: [galaxy-dev] Tool shed and datatypes

2011-10-06 Thread Greg Von Kuster
Software Engineer > Illumina, Inc. > 9885 Towne Centre Drive > San Diego, CA 92121 > Tel: 858-736-3584 > E-mail: jdu...@illumina.com > > From: Greg Von Kuster [mailto:g...@bx.psu.edu] > Sent: Wednesday, October 05, 2011 6:25 PM > To: Duddy, John > Cc: galaxy-dev@

Re: [galaxy-dev] Tool shed and datatypes

2011-10-06 Thread Duddy, John
y-dev@lists.bx.psu.edu; Nate Coraor Subject: Re: [galaxy-dev] Tool shed and datatypes On Thu, Oct 6, 2011 at 5:02 PM, Duddy, John wrote: > As I understand it, Isilion is built up from "bricks" that have storage > and compute power. They replicate files amongst themselves, so > t

Re: [galaxy-dev] Tool shed and datatypes

2011-10-06 Thread Peter Cock
On Thu, Oct 6, 2011 at 5:02 PM, Duddy, John wrote: > As I understand it, Isilion is built up from "bricks" that have storage > and compute power. They replicate files amongst themselves, so > that for every IO request there are multiple systems that could > respond. They are interconnected by an u

Re: [galaxy-dev] Tool shed and datatypes

2011-10-06 Thread Duddy, John
Duddy, John Cc: Greg Von Kuster; galaxy-dev@lists.bx.psu.edu; Nate Coraor Subject: Re: [galaxy-dev] Tool shed and datatypes On Thu, Oct 6, 2011 at 3:48 PM, Duddy, John wrote: > I'd be up for that something like that, although I have other tasking > in the short term after I finish my p

Re: [galaxy-dev] Tool shed and datatypes

2011-10-06 Thread Peter Cock
On Thu, Oct 6, 2011 at 3:48 PM, Duddy, John wrote: > I'd be up for that something like that, although I have other tasking > in the short term after I finish my parallelism work. I'd rather not have > Galaxy do the compression/decompression, because that will not > effectively utilize the distribu

Re: [galaxy-dev] Tool shed and datatypes

2011-10-06 Thread Duddy, John
galaxy-dev@lists.bx.psu.edu; Nate Coraor Subject: Re: [galaxy-dev] Tool shed and datatypes On Thu, Oct 6, 2011 at 4:48 AM, Duddy, John wrote: > One of the things we're facing is the sheer size of a whole human genome at > 30x coverage. An effective way to deal with that is by compressing th

Re: [galaxy-dev] Tool shed and datatypes

2011-10-06 Thread Peter Cock
On Thu, Oct 6, 2011 at 4:48 AM, Duddy, John wrote: > One of the things we’re facing is the sheer size of a whole human genome at > 30x coverage. An effective way to deal with that is by compressing the FASTQ > files. That works for BWA and our ELAND, which can directly read a > compressed FASTQ, b

Re: [galaxy-dev] Tool shed and datatypes

2011-10-05 Thread Duddy, John
6:25 PM To: Duddy, John Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Tool shed and datatypes Hello John, The Galaxy tool shed currently is not enabled to automatically edit the datatypes_conf.xml file, although I could add this feature if the need exists. Can you elaborate on what you a

Re: [galaxy-dev] Tool shed and datatypes

2011-10-05 Thread Greg Von Kuster
Hello John, The Galaxy tool shed currently is not enabled to automatically edit the datatypes_conf.xml file, although I could add this feature if the need exists. Can you elaborate on what you are looking to do regarding this? Thanks! On Oct 5, 2011, at 1:52 PM, Duddy, John wrote: > Can we