Re: [galaxy-dev] Workflows with Conditionals and Loops

[Aaron Petkau]

I remember this being mentioned in a project called Tavaxy
http://www.biomedcentral.com/1471-2105/13/77, but it's a separate piece of
software (using Galaxy) and I don't know if it's been updated in a while so
it may not be as useful for you.  Not sure if anybody else had any
experience with it?

It would be pretty cool to have conditionals and loops within workflows
though.

Aaron


On Wed, Mar 5, 2014 at 9:05 PM, Michael E. Cotterell  wrote:

> Is there a way to have conditionals and loops for the various steps in a
> Galaxy workflow?
>
> For example, a conditional might be used to branch to various steps based
> on the previous steps. Also, a step may need to be repeated until it
> produces a particular output.
>
> Here is an actual use case: We are invoking a web service in one of our
> workflows. The first step returns a unique identifier that you need to use
> later in order to get the actual results (since the web service operation
> is asynchronous). Later, using another operation, you can check on the
> status of the job being performed. Sometimes, this operation/step will
> inform us that the results are not ready. We would like to be able to loop
> on that second operation/step until it says the job is done, and then move
> on to a third operation/step to actually get the results.
>
> Any ideas?
>
> Sincerely,
> Michael E. Cotterell
>
> Ph.D. Student in Computer Science, University of Georgia
> Instructor of Record, Graduate RA & TA, University of Georgia
> Department Liaison, CS Graduate Student Association, University of Georgia
> mepcotter...@gmail.com (mailto:mepcotter...@gmail.com)
> mepc...@uga.edu (mailto:mepc...@uga.edu)
> m...@cs.uga.edu (mailto:m...@cs.uga.edu)
> http://michaelcotterell.com/
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Workflows with Conditionals and Loops

[Michael E. Cotterell]

I’m sorry for the confusion. These really are three different operations that 
handled by three different web service operations with the web service. Other 
WSDLs might have similar operations, but the process might be different. There 
is no generic way to do what you just described.  

Sincerely,
Michael E. Cotterell

Ph.D. Student in Computer Science, University of Georgia
Instructor of Record, Graduate RA & TA, University of Georgia
Department Liaison, CS Graduate Student Association, University of Georgia
mepcotter...@gmail.com (mailto:mepcotter...@gmail.com)
mepc...@uga.edu (mailto:mepc...@uga.edu)
m...@cs.uga.edu (mailto:m...@cs.uga.edu)
http://michaelcotterell.com/


On Thursday, March 6, 2014 at 10:01 AM, Eric Rasche wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>  
> We're not suggesting adding a "polling tool"; instead the suggestion was
> that your "WSDL tool generator" generate the XML tool definition and a
> wrapper script.
>  
> That wrapper script would handle whatever WSDL operation it was created
> for, polling, and collection of results (ALL IN ONE). Since polling is a
> WSDL specific operation as you say, this would solve that, as every
> generated tool would have code to submit+poll+collect.
>  
>  
> Conditionals and loops would generate /many/ "empty" jobs saying "your
> results are not done yet", which would not lend themselves to inclusion
> in workflows. galaxy workflow loops are probably not the best way to
> solve this particular problem.
>  
> Additionally, If you write your WSDL tool generator in such a way that
> the generated tools handles the entire process: from submission to
> retrieval of results, then they could be used in workflows and used with
> the existing infrastructure, and jobs down the line could depend on the
> output of your generated tools.
>  
> Cheers,
> Eric
>  
> On 03/06/2014 08:25 AM, Michael E. Cotterell wrote:
> > The tool is generic. This means that it is supposed to work (and does work) 
> > with any WSDL. In the described use case, one of the operations/“generated 
> > tools" already serves as a polling tool: it takes a unique id and requests 
> > a result from the web service, then the web service either returns the 
> > results or returns a message indicating that the results are not ready. 
> > Conditionals/Loops would be a perfect way to solve this.
> >  
> >  
> > If we added a polling tool like you described, then that tool would need to 
> > Web service specific since not all web service adhere to the same naming 
> > conventions and level of asynchrony.  
> >  
> > Sincerely,
> > Michael E. Cotterell
> >  
> > Ph.D. Student in Computer Science, University of Georgia
> > Instructor of Record, Graduate RA & TA, University of Georgia
> > Department Liaison, CS Graduate Student Association, University of Georgia
> > mepcotter...@gmail.com (mailto:mepcotter...@gmail.com)
> > mepc...@uga.edu (mailto:mepc...@uga.edu)
> > m...@cs.uga.edu (mailto:m...@cs.uga.edu)
> > http://michaelcotterell.com/
> >  
> >  
> > On Thursday, March 6, 2014 at 9:17 AM, Peter Cock wrote:
> >  
> > > On Thu, Mar 6, 2014 at 2:12 PM, Michael E. Cotterell
> > > mailto:mepcotter...@gmail.com)> wrote:
> > > > While I agree that would work, the tool I'm working with generates
> > > > tools for web operations in a generic fashion. That is, you provide
> > > > it a WDSL and a list of operations you want from that WSDL, and
> > > > then tool XML files are generated for each of those operations.
> > >  
> > >  
> > >  
> > >  
> > >  
> > > So could your "WDSL Galaxy Tool Factory" also produce a wrapper
> > > script to go with the Galaxy Tool XML, where the wrapper script
> > > handles polling the service with the unique identifier assigned by
> > > the service?
> > >  
> > > Peter  
>  
>  
> - --  
> Eric Rasche
> Programmer II
> Center for Phage Technology
> Texas A&M University
> College Station, TX 77843
> 404-692-2048
> e...@tamu.edu (mailto:e...@tamu.edu)
> rasche.e...@yandex.ru (mailto:rasche.e...@yandex.ru)
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v2.0.17 (GNU/Linux)
>  
> iQIcBAEBAgAGBQJTGI2vAAoJEMqDXdrsMcpVMsAQALuigNEIfG0qqR2pxULOEK39
> GYkJ9q8RFRgmQAr5STM97XoeZD+mYyqzjrsdB8XL6LldkN0+XN+Aak8I4KbQlidu
> cWnZtjeY46DAxQdOcEzHCfL0apjgeOb+uCcxAtewVUBC4QMU11GkoL+54ZKoNWVV
> Vitt8N0hM4WJE+MukwZ9tCjjvpW7sVM1TUfx0bBb47qyx+sxH14JiaICBYpp6jDP
> Y3Kx/ncMwr62O2pMDuoP6YHn+ozKRwKa85uHEKx9mv/aQ1/0/F2Y3sd3hoj2fv4k
> wxw+7GY05CBUat6d4OH3bq8K6C2Z7Ye/nQc1FClM4ssjfXUEulXgzbOwwxmoyPqw
> fQPGIvWxr2YrewYsTScIO2m7kg7Y70d2gW+jRHFZJd9Yiz90ze5et5veqJ9vw7sl
> v71jZkkANF8ONdHE4cIkEYBbsNuhTJvSaYOvt0YAhMe18vvBe/AwQ4AwWKcsRYhn
> CnilCjICOcwJFDrRsVtAIIU/PmEl9w5tztRozHe+RtMNt4PFyn7qVdlgsFQogEQm
> FWhIpTAjioyc/46kIg5vqcA0BR8KJNllCOiFDDI7ZGJJAzykPjEeZshbQcDdA0Do
> Axfrr1ata/iHZsIuXsMQrVfiY8lr5XPimdXAdMPxdt+jThUP803lwDIgu9gD7wWm
> LBL7haO0S65UDiZ4Be8m
> =SvuS
> -END PGP SIGNATURE-
> ___
> Please keep all replies on the lis

Re: [galaxy-dev] Workflows with Conditionals and Loops

[Eric Rasche]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

We're not suggesting adding a "polling tool"; instead the suggestion was
that your "WSDL tool generator" generate the XML tool definition and a
wrapper script.

That wrapper script would handle whatever WSDL operation it was created
for, polling, and collection of results (ALL IN ONE). Since polling is a
WSDL specific operation as you say, this would solve that, as every
generated tool would have code to submit+poll+collect.


Conditionals and loops would generate /many/ "empty" jobs saying "your
results are not done yet", which would not lend themselves to inclusion
in workflows. galaxy workflow loops are probably not the best way to
solve this particular problem.

Additionally, If you write your WSDL tool generator in such a way that
the generated tools handles the entire process: from submission to
retrieval of results, then they could be used in workflows and used with
the existing infrastructure, and jobs down the line could depend on the
output of your generated tools.

Cheers,
Eric

On 03/06/2014 08:25 AM, Michael E. Cotterell wrote:
> The tool is generic. This means that it is supposed to work (and does work) 
> with any WSDL. In the described use case, one of the operations/“generated 
> tools" already serves as a polling tool: it takes a unique id and requests a 
> result from the web service, then the web service either returns the results 
> or returns a message indicating that the results are not ready. 
> Conditionals/Loops would be a perfect way to solve this.
> 
> 
> If we added a polling tool like you described, then that tool would need to 
> Web service specific since not all web service adhere to the same naming 
> conventions and level of asynchrony.  
> 
> Sincerely,
> Michael E. Cotterell
> 
> Ph.D. Student in Computer Science, University of Georgia
> Instructor of Record, Graduate RA & TA, University of Georgia
> Department Liaison, CS Graduate Student Association, University of Georgia
> mepcotter...@gmail.com (mailto:mepcotter...@gmail.com)
> mepc...@uga.edu (mailto:mepc...@uga.edu)
> m...@cs.uga.edu (mailto:m...@cs.uga.edu)
> http://michaelcotterell.com/
> 
> 
> On Thursday, March 6, 2014 at 9:17 AM, Peter Cock wrote:
> 
>> On Thu, Mar 6, 2014 at 2:12 PM, Michael E. Cotterell
>> mailto:mepcotter...@gmail.com)> wrote:
>>> While I agree that would work, the tool I'm working with generates
>>> tools for web operations in a generic fashion. That is, you provide
>>> it a WDSL and a list of operations you want from that WSDL, and
>>> then tool XML files are generated for each of those operations.
>>  
>>  
>>  
>> So could your "WDSL Galaxy Tool Factory" also produce a wrapper
>> script to go with the Galaxy Tool XML, where the wrapper script
>> handles polling the service with the unique identifier assigned by
>> the service?
>>  
>> Peter  
> 
> 

- -- 
Eric Rasche
Programmer II
Center for Phage Technology
Texas A&M University
College Station, TX 77843
404-692-2048
e...@tamu.edu
rasche.e...@yandex.ru
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (GNU/Linux)

iQIcBAEBAgAGBQJTGI2vAAoJEMqDXdrsMcpVMsAQALuigNEIfG0qqR2pxULOEK39
GYkJ9q8RFRgmQAr5STM97XoeZD+mYyqzjrsdB8XL6LldkN0+XN+Aak8I4KbQlidu
cWnZtjeY46DAxQdOcEzHCfL0apjgeOb+uCcxAtewVUBC4QMU11GkoL+54ZKoNWVV
Vitt8N0hM4WJE+MukwZ9tCjjvpW7sVM1TUfx0bBb47qyx+sxH14JiaICBYpp6jDP
Y3Kx/ncMwr62O2pMDuoP6YHn+ozKRwKa85uHEKx9mv/aQ1/0/F2Y3sd3hoj2fv4k
wxw+7GY05CBUat6d4OH3bq8K6C2Z7Ye/nQc1FClM4ssjfXUEulXgzbOwwxmoyPqw
fQPGIvWxr2YrewYsTScIO2m7kg7Y70d2gW+jRHFZJd9Yiz90ze5et5veqJ9vw7sl
v71jZkkANF8ONdHE4cIkEYBbsNuhTJvSaYOvt0YAhMe18vvBe/AwQ4AwWKcsRYhn
CnilCjICOcwJFDrRsVtAIIU/PmEl9w5tztRozHe+RtMNt4PFyn7qVdlgsFQogEQm
FWhIpTAjioyc/46kIg5vqcA0BR8KJNllCOiFDDI7ZGJJAzykPjEeZshbQcDdA0Do
Axfrr1ata/iHZsIuXsMQrVfiY8lr5XPimdXAdMPxdt+jThUP803lwDIgu9gD7wWm
LBL7haO0S65UDiZ4Be8m
=SvuS
-END PGP SIGNATURE-
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Workflows with Conditionals and Loops

[Michael E. Cotterell]

The tool is generic. This means that it is supposed to work (and does work) 
with any WSDL. In the described use case, one of the operations/“generated 
tools" already serves as a polling tool: it takes a unique id and requests a 
result from the web service, then the web service either returns the results or 
returns a message indicating that the results are not ready. Conditionals/Loops 
would be a perfect way to solve this.


If we added a polling tool like you described, then that tool would need to Web 
service specific since not all web service adhere to the same naming 
conventions and level of asynchrony.  

Sincerely,
Michael E. Cotterell

Ph.D. Student in Computer Science, University of Georgia
Instructor of Record, Graduate RA & TA, University of Georgia
Department Liaison, CS Graduate Student Association, University of Georgia
mepcotter...@gmail.com (mailto:mepcotter...@gmail.com)
mepc...@uga.edu (mailto:mepc...@uga.edu)
m...@cs.uga.edu (mailto:m...@cs.uga.edu)
http://michaelcotterell.com/


On Thursday, March 6, 2014 at 9:17 AM, Peter Cock wrote:

> On Thu, Mar 6, 2014 at 2:12 PM, Michael E. Cotterell
> mailto:mepcotter...@gmail.com)> wrote:
> > While I agree that would work, the tool I'm working with generates
> > tools for web operations in a generic fashion. That is, you provide
> > it a WDSL and a list of operations you want from that WSDL, and
> > then tool XML files are generated for each of those operations.
>  
>  
>  
> So could your "WDSL Galaxy Tool Factory" also produce a wrapper
> script to go with the Galaxy Tool XML, where the wrapper script
> handles polling the service with the unique identifier assigned by
> the service?
>  
> Peter  



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Workflows with Conditionals and Loops

[Peter Cock]

On Thu, Mar 6, 2014 at 2:12 PM, Michael E. Cotterell
 wrote:
> While I agree that would work, the tool I'm working with generates
> tools for web operations in a generic fashion. That is, you provide
> it a WDSL and a list of operations you want from that WSDL, and
> then tool XML files are generated for each of those operations.
>

So could your "WDSL Galaxy Tool Factory" also produce a wrapper
script to go with the Galaxy Tool XML, where the wrapper script
handles polling the service with the unique identifier assigned by
the service?

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Workflows with Conditionals and Loops

[Michael E. Cotterell]

While I agree that would work, the tool I’m working with generates tools for 
web operations in a generic fashion. That is, you provide it a WDSL and a list 
of operations you want from that WSDL, and then tool XML files are generated 
for each of those operations.   

Sincerely,
Michael E. Cotterell

Ph.D. Student in Computer Science, University of Georgia
Instructor of Record, Graduate RA & TA, University of Georgia
Department Liaison, CS Graduate Student Association, University of Georgia
mepcotter...@gmail.com (mailto:mepcotter...@gmail.com)
mepc...@uga.edu (mailto:mepc...@uga.edu)
m...@cs.uga.edu (mailto:m...@cs.uga.edu)
http://michaelcotterell.com/


On Thursday, March 6, 2014 at 9:08 AM, Eric Rasche wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>  
> Hi Michael,
>  
> I, too, would love to have conditionals and loops, however may want to
> consider the following approach:
>  
> Instead of splitting the web service submission, check for results, and
> download of results into three steps, make it a single script.
>  
> I believe this would solve your problem completely. It largely simplify
> your use case, as a single script could
>  
> - - submit and remember a job's ID
> - - check periodically using that language's web bindings
> - - collect the results when done
>  
> This would mean it's a long-running task, which is perfectly suited to
> use in galaxy.
>  
> Cheers,
> Eric
>  
> On 03/06/2014 07:51 AM, John Chilton wrote:
> > Hey Michael,
> >  
> > I know there is a lot of interest in this, but these sorts of things
> > are not possible with Galaxy workflows at this time. If you want the
> > traceibility and UI for viewing results provided by Galaxy - you could
> > consider driving tools externally via the API - at that point though
> > the utility of running these jobs through Galaxy has dropped
> > considerably.
> >  
> > There is a Trello card I would encourage you to vote for -
> > https://trello.com/c/rwSeAwos
> >  
> > I have included a link to this e-mail as a reference for specific use
> > cases to consider if or when this idea gets worked on.
> >  
> > Sorry,
> > -John
> >  
> >  
> > On Wed, Mar 5, 2014 at 9:05 PM, Michael E. Cotterell
> > mailto:mepcotter...@gmail.com)> wrote:
> > > Is there a way to have conditionals and loops for the various steps in a 
> > > Galaxy workflow?
> > >  
> > > For example, a conditional might be used to branch to various steps based 
> > > on the previous steps. Also, a step may need to be repeated until it 
> > > produces a particular output.
> > >  
> > > Here is an actual use case: We are invoking a web service in one of our 
> > > workflows. The first step returns a unique identifier that you need to 
> > > use later in order to get the actual results (since the web service 
> > > operation is asynchronous). Later, using another operation, you can check 
> > > on the status of the job being performed. Sometimes, this operation/step 
> > > will inform us that the results are not ready. We would like to be able 
> > > to loop on that second operation/step until it says the job is done, and 
> > > then move on to a third operation/step to actually get the results.
> > >  
> > > Any ideas?
> > >  
> > > Sincerely,
> > > Michael E. Cotterell
> > >  
> > > Ph.D. Student in Computer Science, University of Georgia
> > > Instructor of Record, Graduate RA & TA, University of Georgia
> > > Department Liaison, CS Graduate Student Association, University of Georgia
> > > mepcotter...@gmail.com (mailto:mepcotter...@gmail.com)
> > > mepc...@uga.edu (mailto:mepc...@uga.edu)
> > > m...@cs.uga.edu (mailto:m...@cs.uga.edu)
> > > http://michaelcotterell.com/
> > >  
> > >  
> > > ___
> > > Please keep all replies on the list by using "reply all"
> > > in your mail client. To manage your subscriptions to this
> > > and other Galaxy lists, please use the interface at:
> > > http://lists.bx.psu.edu/
> > >  
> > > To search Galaxy mailing lists use the unified search at:
> > > http://galaxyproject.org/search/mailinglists/
> >  
> >  
> >  
> > ___
> > Please keep all replies on the list by using "reply all"
> > in your mail client. To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> > http://lists.bx.psu.edu/
> >  
> > To search Galaxy mailing lists use the unified search at:
> > http://galaxyproject.org/search/mailinglists/
>  
>  
>  
> - --  
> Eric Rasche
> Programmer II
> Center for Phage Technology
> Texas A&M University
> College Station, TX 77843
> 404-692-2048
> e...@tamu.edu (mailto:e...@tamu.edu)
> rasche.e...@yandex.ru (mailto:rasche.e...@yandex.ru)
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v2.0.17 (GNU/Linux)
>  
> iQIcBAEBAgAGBQJTGIFFAAoJEMqDXdrsMcpV6hsQAKGv/W0dtZXAiuf855D2PU8k
> cbyMxliigbvlg0ieZNb0ZFAsx5LHRuek8sU0HMf0v14wxgs6vlwDtp143x5R0MIY
> zjhNQkj+C9IXyq/qjrqT8IObbMUGgPFJD/hzrP9YCT

Re: [galaxy-dev] Workflows with Conditionals and Loops

[Eric Rasche]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Michael,

I, too, would love to have conditionals and loops, however may want to
consider the following approach:

Instead of splitting the web service submission, check for results, and
download of results into three steps, make it a single script.

I believe this would solve your problem completely. It largely simplify
your use case, as a single script could

- - submit and remember a job's ID
- - check periodically using that language's web bindings
- - collect the results when done

This would mean it's a long-running task, which is perfectly suited to
use in galaxy.

Cheers,
Eric

On 03/06/2014 07:51 AM, John Chilton wrote:
> Hey Michael,
> 
> I know there is a lot of interest in this, but these sorts of things
> are not possible with Galaxy workflows at this time. If you want the
> traceibility and UI for viewing results provided by Galaxy - you could
> consider driving tools externally via the API - at that point though
> the utility of running these jobs through Galaxy has dropped
> considerably.
> 
> There is a Trello card I would encourage you to vote for -
> https://trello.com/c/rwSeAwos
> 
> I have included a link to this e-mail as a reference for specific use
> cases to consider if or when this idea gets worked on.
> 
> Sorry,
> -John
> 
> 
> On Wed, Mar 5, 2014 at 9:05 PM, Michael E. Cotterell
>  wrote:
>> Is there a way to have conditionals and loops for the various steps in a 
>> Galaxy workflow?
>>
>> For example, a conditional might be used to branch to various steps based on 
>> the previous steps. Also, a step may need to be repeated until it produces a 
>> particular output.
>>
>> Here is an actual use case: We are invoking a web service in one of our 
>> workflows. The first step returns a unique identifier that you need to use 
>> later in order to get the actual results (since the web service operation is 
>> asynchronous). Later, using another operation, you can check on the status 
>> of the job being performed. Sometimes, this operation/step will inform us 
>> that the results are not ready. We would like to be able to loop on that 
>> second operation/step until it says the job is done, and then move on to a 
>> third operation/step to actually get the results.
>>
>> Any ideas?
>>
>> Sincerely,
>> Michael E. Cotterell
>>
>> Ph.D. Student in Computer Science, University of Georgia
>> Instructor of Record, Graduate RA & TA, University of Georgia
>> Department Liaison, CS Graduate Student Association, University of Georgia
>> mepcotter...@gmail.com (mailto:mepcotter...@gmail.com)
>> mepc...@uga.edu (mailto:mepc...@uga.edu)
>> m...@cs.uga.edu (mailto:m...@cs.uga.edu)
>> http://michaelcotterell.com/
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
> 
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
> 

- -- 
Eric Rasche
Programmer II
Center for Phage Technology
Texas A&M University
College Station, TX 77843
404-692-2048
e...@tamu.edu
rasche.e...@yandex.ru
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (GNU/Linux)

iQIcBAEBAgAGBQJTGIFFAAoJEMqDXdrsMcpV6hsQAKGv/W0dtZXAiuf855D2PU8k
cbyMxliigbvlg0ieZNb0ZFAsx5LHRuek8sU0HMf0v14wxgs6vlwDtp143x5R0MIY
zjhNQkj+C9IXyq/qjrqT8IObbMUGgPFJD/hzrP9YCT7gUKE9Dy1fxN2aznUJgJJU
fmMD48E0/+7/y4dwdRaZ/LQv9jAKHKg1S6df6rYYWa1/hZtWrAGYrycygcrnt0aW
Ai0iE6IXDTbPYZSjNcWLMPJ7FPU/MGdIF1Lia1PB9GLAqwY5Iln6WJ6OUdahNQzw
Yx+0Te80r6wZVKHb5Bx8bszOEfPzxHcR0XABGn6r6zREoFD1kTQqugWyn0FL/v0C
zibHMbFje+LDgUU47QoyNRGXcM6RoiVJGo7KTcDbP5HEWqQ7ozmZSN2ECeX1s1gf
mzE9sJadLH3P8+vA6wO+xKtPpEzvC8VJHFROZNPJNVpVFAKU1RBRSo8qD1ZDZtTx
LnwXmx0lK0spaJXXGg8g8tO23ilwy0JXRqxt56oPp7MlUtDJ7eDyQtmdQlPlNY4v
+eqeJlGoRBlT3AS7CMfY4PWhrg3WZ0dLS+w/bG7Pg4uOj+iZ1qVEHDS4ugwqszNK
eksGuXb/Uc9GGwB1xUssF9XMwf6u/xK9OqCuW8rgzFhm85BhqEJFqiJO408Ibx/S
HPnswg0jhak5c8DKlajS
=TirM
-END PGP SIGNATURE-
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Workflows with Conditionals and Loops

[John Chilton]

Hey Michael,

I know there is a lot of interest in this, but these sorts of things
are not possible with Galaxy workflows at this time. If you want the
traceibility and UI for viewing results provided by Galaxy - you could
consider driving tools externally via the API - at that point though
the utility of running these jobs through Galaxy has dropped
considerably.

There is a Trello card I would encourage you to vote for -
https://trello.com/c/rwSeAwos

I have included a link to this e-mail as a reference for specific use
cases to consider if or when this idea gets worked on.

Sorry,
-John


On Wed, Mar 5, 2014 at 9:05 PM, Michael E. Cotterell
 wrote:
> Is there a way to have conditionals and loops for the various steps in a 
> Galaxy workflow?
>
> For example, a conditional might be used to branch to various steps based on 
> the previous steps. Also, a step may need to be repeated until it produces a 
> particular output.
>
> Here is an actual use case: We are invoking a web service in one of our 
> workflows. The first step returns a unique identifier that you need to use 
> later in order to get the actual results (since the web service operation is 
> asynchronous). Later, using another operation, you can check on the status of 
> the job being performed. Sometimes, this operation/step will inform us that 
> the results are not ready. We would like to be able to loop on that second 
> operation/step until it says the job is done, and then move on to a third 
> operation/step to actually get the results.
>
> Any ideas?
>
> Sincerely,
> Michael E. Cotterell
>
> Ph.D. Student in Computer Science, University of Georgia
> Instructor of Record, Graduate RA & TA, University of Georgia
> Department Liaison, CS Graduate Student Association, University of Georgia
> mepcotter...@gmail.com (mailto:mepcotter...@gmail.com)
> mepc...@uga.edu (mailto:mepc...@uga.edu)
> m...@cs.uga.edu (mailto:m...@cs.uga.edu)
> http://michaelcotterell.com/
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Workflows with Conditionals and Loops

[Michael E. Cotterell]

Is there a way to have conditionals and loops for the various steps in a Galaxy 
workflow?

For example, a conditional might be used to branch to various steps based on 
the previous steps. Also, a step may need to be repeated until it produces a 
particular output. 

Here is an actual use case: We are invoking a web service in one of our 
workflows. The first step returns a unique identifier that you need to use 
later in order to get the actual results (since the web service operation is 
asynchronous). Later, using another operation, you can check on the status of 
the job being performed. Sometimes, this operation/step will inform us that the 
results are not ready. We would like to be able to loop on that second 
operation/step until it says the job is done, and then move on to a third 
operation/step to actually get the results.

Any ideas? 

Sincerely,
Michael E. Cotterell

Ph.D. Student in Computer Science, University of Georgia
Instructor of Record, Graduate RA & TA, University of Georgia
Department Liaison, CS Graduate Student Association, University of Georgia
mepcotter...@gmail.com (mailto:mepcotter...@gmail.com)
mepc...@uga.edu (mailto:mepc...@uga.edu)
m...@cs.uga.edu (mailto:m...@cs.uga.edu)
http://michaelcotterell.com/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Workflows with multiple editors

[John Chilton]

Hey Eric,

  Thanks for thinking through some of those details. I don't know if
this is an extension of that or an alternative to that - but the
Galaxy tool shed has some support for workflows and stored these in a
VCS repository (mercurial). Augmenting that to support a sort of
one-click (or couple click) export from Galaxy to the tool shed was
something on my radar, but going further and providing tracking,
branching, etc... over time of the workflow as you have outlined here
could make a lot of sense.

  Echoing his comments here, I know Peter is not a fan of tool shed
repositories being implemented as mercurial repositories. Continuing
to make progress on hiding this fact from users while allowing
leveraging of more advanced features as you described here should help
that decision bear more fruit going forward.

  The other feature requests related to allowing multiple editors - I
think this is a great idea. I think histories, pages, and workflows
all need to support more sophisticated permissions.

  I have created Trello cards related to these requests here, feel
free to comment on these to flush out the details:
https://trello.com/c/3blHqhxb
https://trello.com/c/YvKxFOdd

Thanks again for the suggestions,
-John

On Thu, Oct 31, 2013 at 12:16 PM, Eric Rasche  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi Peter,
>
> Please bear with me, I'm mostly tossing these ideas out for my
> benefit/to spawn discussion on how one might implement this. This was
> originally a short email before I figured out how much work would be
> involved...
>
> Upon cursory inspection I see that workflows are stored as lines in the
> database. Were it to be easily stored in a VCS, all database workflow
> steps would probably have to be migrated to disk. In the toolsheds
> workflows are stored as JSON, so it'd just be a matter of dumping
> existing internal workflows to disk.
>
> On disk it would be stored in a "workflow" folder, with a set of
> numbered sub-folders. Each subfolder would store a VCS repo. These
> numbered sub-folders would be referenced in the database.
> Additionally, there would be a "cache" folder in the workflow folder, to
> store the current/active version of the workflow, per workflow, per user.
>
> Continuing just to toss ideas out, I could see implementing this in a
> VCS while hiding as much of that as possible away from the user. How
> about this:
>
> - - Sharing creates a tag in the vcs (your permanent, immutable version)
> - - If it was shared to them with edit capabilities, a branch is created
> to allow them to make their changes and eventually merge back.
> - - Upon save, we'd ask them if they'd like to add a short message
> describing their changes (by default it'd read "{UserRealName} made
> modifications" or something like that so they could just click through
> if they didn't want to write a message)
> - - In the edit view, there'd be an additional button which reads "Merge
> these changes back into {Sharer'sRealName}'s copy of this workflow"
> which would execute a merge (I guess it could create a merge request but
> that seems like a bit much)
> - - In the workflow list we would now have an area to select previous
> revisions of workflows and open those up in the edit window.
> - - selecting a previous version of the workflow would set the user's
> active workflow (somewhere in DB) to that revision.
>
> Keeping track of which workflow was "active" might be a bit difficult...
> I would imagine it would work like so:
>
> - - workflows are separated from users.
> - - the DB table of "stored_workflow" would be modified to store which
> repository (in the workflow/NNN/ folder), as well as the current head
> - - a table would be created, mapping users to specific revisions of a
> workflow. (workflow_user_revision) [user_id,workflow_id,revision]
> - maybe revision would be either revision hash or "HEAD" to say they
> always want the latest version?
> - - sharing a workflow (read/write) would add a new row in the
> workflow_user_revision, with an updated user_id, and a copy of the that
> user's current revision hash (because that user would be sharing that
> specific version).
> - - sharing a workflow (read only) would do pretty much the same.
>
>
> This made sense in my mind at least...does this seem like a reasonable
> way to implement this? Do you (or anyone else) have comments to make on
> this proposal?
>
> (Mind you, this is just me thinking out loud for the time being...I have
> a lot of other galaxy modifications to get through first for my
> organisation (native LDAP integration, script ACLs))
>
>
>
>
>
> On 10/30/2013 05:11 PM, Peter Cock wrote:
>> Hi Eric,
>>
>> I've being pondering this too - it can easily get very complicated
>> with multiple copies of a workflow with minor variations.
>>
>> I agree that "read only" sharing makes sense as a short term
>> goal, but I see two variants being useful here - an read only
>> snapshot copy from the instant it was shared (

Re: [galaxy-dev] Workflows with multiple editors

[Eric Rasche]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Peter,

Please bear with me, I'm mostly tossing these ideas out for my
benefit/to spawn discussion on how one might implement this. This was
originally a short email before I figured out how much work would be
involved...

Upon cursory inspection I see that workflows are stored as lines in the
database. Were it to be easily stored in a VCS, all database workflow
steps would probably have to be migrated to disk. In the toolsheds
workflows are stored as JSON, so it'd just be a matter of dumping
existing internal workflows to disk.

On disk it would be stored in a "workflow" folder, with a set of
numbered sub-folders. Each subfolder would store a VCS repo. These
numbered sub-folders would be referenced in the database.
Additionally, there would be a "cache" folder in the workflow folder, to
store the current/active version of the workflow, per workflow, per user.

Continuing just to toss ideas out, I could see implementing this in a
VCS while hiding as much of that as possible away from the user. How
about this:

- - Sharing creates a tag in the vcs (your permanent, immutable version)
- - If it was shared to them with edit capabilities, a branch is created
to allow them to make their changes and eventually merge back.
- - Upon save, we'd ask them if they'd like to add a short message
describing their changes (by default it'd read "{UserRealName} made
modifications" or something like that so they could just click through
if they didn't want to write a message)
- - In the edit view, there'd be an additional button which reads "Merge
these changes back into {Sharer'sRealName}'s copy of this workflow"
which would execute a merge (I guess it could create a merge request but
that seems like a bit much)
- - In the workflow list we would now have an area to select previous
revisions of workflows and open those up in the edit window.
- - selecting a previous version of the workflow would set the user's
active workflow (somewhere in DB) to that revision.

Keeping track of which workflow was "active" might be a bit difficult...
I would imagine it would work like so:

- - workflows are separated from users.
- - the DB table of "stored_workflow" would be modified to store which
repository (in the workflow/NNN/ folder), as well as the current head
- - a table would be created, mapping users to specific revisions of a
workflow. (workflow_user_revision) [user_id,workflow_id,revision]
- maybe revision would be either revision hash or "HEAD" to say they
always want the latest version?
- - sharing a workflow (read/write) would add a new row in the
workflow_user_revision, with an updated user_id, and a copy of the that
user's current revision hash (because that user would be sharing that
specific version).
- - sharing a workflow (read only) would do pretty much the same.


This made sense in my mind at least...does this seem like a reasonable
way to implement this? Do you (or anyone else) have comments to make on
this proposal?

(Mind you, this is just me thinking out loud for the time being...I have
a lot of other galaxy modifications to get through first for my
organisation (native LDAP integration, script ACLs))





On 10/30/2013 05:11 PM, Peter Cock wrote:
> Hi Eric,
> 
> I've being pondering this too - it can easily get very complicated
> with multiple copies of a workflow with minor variations.
> 
> I agree that "read only" sharing makes sense as a short term
> goal, but I see two variants being useful here - an read only
> snapshot copy from the instant it was shared (where the shared
> version is locked forever but may of course optionally be
> copied by the receiver, and the copy edited), and as an
> always up to date live pointer (in case the original owner
> revises it - then all the users get the changes immediately).
> 
> Some of your wish list  ideas also sound very good, but may
> not be practical - not everyone will understand a VCS
> approach to managing workflows :(
> 
> Regards,
> 
> Peter
> 
> 
> On Wed, Oct 30, 2013 at 8:02 PM, Eric Rasche  wrote:
> I'm working with a number of people in my department to develop a single
> workflow that will be used for a course we teach. So far I've found that
> "sharing" a workflow with individual users/roles isn't very optimal.
> They email me with changes they'd like to happen, or they create their
> own copies which immediately branch off and don't help everyone else who
> might need them.
> 
> As such, I'd like to propose changes and would like community feedback
> on my proposed changes so when I create the trello card/maybe try my
> hand at implementing them, they aren't completely specific to my use case.
> 
> In my mind, the 'optimal' version of managing workflows would be
> something along the following lines:
> 
> Top priorities:
> 
> - workflows can be shared as "write" or "read only" with multiple
> users/roles.
> - This would allow there to be a single copy of a workflow in cases
> where multiple p

Re: [galaxy-dev] Workflows with multiple editors

[Peter Cock]

Hi Eric,

I've being pondering this too - it can easily get very complicated
with multiple copies of a workflow with minor variations.

I agree that "read only" sharing makes sense as a short term
goal, but I see two variants being useful here - an read only
snapshot copy from the instant it was shared (where the shared
version is locked forever but may of course optionally be
copied by the receiver, and the copy edited), and as an
always up to date live pointer (in case the original owner
revises it - then all the users get the changes immediately).

Some of your wish list  ideas also sound very good, but may
not be practical - not everyone will understand a VCS
approach to managing workflows :(

Regards,

Peter


On Wed, Oct 30, 2013 at 8:02 PM, Eric Rasche  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> I'm working with a number of people in my department to develop a single
> workflow that will be used for a course we teach. So far I've found that
> "sharing" a workflow with individual users/roles isn't very optimal.
> They email me with changes they'd like to happen, or they create their
> own copies which immediately branch off and don't help everyone else who
> might need them.
>
> As such, I'd like to propose changes and would like community feedback
> on my proposed changes so when I create the trello card/maybe try my
> hand at implementing them, they aren't completely specific to my use case.
>
> In my mind, the 'optimal' version of managing workflows would be
> something along the following lines:
>
> Top priorities:
>
> - - workflows can be shared as "write" or "read only" with multiple
> users/roles.
> - This would allow there to be a single copy of a workflow in cases
> where multiple people might make changes to a single workflow
> - For "read only", this allows the previous/traditional model of
> sharing a workflow and the receiver being able to run it as is, or to
> copy and modify it.
>
> Note:
>
> For my case at least, realtime multiple person editing is not needed. If
> a banner shows up which reads "someone started editing before you,
> either ask them to save + leave or save as a copy" would work in my case.
>
> Wishlist features:
>
> - - workflows are treated as VCS repositories
> - Note that full merge/branching probably not really necessary for 
> this
> case (that would be really nice, but probably too much of a nightmare to
> code...)
> - - You can see previous iterations of workflows.
> - - Workflows can be "owned" by either a user or a role
> - users could create a workflow and transfer ownership to a role
> - role owned workflows are delete-able (or maybe require some X% of
> users in that role to confirmation deletion?)
> - This would sort of work like a group owned VCS repository, where
> multiple users can create workflows of interest to a group of people
> (role) and have them, by default, be available.
>
>
>
> Anyone have input on this proposal?
>
> Cheers,
> Eric
>
> - --
> Eric Rasche
> Programmer II
> Center for Phage Technology
> Texas A&M University
> College Station, TX 77843
> 404-692-2048
> e...@tamu.edu
> rasche.e...@yandex.ru
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBAgAGBQJScWXTAAoJEMqDXdrsMcpVFH4QAMO0V4tSJGzM3PGGIwdonY0v
> nyuHTCJJ5qliWuapUJUK9E0UQBblnaJYooLjh2l1vgo95JaXdHEAC9NVecQLH8J2
> ey5cu1xODf6JmBGlicRGxQWlLZmo9U3R4T/OhHco40I239aTpuSDPcwcBJUm1cAE
> czz+/2pgdCpkJqYQJ9EE956TAYdyPBYFvSrIO7ebWkSvC5He40qca0x06tgq14uR
> Dta+8QH8brVYX1G+TAgdh3hGlLUwIlPMiZsWOEMcwSUtyXP3I7OKx/XIOy3ZYVTu
> Py/XlvNlZ87yiaiwDSxjxR9qaDeGQ3J8CRCG9mqWu8O5Y9QKkyrC410I+hZH5lE+
> my28z7l5rK0i05pR6o8euASJ7D1eFHEDYmBmN6j3qRcOgzzdup2oQpm6TA7ue1sd
> NAS4rnW7QGmfNj+9xNSQhDWSj7CeIUeHXbDe55suBX3JofCNg8Qq6uYzPd7KGj7n
> jvWp8tGnpwbgTmvDPlR03gJiXMZld+gGwKrlCyn1kdBkfI1r5l0UfoaIUE+8HDm3
> wXviQ8GU7seYU5bJFK798owVNawjY1H3JenzIYt1XvF1z53Uc39x58Q9lxXbVfSe
> KgiqCfF0YuvVtOczGTAQ57qCFwlpKg2ra+X+HtAZyMDNMsb3ANgfogTqsB+7HYvF
> czq2QJHq9mBC5N9JLhRo
> =conD
> -END PGP SIGNATURE-
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Workflows with multiple editors

[Eric Rasche]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

I'm working with a number of people in my department to develop a single
workflow that will be used for a course we teach. So far I've found that
"sharing" a workflow with individual users/roles isn't very optimal.
They email me with changes they'd like to happen, or they create their
own copies which immediately branch off and don't help everyone else who
might need them.

As such, I'd like to propose changes and would like community feedback
on my proposed changes so when I create the trello card/maybe try my
hand at implementing them, they aren't completely specific to my use case.

In my mind, the 'optimal' version of managing workflows would be
something along the following lines:

Top priorities:

- - workflows can be shared as "write" or "read only" with multiple
users/roles.
- This would allow there to be a single copy of a workflow in cases
where multiple people might make changes to a single workflow
- For "read only", this allows the previous/traditional model of
sharing a workflow and the receiver being able to run it as is, or to
copy and modify it.

Note:

For my case at least, realtime multiple person editing is not needed. If
a banner shows up which reads "someone started editing before you,
either ask them to save + leave or save as a copy" would work in my case.

Wishlist features:

- - workflows are treated as VCS repositories
- Note that full merge/branching probably not really necessary for this
case (that would be really nice, but probably too much of a nightmare to
code...)
- - You can see previous iterations of workflows.
- - Workflows can be "owned" by either a user or a role
- users could create a workflow and transfer ownership to a role
- role owned workflows are delete-able (or maybe require some X% of
users in that role to confirmation deletion?)
- This would sort of work like a group owned VCS repository, where
multiple users can create workflows of interest to a group of people
(role) and have them, by default, be available.



Anyone have input on this proposal?

Cheers,
Eric

- -- 
Eric Rasche
Programmer II
Center for Phage Technology
Texas A&M University
College Station, TX 77843
404-692-2048
e...@tamu.edu
rasche.e...@yandex.ru
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJScWXTAAoJEMqDXdrsMcpVFH4QAMO0V4tSJGzM3PGGIwdonY0v
nyuHTCJJ5qliWuapUJUK9E0UQBblnaJYooLjh2l1vgo95JaXdHEAC9NVecQLH8J2
ey5cu1xODf6JmBGlicRGxQWlLZmo9U3R4T/OhHco40I239aTpuSDPcwcBJUm1cAE
czz+/2pgdCpkJqYQJ9EE956TAYdyPBYFvSrIO7ebWkSvC5He40qca0x06tgq14uR
Dta+8QH8brVYX1G+TAgdh3hGlLUwIlPMiZsWOEMcwSUtyXP3I7OKx/XIOy3ZYVTu
Py/XlvNlZ87yiaiwDSxjxR9qaDeGQ3J8CRCG9mqWu8O5Y9QKkyrC410I+hZH5lE+
my28z7l5rK0i05pR6o8euASJ7D1eFHEDYmBmN6j3qRcOgzzdup2oQpm6TA7ue1sd
NAS4rnW7QGmfNj+9xNSQhDWSj7CeIUeHXbDe55suBX3JofCNg8Qq6uYzPd7KGj7n
jvWp8tGnpwbgTmvDPlR03gJiXMZld+gGwKrlCyn1kdBkfI1r5l0UfoaIUE+8HDm3
wXviQ8GU7seYU5bJFK798owVNawjY1H3JenzIYt1XvF1z53Uc39x58Q9lxXbVfSe
KgiqCfF0YuvVtOczGTAQ57qCFwlpKg2ra+X+HtAZyMDNMsb3ANgfogTqsB+7HYvF
czq2QJHq9mBC5N9JLhRo
=conD
-END PGP SIGNATURE-
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Workflows with

[Peter Cock]

On Wed, Oct 30, 2013 at 6:02 PM, John Chilton  wrote:
> I am pretty confident this worked at some point, this is potentially a
> regression of some kind (or I am crazy person). I will try to find
> time to take a closer look.

OK, likewise - I'll try to put together a reproducible test case,
and try it on next-stable and the current development tip too.

Thanks,

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Workflows with

[John Chilton]

I am pretty confident this worked at some point, this is potentially a
regression of some kind (or I am crazy person). I will try to find
time to take a closer look.

I have created a Trello card for this:

https://trello.com/c/9knsbvmx

Sorry about that!

-John

On Wed, Oct 30, 2013 at 12:35 PM, Peter Cock  wrote:
> Hi John,
>
> I've just tried a workflow using my MIRA4 de novo wrapper:
> http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler
>
> To keep it simple, in this example I just had two input FASTQ
> files, the forward and reverse files for paired end MiSeq data.
>
> This seemed to work fine for re-running the workflow :)
>
> However, on loading the workflow to edit it, only the first
> input file was drawn linked to the MIRA4 tool. On making
> some minor workflow changes (renaming some output),
> and trying to run it, the second input file seemed not to
> be used.
>
> Is this something you are aware of?
>
> The good news is that I could "re-draw" the missing
> connection from the second FASTQ file in the workflow
> editor, and that appears to be working...
>
> Peter
>
> --
>
> This was using the current stable release,
>
> $ hg branch
> default
>
> $ hg heads
> changeset:   10421:a477486bf18e
> branch:  stable
> tag: tip
> user:Nate Coraor 
> date:Thu Sep 26 11:02:58 2013 -0400
> summary: Bugfix for tool-to-destination mapping, tool ids are
> lowercased but the mapping id was not lowercased.
>
> changeset:   10411:c42567f43aa7
> user:greg
> date:Mon Aug 19 13:19:56 2013 -0400
> summary: Filter invalid objects when generating the list of
> repository_dependencies objects that are associated with a tool shed
> repository installed into Galaxy.
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Workflows with

[Peter Cock]

Hi John,

I've just tried a workflow using my MIRA4 de novo wrapper:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler

To keep it simple, in this example I just had two input FASTQ
files, the forward and reverse files for paired end MiSeq data.

This seemed to work fine for re-running the workflow :)

However, on loading the workflow to edit it, only the first
input file was drawn linked to the MIRA4 tool. On making
some minor workflow changes (renaming some output),
and trying to run it, the second input file seemed not to
be used.

Is this something you are aware of?

The good news is that I could "re-draw" the missing
connection from the second FASTQ file in the workflow
editor, and that appears to be working...

Peter

--

This was using the current stable release,

$ hg branch
default

$ hg heads
changeset:   10421:a477486bf18e
branch:  stable
tag: tip
user:Nate Coraor 
date:Thu Sep 26 11:02:58 2013 -0400
summary: Bugfix for tool-to-destination mapping, tool ids are
lowercased but the mapping id was not lowercased.

changeset:   10411:c42567f43aa7
user:greg
date:Mon Aug 19 13:19:56 2013 -0400
summary: Filter invalid objects when generating the list of
repository_dependencies objects that are associated with a tool shed
repository installed into Galaxy.
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] workflows stop prematurely

[mark.rose]

Ok, I added
track_jobs_in_database = True

to universe_wsgi.ini and now workflows appear to go to completion.

Another problem I was having was that steps in a workflow would stay gray even 
though examination of the log revealed the steps had completed and produced 
correct output (as verified by directly opening the dat files).  This problem 
also seems to have been fixed by the above.

Mark

From: galaxy-dev-boun...@lists.bx.psu.edu 
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of mark.r...@syngenta.com
Sent: Monday, January 28, 2013 1:18 PM
To: galaxy-dev@lists.bx.psu.edu
Subject: [galaxy-dev] workflows stop prematurely

Hi All

On my local install of galaxy I'm having  problems executing workflows.  I will 
upload data into a new history and then select to run a saved workflow.  This 
initiates fine but, after the completion of a variable number of steps, the 
workflow just stops after the successful completion of a step.  No errors 
appear in the web interface and all I can see in the paster.log is something 
like


90 HTTP/1.1" 200 - "http://127.0.0.1:8081/history"; "Mozilla/5.0 (X11; Ubuntu; 
Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0"
127.0.0.1 - - [28/Jan/2013:12:29:07 -0400] "GET /api/histories/82b264d8c3d11790 
HTTP/1.1" 200 - "http://127.0.0.1:8081/history"; "Mozilla/5.0 (X11; Ubuntu; 
Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0"
galaxy.jobs.runners.local DEBUG 2013-01-28 12:29:08,621 execution finished: 
python 
/home/rosema1/Projects/galaxy-dist/tools/fastq/fastq_trimmer_by_quality.py 
'/home/rosema1/Projects/galaxy-dist/database/files/000/dataset_590.dat' 
'/home/rosema1/Projects/galaxy-dist/database/files/000/dataset_592.dat' -f 
'sanger' -s '5'  -t '1' -e '53' -a 'mean' -x '0' -c '>=' -q '20.0'
galaxy.jobs DEBUG 2013-01-28 12:29:08,838 Tool did not define exit code or 
stdio handling; checking stderr for success
galaxy.tools DEBUG 2013-01-28 12:29:09,135 Error opening galaxy.json file: 
[Errno 2] No such file or directory: 
'/home/rosema1/Projects/galaxy-dist/database/job_working_directory/000/418/galaxy.json'
galaxy.jobs DEBUG 2013-01-28 12:29:09,439 job 418 ended
galaxy.jobs DEBUG 2013-01-28 12:29:10,041 (420) Working directory for job is: 
/home/rosema1/Projects/galaxy-dist/database/job_working_directory/000/420
galaxy.jobs.handler DEBUG 2013-01-28 12:29:10,042 dispatching job 420 to local 
runner
galaxy.jobs.handler INFO 2013-01-28 12:29:10,200 (420) Job dispatched
galaxy.jobs.runners.local DEBUG 2013-01-28 12:29:10,385 Local runner: starting 
job 420
galaxy.jobs.runners.local DEBUG 2013-01-28 12:29:10,759 executing: python 
/home/rosema1/Projects/galaxy-dist/tools/fastq/fastq_filter.py 
/home/rosema1/Projects/galaxy-dist/database/files/000/dataset_592.dat 
/home/rosema1/Projects/galaxy-dist/database/job_working_directory/000/420/tmph0Ll2V
 /home/rosema1/Projects/galaxy-dist/database/files/000/dataset_594.dat 
/home/rosema1/Projects/galaxy-dist/database/job_working_directory/000/420/dataset_594_files
 'sanger'
127.0.0.1 - - [28/Jan/2013:12:29:12 -0400] "GET /api/histories/82b264d8c3d11790 
HTTP/1.1" 200 - "http://127.0.0.1:8081/history"; "Mozilla/5.0 (X11; Ubuntu; 
Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0"
127.0.0.1 - - [28/Jan/2013:12:29:12 -0400] "GET /api/users/f2db41e1fa331b3e 
HTTP/1.1" 200 - "http://127.0.0.1:8081/history"; "Mozilla/5.0 (X11; Ubuntu; 
Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0"
127.0.0.1 - - [28/Jan/2013:12:29:12 -0400] "GET 
/api/histories/82b264d8c3d11790/contents?ids=3da459a26c8f00b4%2C736af8f0a76a2e71
 HTTP/1.1" 200 - "http://127.0.0.1:8081/history"; "Mozilla/5.0 (X11; Ubuntu; 
Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0"

127.0.0.1 - - [28/Jan/2013:12:29:16 -0400] "GET /api/histories/82b264d8c3d11790 
HTTP/1.1" 200 - "http://127.0.0.1:8081/history"; "Mozilla/5.0 (X11; Ubuntu; 
Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0"

Though I seem to see this even when a step completes and the workflow proceeds.

Moreover, if I then begin executing the next steps in the workflow, they work, 
but create new entries in the history.

So, some questions:

Why is this happening and what can I do to get workflows to run to completion?

Also,  is it possible to reinitiate the workflow from the premature stopping 
point without having to launch each step individually and point each at new 
inputs rather than the ones dictated in the workflow?

Thanks

Mark






This message may contain confidential information. If you are not the 
designated recipient, please notify the sender immediately, and delete the 
original and any copies. Any use of the message by you is prohibited.



This 

[galaxy-dev] workflows stop prematurely

[mark.rose]

Hi All

On my local install of galaxy I'm having  problems executing workflows.  I will 
upload data into a new history and then select to run a saved workflow.  This 
initiates fine but, after the completion of a variable number of steps, the 
workflow just stops after the successful completion of a step.  No errors 
appear in the web interface and all I can see in the paster.log is something 
like


90 HTTP/1.1" 200 - "http://127.0.0.1:8081/history"; "Mozilla/5.0 (X11; Ubuntu; 
Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0"
127.0.0.1 - - [28/Jan/2013:12:29:07 -0400] "GET /api/histories/82b264d8c3d11790 
HTTP/1.1" 200 - "http://127.0.0.1:8081/history"; "Mozilla/5.0 (X11; Ubuntu; 
Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0"
galaxy.jobs.runners.local DEBUG 2013-01-28 12:29:08,621 execution finished: 
python 
/home/rosema1/Projects/galaxy-dist/tools/fastq/fastq_trimmer_by_quality.py 
'/home/rosema1/Projects/galaxy-dist/database/files/000/dataset_590.dat' 
'/home/rosema1/Projects/galaxy-dist/database/files/000/dataset_592.dat' -f 
'sanger' -s '5'  -t '1' -e '53' -a 'mean' -x '0' -c '>=' -q '20.0'
galaxy.jobs DEBUG 2013-01-28 12:29:08,838 Tool did not define exit code or 
stdio handling; checking stderr for success
galaxy.tools DEBUG 2013-01-28 12:29:09,135 Error opening galaxy.json file: 
[Errno 2] No such file or directory: 
'/home/rosema1/Projects/galaxy-dist/database/job_working_directory/000/418/galaxy.json'
galaxy.jobs DEBUG 2013-01-28 12:29:09,439 job 418 ended
galaxy.jobs DEBUG 2013-01-28 12:29:10,041 (420) Working directory for job is: 
/home/rosema1/Projects/galaxy-dist/database/job_working_directory/000/420
galaxy.jobs.handler DEBUG 2013-01-28 12:29:10,042 dispatching job 420 to local 
runner
galaxy.jobs.handler INFO 2013-01-28 12:29:10,200 (420) Job dispatched
galaxy.jobs.runners.local DEBUG 2013-01-28 12:29:10,385 Local runner: starting 
job 420
galaxy.jobs.runners.local DEBUG 2013-01-28 12:29:10,759 executing: python 
/home/rosema1/Projects/galaxy-dist/tools/fastq/fastq_filter.py 
/home/rosema1/Projects/galaxy-dist/database/files/000/dataset_592.dat 
/home/rosema1/Projects/galaxy-dist/database/job_working_directory/000/420/tmph0Ll2V
 /home/rosema1/Projects/galaxy-dist/database/files/000/dataset_594.dat 
/home/rosema1/Projects/galaxy-dist/database/job_working_directory/000/420/dataset_594_files
 'sanger'
127.0.0.1 - - [28/Jan/2013:12:29:12 -0400] "GET /api/histories/82b264d8c3d11790 
HTTP/1.1" 200 - "http://127.0.0.1:8081/history"; "Mozilla/5.0 (X11; Ubuntu; 
Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0"
127.0.0.1 - - [28/Jan/2013:12:29:12 -0400] "GET /api/users/f2db41e1fa331b3e 
HTTP/1.1" 200 - "http://127.0.0.1:8081/history"; "Mozilla/5.0 (X11; Ubuntu; 
Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0"
127.0.0.1 - - [28/Jan/2013:12:29:12 -0400] "GET 
/api/histories/82b264d8c3d11790/contents?ids=3da459a26c8f00b4%2C736af8f0a76a2e71
 HTTP/1.1" 200 - "http://127.0.0.1:8081/history"; "Mozilla/5.0 (X11; Ubuntu; 
Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0"
127.0.0.1 - - [28/Jan/2013:12:29:16 -0400] "GET /api/histories/82b264d8c3d11790 
HTTP/1.1" 200 - "http://127.0.0.1:8081/history"; "Mozilla/5.0 (X11; Ubuntu; 
Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0"

Though I seem to see this even when a step completes and the workflow proceeds.

Moreover, if I then begin executing the next steps in the workflow, they work, 
but create new entries in the history.

So, some questions:

Why is this happening and what can I do to get workflows to run to completion?

Also,  is it possible to reinitiate the workflow from the premature stopping 
point without having to launch each step individually and point each at new 
inputs rather than the ones dictated in the workflow?

Thanks

Mark





This message may contain confidential information. If you are not the 
designated recipient, please notify the sender immediately, and delete the 
original and any copies. Any use of the message by you is prohibited. 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Workflows

[Geert Vandeweyer]

Hi, 

I've put together some exempels for activating and using the api. You can find 
them here: 

http://geertvandeweyer.zymichost.com/index.php?page=read&id=29

Best regards, 

Geert vandeweyer

Op 16-jan.-2013 om 02:52 heeft  het volgende geschreven:

> Hi
>Sorry if these questions are obvious but I just don’t know how to find the 
> answers.
>  
> I’m trying to get one of the API examples to work in 
> http://wiki.galaxyproject.org/Learn/API/Examples .
>  
> I’ve got my API key but how do I get/find the workflow id (f2db41e1fa331b3e 
> in the examples). I’ve created workflow but don’t know how to access this key.
>  
> Also in the first example it states /api/workflows However I 
> don’t have the “api/workflows” directory structure in my galaxy-dist, do I 
> need to create this?
>  
> Thanks
> Neil
>  
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> 
>  http://lists.bx.psu.edu/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Workflows

[James Taylor]

Neil,

.../api/workflows is not a path in the galaxy directory, but a URL
that you should access over HTTP. For example, if you login to Galaxy
Test, and then access https://test.g2.bx.psu.edu/api/workflows you
will get a json dict of all of your workflows, including the workflow
id that you are looking for.


--
James Taylor, Assistant Professor, Biology/CS, Emory University


On Tue, Jan 15, 2013 at 8:52 PM,   wrote:
> Hi
>
>Sorry if these questions are obvious but I just don’t know how to find
> the answers.
>
>
>
> I’m trying to get one of the API examples to work in
> http://wiki.galaxyproject.org/Learn/API/Examples .
>
>
>
> I’ve got my API key but how do I get/find the workflow id (f2db41e1fa331b3e
> in the examples). I’ve created workflow but don’t know how to access this
> key.
>
>
>
> Also in the first example it states /api/workflows However I
> don’t have the “api/workflows” directory structure in my galaxy-dist, do I
> need to create this?
>
>
>
> Thanks
>
> Neil
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
>   http://lists.bx.psu.edu/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] Workflows

[Neil.Burdett]

Hi
   Sorry if these questions are obvious but I just don't know how to find the 
answers.

I'm trying to get one of the API examples to work in 
http://wiki.galaxyproject.org/Learn/API/Examples .


I've got my API key but how do I get/find the workflow id (f2db41e1fa331b3e in 
the examples). I've created workflow but don't know how to access this key.



Also in the first example it states /api/workflows However I don't 
have the "api/workflows" directory structure in my galaxy-dist, do I need to 
create this?



Thanks

Neil


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Workflows do not open anymore

[Jeremy Goecks]

Fixed in 4d22e26e595a

Thanks,
J.

On Aug 8, 2012, at 6:05 AM, Björn Grüning wrote:

> Hello,
> 
> after commit 29680fa5c35e we are not able to open workflows anymore.
> 
> The error message is:
> 
> TypeError:  is not JSON
> serializable
> 
> And the problematic changeset is here:
> 
> https://bitbucket.org/galaxy/galaxy-central/changeset/29680fa5c35e6c22a6b7208015237b95d4bce2cc#chg-lib/galaxy/tools/__init__.py
> 
> Hope that helps,
> Bjoern
> 
> 
> 
> -- 
> Björn Grüning
> Albert-Ludwigs-Universität Freiburg
> Institute of Pharmaceutical Sciences
> Pharmaceutical Bioinformatics
> Hermann-Herder-Strasse 9
> D-79104 Freiburg i. Br.
> 
> Tel.:  +49 761 203-4872
> Fax.:  +49 761 203-97769
> E-Mail: bjoern.gruen...@pharmazie.uni-freiburg.de
> Web: http://www.pharmaceutical-bioinformatics.org/
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> 
>  http://lists.bx.psu.edu/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] Workflows do not open anymore

[Björn Grüning]

Hello,

after commit 29680fa5c35e we are not able to open workflows anymore.

The error message is:

TypeError:  is not JSON
serializable

And the problematic changeset is here:

https://bitbucket.org/galaxy/galaxy-central/changeset/29680fa5c35e6c22a6b7208015237b95d4bce2cc#chg-lib/galaxy/tools/__init__.py

Hope that helps,
Bjoern



-- 
Björn Grüning
Albert-Ludwigs-Universität Freiburg
Institute of Pharmaceutical Sciences
Pharmaceutical Bioinformatics
Hermann-Herder-Strasse 9
D-79104 Freiburg i. Br.

Tel.:  +49 761 203-4872
Fax.:  +49 761 203-97769
E-Mail: bjoern.gruen...@pharmazie.uni-freiburg.de
Web: http://www.pharmaceutical-bioinformatics.org/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] workflows are broken in current galaxy-central version

[Edward Kirton]

yup, works.  thanks for the prompt reply

On Thu, Jul 21, 2011 at 10:15 AM, Kanwei Li  wrote:

> Fixed on trunk, thanks for reporting!
>
> K
>
> On Thu, Jul 21, 2011 at 12:41 PM, Edward Kirton  wrote:
>
>>
>> URL: https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8
>> Module paste.exceptions.errormiddleware:*143* in __call__
>> Module paste.debug.prints:*98* in __call__
>> Module paste.wsgilib:*539* in intercept_output
>> Module paste.recursive:*80* in __call__
>> Module paste.httpexceptions:*632* in __call__
>> Module galaxy.web.framework.base:*145* in __call__
>> >>   
>> body *=* method*(* trans*,* kwargs *)*
>> Module galaxy.web.framework:*84* in decorator
>> >>   *
>> return* func*(* self*,* trans*,* ***args*,* kwargs *)*
>> Module galaxy.web.controllers.workflow:*686* in editor
>> >>   *
>> return* trans*.*fill_template*(* "workflow/editor.mako"*,* stored*=*
>> stored*,* annotation*=*self*.*get_item_annotation_str*(* trans*.*
>> sa_session*,* trans*.*user*,* stored *)* *)*
>> Module galaxy.web.framework:*661* in fill_template
>> >>   *
>> return* self*.*fill_template_mako*(* filename*,* kwargs *)*
>> Module galaxy.web.framework:*672* in fill_template_mako
>> >>   *
>> return* template*.*render*(* data *)*
>> Module mako.template:*296* in render
>> >>   *
>> return* runtime*.*_render*(*self*,* self*.*callable_*,* args*,* data*)*
>> Module mako.runtime:*660* in _render
>> >>   ***
>> *_kwargs_for_callable*(*callable_*,* data*)**)*
>> Module mako.runtime:*692* in _render_context
>> >>   
>> _exec_template*(*inherit*,* lclcontext*,* args*=*args*,* kwargs*=*kwargs*
>> )*
>> Module mako.runtime:*718* in _exec_template
>> >>   
>> callable_*(*context*,* ***args*,* kwargs*)*
>> Module _base_panels_mako:*87* in render_body
>> >>   
>> __M_writer*(*unicode*(*self*.*overlay*(*visible*=*self*.*overlay_visible*
>> )**)**)*
>> *TypeError: render_overlay() got an unexpected keyword argument 'visible'
>> *
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>
>>  http://lists.bx.psu.edu/
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] workflows are broken in current galaxy-central version

[Kanwei Li]

Fixed on trunk, thanks for reporting!

K

On Thu, Jul 21, 2011 at 12:41 PM, Edward Kirton  wrote:

>
> URL: https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8
> Module paste.exceptions.errormiddleware:*143* in __call__
> Module paste.debug.prints:*98* in __call__
> Module paste.wsgilib:*539* in intercept_output
> Module paste.recursive:*80* in __call__
> Module paste.httpexceptions:*632* in __call__
> Module galaxy.web.framework.base:*145* in __call__
> >>   body
>  *=* method*(* trans*,* kwargs *)*
> Module galaxy.web.framework:*84* in decorator
> >>   *
> return* func*(* self*,* trans*,* ***args*,* kwargs *)*
> Module galaxy.web.controllers.workflow:*686* in editor
> >>   *
> return* trans*.*fill_template*(* "workflow/editor.mako"*,* stored*=*stored
> *,* annotation*=*self*.*get_item_annotation_str*(* trans*.*sa_session*,*
> trans*.*user*,* stored *)* *)*
> Module galaxy.web.framework:*661* in fill_template
> >>   *
> return* self*.*fill_template_mako*(* filename*,* kwargs *)*
> Module galaxy.web.framework:*672* in fill_template_mako
> >>   *
> return* template*.*render*(* data *)*
> Module mako.template:*296* in render
> >>   *
> return* runtime*.*_render*(*self*,* self*.*callable_*,* args*,* data*)*
> Module mako.runtime:*660* in _render
> >>   
> _kwargs_for_callable*(*callable_*,* data*)**)*
> Module mako.runtime:*692* in _render_context
> >>   
> _exec_template*(*inherit*,* lclcontext*,* args*=*args*,* kwargs*=*kwargs*)
> *
> Module mako.runtime:*718* in _exec_template
> >>   
> callable_*(*context*,* ***args*,* kwargs*)*
> Module _base_panels_mako:*87* in render_body
> >>   
> __M_writer*(*unicode*(*self*.*overlay*(*visible*=*self*.*overlay_visible*)
> **)**)*
> *TypeError: render_overlay() got an unexpected keyword argument 'visible'*
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
>  http://lists.bx.psu.edu/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] workflows are broken in current galaxy-central version

[Edward Kirton]

URL: https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8
Module paste.exceptions.errormiddleware:*143* in __call__
Module paste.debug.prints:*98* in __call__
Module paste.wsgilib:*539* in intercept_output
Module paste.recursive:*80* in __call__
Module paste.httpexceptions:*632* in __call__
Module galaxy.web.framework.base:*145* in __call__
>>   body *
=* method*(* trans*,* kwargs *)*
Module galaxy.web.framework:*84* in decorator
>>   *
return* func*(* self*,* trans*,* ***args*,* kwargs *)*
Module galaxy.web.controllers.workflow:*686* in editor
>>   *
return* trans*.*fill_template*(* "workflow/editor.mako"*,* stored*=*stored*,
* annotation*=*self*.*get_item_annotation_str*(* trans*.*sa_session*,* trans
*.*user*,* stored *)* *)*
Module galaxy.web.framework:*661* in fill_template
>>   *
return* self*.*fill_template_mako*(* filename*,* kwargs *)*
Module galaxy.web.framework:*672* in fill_template_mako
>>   *
return* template*.*render*(* data *)*
Module mako.template:*296* in render
>>   *
return* runtime*.*_render*(*self*,* self*.*callable_*,* args*,* data*)*
Module mako.runtime:*660* in _render
>>   
_kwargs_for_callable*(*callable_*,* data*)**)*
Module mako.runtime:*692* in _render_context
>>   
_exec_template*(*inherit*,* lclcontext*,* args*=*args*,* kwargs*=*kwargs*)*
Module mako.runtime:*718* in _exec_template
>>   
callable_*(*context*,* ***args*,* kwargs*)*
Module _base_panels_mako:*87* in render_body
>>   
__M_writer*(*unicode*(*self*.*overlay*(*visible*=*self*.*overlay_visible*)**
)**)*
*TypeError: render_overlay() got an unexpected keyword argument 'visible'*
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] Workflows and unknown number of output datasets

[Louise-Amélie Schmitt]

Hi everyone

First let me thank all the Team Galaxy, the conference was really great.

Kanwei asked me to send the tool I told him about. It's one of these 
tools for which you can't know the exact number of output datasets 
before tool run. Here are the files. It's really simple actually, but it 
would be nice if I could integrate it in workflows without it having to 
be the final step. Would it even be possible?


And please ignore the idiotic comments in the files, I was really tired 
that day.


Cheers,
L-A
# -*- coding: UTF-8 -*-

import os, sys, string

mpxdata = sys.argv[1]
barcodes= sys.argv[2]
output1 = sys.argv[3]
output1id   = sys.argv[4]
newfilepath = sys.argv[5]

# Building the command line
cmd = "java -cp /g/steinmetz/projects/solexa/java/solexaJ/bin/:/g/steinmetz/projects/solexa/software/picard/trunk/dist/picard-1.18.jar:/g/steinmetz/projects/solexa/software/picard/trunk/dist/sam-1.18.jar"
cmd+= " deBarcoding F1="
cmd+= mpxdata
cmd+= " BL="
cmd+= barcodes
cmd+= " DR=\""
cmd+= newfilepath
cmd+= "\""

# Executing deBarcoding
status = os.system(cmd)

# In the unlikely event of a fire, please use the nearest emergency exit
if status != 0:
print "Demultiplexing failed."
sys.exit(status)

oldnames=[]

# Reconstructing the output file names as deBarcoding writes them
bc = open(barcodes, "r")
for l in bc.readlines():
l = l.split()
if l[0] != "":
oldnames.append(l[0])
for i in range(len(oldnames)):
oldnames[i] = oldnames[i] + ".txt"

newnames=[]

# Creating the required paths for multiple outputs
if os.path.isdir(newfilepath):
for f in oldnames:
if os.path.isfile(newfilepath+"/"+f):
name = os.path.splitext(f)[0]
s = "primary_"
s+= output1id
s+= "_"
s+= string.replace(name, "_", "-")
s+= "_visible_fastq"
newnames.append(newfilepath+"/"+s)

# Adding the appropriate prefixes to the old filenames
for i in range(len(oldnames)):
oldnames[i] = newfilepath+"/"+oldnames[i]

# Setting the first file as the mandatory output file defined in the xml
newnames[0] = output1

# Moving everything where it will be seen properly by Galaxy
for i in range(len(oldnames)):
os.rename(oldnames[i],newnames[i])

# Ta-da!

	Demultiplexes multiplexed data (who would have guessed?)
	debarcoding.py $mpxdata $barcodes $output1 $output1.id $__new_file_path__
	
		
		

	
	
		
	

	
**Program:** debarcoding.py (v1.0.0)

**Author:**  This is a wrapper for Wave's deBarcoding java tool

**Summary:** This tool demutiplexes data according to a list of barcodes containing a column withe the sample name and a second column with the barcode sequence.

**Usage:**   Here is an example of the java tool's usage::

 Two failed PE runs (multiplex 2 and multiplex8) yielded SE data that can be used for assembly together with the corrected PE ones. To this end, the deBarcoding script from wave was used:

 javasol deBarcoding

 where javasol is an alias to:

 java -cp /g/steinmetz/projects/solexa/java/solexaJ/bin/:/g/steinmetz/projects/solexa/software/picard/trunk/dist/picard-1.18.jar:/g/steinmetz/projects/solexa/software/picard/trunk/dist/sam-1.18.jar

 The command lines were (thanks to Wave for the clarifications):

 For the mplex num. 3, some pre-processing was necessary
 
 R  src/getBarcodeSequencing.R
 javasol deBarcoding F1=s_1_LESAFFRE_sequence.txt.gz BL=barcodeList.txt DR="seq_lane1"
 javasol deBarcoding F1=s_2_LESAFFRE_sequence.txt.gz BL=barcodeList.txt DR="seq_lane2"
 javasol deBarcoding F1=s_3_LESAFFRE_sequence.txt.gz BL=barcodeList.txt DR="seq_lane3"
 cat seq_lane1/111.txt seq_lane2/111.txt seq_lane3/111.txt >sequences/111.txt
 cat seq_lane1/112.txt seq_lane2/112.txt seq_lane3/112.txt >sequences/112.txt
 cat seq_lane1/1251.txt seq_lane2/1251.txt seq_lane3/1251.txt >sequences/1251.txt
 cat seq_lane1/1303.txt seq_lane2/1303.txt seq_lane3/1303.txt >sequences/1303.txt
 cat seq_lane1/93ep.txt seq_lane2/93ep.txt seq_lane3/93ep.txt >sequences/93ep.txt
 rename ".txt" "_s_sequence.txt" *.txt
 gzip *.txt




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/