On Wed, Aug 7, 2013 at 3:33 PM, Ganote, Carrie L <cgan...@iu.edu> wrote:
> Hi John,
>
> That was it. I feel silly. I still have a lot of tooth-cutting to do on
> python!
>
> I saw the parallel tags in the Blast tool and was very intrigued, but
> couldn't find reference to it in the read-the-docs or on the Galaxy wiki.
> Perhaps there is some documentation of this that I missed?
No I don't think there is documentation unless you count the code base
the mailing list archive. I think setting use_tasked_jobs to True in
universe_wsgi.ini might be all you need to do to start splitting such
blast inputs. I think the parallelism tag in the tool file describes
how to split the inputs.
# This enables splitting of jobs into tasks, if specified by the
particular tool config.
# This is a new feature and not recommended for production servers yet.
#use_tasked_jobs = False
I don't use this functionality (at least not in this fashion) so I
don't have a lot of advice. Otherwise, if you have a AMQP thing
working you should probably just stick with that sounds like a
perfectly good way to go.
-John
>
> In our case, the python splitting program is doing this:
> * Take the blast query
> * Split the sequences up
> * For each sequence, submit the query and the command to a queue on a
> RabbitMQ server (Consumers are set up to listen for queries and then run the
> jobs).
> * Write each result to a temp file
> * When all of the sequence jobs are finished, concat the files back in the
> correct order and write to the output file Galaxy expects
>
> I made a wrapper for this splitter and it works fine on its own. Now I'm
> trying to add this functionality (run on AMQP) as a user-available option on
> the Blast tool. So for my dynamic runner, I need to know whether to send the
> job to DRMAA or to this AMQP python script. Hopefully that makes more sense...
>
> Thanks so much for the help!
>
> -Carrie
> ________________________________________
> From: jmchil...@gmail.com [jmchil...@gmail.com] on behalf of John Chilton
> [chil...@msi.umn.edu]
> Sent: Wednesday, August 07, 2013 3:52 PM
> To: Ganote, Carrie L
> Cc: galaxy-dev@lists.bx.psu.edu
> Subject: Re: [galaxy-dev] Tool params passed to python dynamic job runner
>
> I am not sure why the quotations marks are there in the first place,
> but I do see the problem. replace and strip return new strings, they
> don't modify the existing variable. This is demonstrated below:
>
> % python
>>>> x = ' "Moo Cow" '
>>>> x
> ' "Moo Cow" '
>>>> x.replace("\"", "").strip()
> 'Moo Cow'
>>>> x
> ' "Moo Cow" '
>>>> x=x.replace("\"", "").strip()
>>>> x
> 'Moo Cow'
>
> You will want to change the line:
>
> modulePath.replace("\"", "").strip()
>
> to
>
> modulePath = modulePath.replace("\"", "").strip()
>
> You should also take a look at the Galaxy task splitting framework. It
> already has the ability to split up blast jobs into multiple tasks. I
> am not sure the dynamic job runner is what you want for that use case.
>
> -John
>
>
> On Wed, Aug 7, 2013 at 12:26 PM, Ganote, Carrie L <cgan...@iu.edu> wrote:
>> Hi List,
>>
>> I've run into a frustration recently with passing values to the dynamic job
>> runner from a tool. Here's what's going on:
>>
>> in ncbi_blastn_wrapper.xml:
>> <param name="pathToModule" type="hidden" value="/abs/path/to/module" />
>> ...
>>
>> in dynamic_runner.py:
>> ...
>> def blastn_wrapper(job):
>> incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] )
>> modulePath = incoming["pathToModule"]
>> modulePath.replace("\"", "").strip()
>>
>> dummy = '/abs/path/to/module'
>> equals = (modulePath == dummy)
>> log.debug( '\n~%s~\n~%s~\nsame? %s' % (mypath, dummy, equals) )
>> ...
>> log:
>> ~"/abs/path/to/module"~
>> ~/abs/path/to/module~
>> same? False
>>
>> I have no idea why these "" are persisting, or if that is even the problem.
>> I can't find the modulePath when using os.path.exists(modulePath).
>> What I want to do is optionally (user checks a box) run another python
>> script that splits up the blast queries into a bunch of jobs and submits
>> each job separately - and I was just going to import this other script using
>> sys.append(modulePath) in some poor attempt to keep the system cleaner.
>>
>> If this is a problem with my understanding of python, please enlighten me -
>> if there are other tools that pass around paths or if there's another way to
>> do this in less hacky way, I'm open to advice.
>>
>> Thanks!
>>
>> Carrie Ganote
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client. To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>> http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>> http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
