Re: [galaxy-dev] AWS launch error
Enis, To clarify, our credentials are in the beta system which dosn't work. To me it seems like an aws authentication issue. David On Wed, Nov 8, 2017, 7:22 PM David Kovalic <kova...@analome.com> wrote: > Enis, > > Yes our credentials are in the beta system. > > Lmk if you need me to do anything to help troubleshoot. > > Right now we are recreating systems through aws. Are there docs for > running cloudman from cli? This would get us going in the short term. > > David > > On Wed, Nov 8, 2017, 6:33 PM Enis Afgan <enis.af...@irb.hr> wrote: > >> Hi David, >> Do you perhaps have credentials stored on the beta server? If not, and >> don't mind storing them, you can try that. I'm not having an issue at the >> moment but I know Nuwan has done some changes recently that I did not catch >> up with yet. CC'ing him in case he has additional input. >> >> On Tue, Nov 7, 2017 at 12:53 PM, David Kovalic <kova...@analome.com> >> wrote: >> >>> Enis, >>> >>> I am having issues launching previously existing clusters today, here is >>> some more info: >>> >>>- I go to https://launch.usegalaxy.org/launch >>>- I fill in the required AWS credentials >>>- I click on "Choose a saved cluster" >>>- the page goes into a loop trying to retrieve the 'Saved Cluster' >>>information to populate the drop down list >>>- I can terminate this loop by clicking on the "x" >>>- I can enter the cluster name manually >>>- The cluster bucket entered does exist on my S3 >>>- After entering the cluster name manually I click "Create a cluster" >>>- The next page is shown "Please wait, your scalable cluster is >>>starting on the Amazon - Virginia cloud." >>>- I see no instance stared on the AWS console >>> >>> I have similar issues on the new page " >>> https://beta.launch.usegalaxy.org/;, on this page my previous clusters >>> are not shown. >>> >>> Nothing has changed on my end, to my knoledge… Any ideas? Need any more >>> information from my end? >>> >>> lmk and thanks, >>> >>> David >>> >>> >>> >>> David Kovalic Ph.D. President, Analome >>> phone: +314.496.8808 >>> web: www.analome.com >>> email: kova...@analome.com >>> <https://www.linkedin.com/in/davidkovalic> >>> >>> >>> >>> On Fri, Nov 3, 2017 11:00 AM, galaxy-dev-requ...@lists.galaxyproject.org >>> wrote: >>> >>>> Send galaxy-dev mailing list submissions to >>>> >>>> galaxy-dev@lists.galaxyproject.org >>>> >>>> >>>> To subscribe or unsubscribe via the World Wide Web, visit >>>> >>>> https://lists.galaxyproject.org/listinfo/galaxy-dev >>>> >>>> or, via email, send a message with subject or body 'help' to >>>> >>>> galaxy-dev-requ...@lists.galaxyproject.org >>>> >>>> >>>> You can reach the person managing the list at >>>> >>>> galaxy-dev-ow...@lists.galaxyproject.org >>>> >>>> >>>> When replying, please edit your Subject line so it is more specific >>>> >>>> than "Re: Contents of galaxy-dev digest..." >>>> >>>> HEY! This is important! If you reply to a thread in a digest, please >>>> >>>> 1. Change the subject of your response from "Galaxy-dev Digest Vol ..." >>>> to the original subject for the thread. >>>> >>>> 2. Strip out everything else in the digest that is not part of the >>>> thread you are responding to. >>>> >>>> >>>> Why? >>>> >>>> 1. This will keep the subject meaningful. People will have some idea >>>> from the subject line if they should read it or not. >>>> >>>> 2. Not doing this greatly increases the number of emails that match >>>> search queries, but that aren't actually informative.Today's Topics: >>>> >>>> >>>> 1. Re: AWS launch error (Enis Afgan) >>>> >>>> 2. Enable viewing of large HTML files (Peter Briggs) >>>> >>>> 3. Re: Enable viewing of large HTML files (Martin Čech) >>>> >>>> ___ >>>> >>>> galaxy-dev mailing list >>>>
Re: [galaxy-dev] AWS launch error
Enis, Yes our credentials are in the beta system. Lmk if you need me to do anything to help troubleshoot. Right now we are recreating systems through aws. Are there docs for running cloudman from cli? This would get us going in the short term. David On Wed, Nov 8, 2017, 6:33 PM Enis Afgan <enis.af...@irb.hr> wrote: > Hi David, > Do you perhaps have credentials stored on the beta server? If not, and > don't mind storing them, you can try that. I'm not having an issue at the > moment but I know Nuwan has done some changes recently that I did not catch > up with yet. CC'ing him in case he has additional input. > > On Tue, Nov 7, 2017 at 12:53 PM, David Kovalic <kova...@analome.com> > wrote: > >> Enis, >> >> I am having issues launching previously existing clusters today, here is >> some more info: >> >>- I go to https://launch.usegalaxy.org/launch >>- I fill in the required AWS credentials >>- I click on "Choose a saved cluster" >>- the page goes into a loop trying to retrieve the 'Saved Cluster' >>information to populate the drop down list >>- I can terminate this loop by clicking on the "x" >>- I can enter the cluster name manually >>- The cluster bucket entered does exist on my S3 >>- After entering the cluster name manually I click "Create a cluster" >>- The next page is shown "Please wait, your scalable cluster is >>starting on the Amazon - Virginia cloud." >>- I see no instance stared on the AWS console >> >> I have similar issues on the new page "https://beta.launch.usegalaxy.org/;, >> on this page my previous clusters are not shown. >> >> Nothing has changed on my end, to my knoledge… Any ideas? Need any more >> information from my end? >> >> lmk and thanks, >> >> David >> >> >> >> David Kovalic Ph.D. President, Analome >> phone: +314.496.8808 >> web: www.analome.com >> email: kova...@analome.com >> <https://www.linkedin.com/in/davidkovalic> >> >> >> >> On Fri, Nov 3, 2017 11:00 AM, galaxy-dev-requ...@lists.galaxyproject.org >> wrote: >> >>> Send galaxy-dev mailing list submissions to >>> >>> galaxy-dev@lists.galaxyproject.org >>> >>> >>> To subscribe or unsubscribe via the World Wide Web, visit >>> >>> https://lists.galaxyproject.org/listinfo/galaxy-dev >>> >>> or, via email, send a message with subject or body 'help' to >>> >>> galaxy-dev-requ...@lists.galaxyproject.org >>> >>> >>> You can reach the person managing the list at >>> >>> galaxy-dev-ow...@lists.galaxyproject.org >>> >>> >>> When replying, please edit your Subject line so it is more specific >>> >>> than "Re: Contents of galaxy-dev digest..." >>> >>> HEY! This is important! If you reply to a thread in a digest, please >>> >>> 1. Change the subject of your response from "Galaxy-dev Digest Vol ..." >>> to the original subject for the thread. >>> >>> 2. Strip out everything else in the digest that is not part of the >>> thread you are responding to. >>> >>> >>> Why? >>> >>> 1. This will keep the subject meaningful. People will have some idea >>> from the subject line if they should read it or not. >>> >>> 2. Not doing this greatly increases the number of emails that match >>> search queries, but that aren't actually informative.Today's Topics: >>> >>> >>> 1. Re: AWS launch error (Enis Afgan) >>> >>> 2. Enable viewing of large HTML files (Peter Briggs) >>> >>> 3. Re: Enable viewing of large HTML files (Martin Čech) >>> >>> ___ >>> >>> galaxy-dev mailing list >>> >>> galaxy-dev@lists.galaxyproject.org >>> >>> https://lists.galaxyproject.org/listinfo/galaxy-dev >>> >>> >>> To search Galaxy mailing lists use the unified search at: >>> >>> http://galaxyproject.org/search/mailinglists/ >>> >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ > > >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/ >> > -- David Kovalic Ph.D. President, Analome phone: +314.496.8808 web: www.analome.com email: kova...@analome.com <https://www.linkedin.com/in/davidkovalic> ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] Data manager for ncbi_blast_plus in the main tool shed?
Peter, Thanks. Dan, What is your opinion (what work needs to be done to make this production-worthy)? Maybe if my chops are up to it I can take a crack… David David Kovalic Ph.D.President,Analomephone:+314.496.8808web:www.analome.comemail: kova...@analome.com On Fri, Aug 11, 2017 4:58 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi David, Daniel Blankenberg wrote one (CC'd): https://github.com/peterjc/galaxy_blast/tree/master/data_managers/ncbi_blastdb http://testtoolshed.g2.bx.psu.edu/view/blankenberg/data_manager_example_blastdb_ncbi_update_blastdb I've never used it in production and don't have enough understandingof it and the Data Manager framework to understand what if anythingneeds working on. Locally we have mirrors of the NCBI blast databases (updated with cron),local databases for organisms of interest - these are all listed in the *.locfiles by hand. Separately, our users can and do make their own databases withinGalaxy using makeblastdb. Regards, Peter On Thu, Aug 10, 2017 at 10:32 PM, David Kovalic <kova...@analome.com> wrote: Hi, Does anyone know if a data manager exists in the tool shed for creating and integrating NCBI BLAST+ indices within galaxy? I can't seem to find it in the tools shed or searching the internet Surly such a tool must exist. Any info would be greatly appreciated. Thanks, David David Kovalic Ph.D.President,Analomephone:+314.496.8808web:www.analome.comemail: kova...@analome.com ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] Data manager for ncbi_blast_plus in the main tool shed?
Hi, Does anyone know if a data manager exists in the tool shed for creating and integrating NCBI BLAST+ indices within galaxy? I can't seem to find it in the tools shed or searching the internet Surly such a tool must exist. Any info would be greatly appreciated. Thanks, David David Kovalic Ph.D.President,Analomephone:+314.496.8808web:www.analome.comemail: kova...@analome.com___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] bwa_mem and "-t" option
Bjoern, Thanks so much, I'll check it out. If I can't figure this out I am sure I can ask around in Montpellier at GCC'17! I am looking forward to the meeting and being able to visit Montpellier for the first time ever The help is greatly appreciated. David On Thu, Jun 1, 2017 2:21 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: HI David, have a look at https://galaxyproject.org/admin/config/jobs/ and this config file: https://github.com/galaxyproject/galaxy/blob/dev/config/job_resource_params_conf.xml.sample and here: https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_advanced#L791 I couldn't find more information, hope this gets you started! Cheers, Bjoern Am 01.06.2017 um 20:48 schrieb David Kovalic: > Bjoern, > > Thanks for the response. > > I agree with the first approach (admin control of the per tool > environment). Do you know… are there docs or similar resources you could > point me to that describe how to achieve this? > > Thanks again, > > David > > > > On Sun, May 28, 2017 4:41 AM, Björn Grüning bjoern.gruen...@gmail.com > <mailto:bjoern.gruen...@gmail.com> wrote: > > __ > > Hi David, > > > this is possible via the GALAXY_SLOTS mechanism. An admin can give every > > tool it's own environment and can set GALAXY_SLOTS dynamically. > > A user can not do this, as this is highly dependent on the node this > > tool will run and normally a user does not know about this. > > > However, if you really want to offer this to a user, Galaxy also offers > > to enable this option per tool level and will add additional parameter > > fields to set GALAXY_SLOTS as a user. > > > Cheers, > > Bjoern > > > Am 26.05.2017 um 20:23 schrieb David Kovalic: > > > Hi, > > > > > > Is it possible, and if so how, to run the galaxy devteam > > > (version 546ada4a9f43) of bwa_mem with the "-t" option and a value >1? > > > > > > "-t INT number of threads [1]" > > > > > > > > > This would be quite useful, and used, for accelerating single bwa > mem jobs. > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > ___ > > > Please keep all replies on the list by using "reply all" > > > in your mail client. To manage your subscriptions to this > > > and other Galaxy lists, please use the interface at: > > > https://lists.galaxyproject.org/ > > > > > > To search Galaxy mailing lists use the unified search at: > > > http://galaxyproject.org/search/ > > > > >___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] bwa_mem and "-t" option
Bjoern, Thanks for the response. I agree with the first approach (admin control of the per tool environment). Do you know… are there docs or similar resources you could point me to that describe how to achieve this? Thanks again, David On Sun, May 28, 2017 4:41 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi David, this is possible via the GALAXY_SLOTS mechanism. An admin can give every tool it's own environment and can set GALAXY_SLOTS dynamically. A user can not do this, as this is highly dependent on the node this tool will run and normally a user does not know about this. However, if you really want to offer this to a user, Galaxy also offers to enable this option per tool level and will add additional parameter fields to set GALAXY_SLOTS as a user. Cheers, Bjoern Am 26.05.2017 um 20:23 schrieb David Kovalic: > Hi, > > Is it possible, and if so how, to run the galaxy devteam > (version 546ada4a9f43) of bwa_mem with the "-t" option and a value >1? > > "-t INT number of threads [1]" > > > This would be quite useful, and used, for accelerating single bwa mem jobs. > > Thanks, > > David > > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/ >___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] bwa_mem and "-t" option
Hi, Is it possible, and if so how, to run the galaxy devteam (version546ada4a9f43) of bwa_mem with the "-t" option and a value >1? "-t INT number of threads [1]" This would be quite useful, and used, for accelerating single bwa mem jobs. Thanks, David___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] Disk organization for Jetstream Galaxy instance
Enis. Thanks for pointing me at the doc, I had not seen that and it is exactly the info I need. Awsome! David On Tue, Mar 14, 2017 3:47 PM, Enis Afgan enis.af...@irb.hr wrote: Hi David,Have you seen this bit of documentation about setting up Galaxy to use an external volume on Jetstream: https://galaxyproject.org/cloud/jetstream/storage/That's the most straightforward way at the moment. Changing all the configs to use the volume would me more involved, particularly because of the database and how it gets installed... On Thu, Mar 9, 2017 at 4:13 PM, David Kovalic <kova...@analome.com> wrote: Hi, When we launch a Galaxy instance on Jetstream, the galaxy user's home dir is set up on the root volume, this has inherent limitations. Is there a good way to change the home dir location to an attached disk? Thanks, David ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Disk organization for Jetstream Galaxy instance
Hi, When we launch a Galaxy instance on Jetstream, the galaxy user's home dir is set up on the root volume, this has inherent limitations. Is there a good way to change the home dir location to an attached disk? Thanks, David___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Deploying custom genome in Galaxy for use with ANNOVAR (Peter Briggs)
Peter, Thanks! The blog post is very helpful. David > -- Forwarded message -- > From: Peter Briggs <peter.bri...@manchester.ac.uk> > To: galaxy-dev@lists.galaxyproject.org > Cc: > Date: Wed, 8 Jun 2016 08:25:03 +0100 > Subject: Re: [galaxy-dev] Deploying custom genome in Galaxy for use with > ANNOVAR > Hello David > > I don't think it will answer all your questions, however: I wrote up my > experiences with installing ANNOVAR in Galaxy last year which might be > useful as a starting point: > > > http://galacticengineer.blogspot.co.uk/2015/03/installing-annovar-in-galaxy.html > > As I recall all the databases we installed were standard ones from > ANNOVAR. However the above instructions might be useful in at least > getting started. > > Apologies if I've misunderstood your question and this doesn't help, > > Best wishes > > Peter > > On 07/06/16 17:48, David Kovalic wrote: > > All, > > > > From my read of the command line docs for ANNOVAR I believe that the > > program needs so ancillary files for its operation (e.g. GenePred file, > > transcripts fasta file, etc...). > > > > Does anyone have any experience deploying custom genomes in Galaxy for > > use with ANNOVAR? Specifically what files need to go where in order to > > enable the Galaxy ANNOVAR front end? > > > > Thanks, > > > > David > > > > > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > >https://lists.galaxyproject.org/ > > > > To search Galaxy mailing lists use the unified search at: > >http://galaxyproject.org/search/mailinglists/ > > > > -- > Peter Briggs peter.bri...@manchester.ac.uk > Bioinformatics Core Facility University of Manchester > B.1083 Michael Smith Bldg Tel: (0161) 2751482 > > > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Deploying custom genome in Galaxy for use with ANNOVAR
All, >From my read of the command line docs for ANNOVAR I believe that the program needs so ancillary files for its operation (e.g. GenePred file, transcripts fasta file, etc...). Does anyone have any experience deploying custom genomes in Galaxy for use with ANNOVAR? Specifically what files need to go where in order to enable the Galaxy ANNOVAR front end? Thanks, David ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Parallelism (job splitting) for ncbi_blast_plus running through CloudMan
Peter, Thanks for the great information. I see where to tune the "split_size" variable and also the READMEs :) I'll do so more sleuthing, let the job run and observe. So far it is ~3hr after job launch and still no load on the workers. I think from looking at the /mnt/galaxy/tmp/job_working_directory all of the sub-job directories are prepared. Maybe an issue with the job scheduler/dispatch. Thanks again. David On Tue, May 3, 2016 at 5:11 PM Peter Cock <p.j.a.c...@googlemail.com> wrote: > On Tue, May 3, 2016 at 9:23 PM, David Kovalic <kova...@analome.com> wrote: > > Peter, > > > > We made the modification to the config file, restarted galaxy and things > > seem to be working from the galaxy end. We see sub-job directories being > > created in /mnt/galaxy/tmp/job_working_directory. We think all of the > > required job chunks have been created (i.e. total sequences/1000 sub-job > > directories now with no more being created now) > > > > Now we have what may be a CloudMan question: our working cluster has a > head > > node and 4 workers. The head node is loaded up but the workers are idle. > I > > would have thought jobs should be pushing out to the workers but we don't > > see any load on these machines. > > > > Any advice? Thanks. > > Wait a bit longer? The downside of the job splitting is the extra > disk I/O overhead of splitting the files (here FASTA inputs) and > then merging the output (e.g. BLAST tabular, XML, etc). IIRC, > this happens on the head node only. > > I've not used CloudMan so I have no specific advice here, > other than ask did you confirm that jobs were getting sent to > the worker nodes before turning on use_tasked_jobs = True > in your config/galaxy.ini file? > > > David > > > > PS. what is the path of the file which contains the split_size="1000" > > configuration? > > This is currently defined by the tool wrapper author in the tool > wrapper XML file. Setting up something which will work well > on a broad range of input file sizes is a bit of an art - simply > always dividing the input into 8 chunks does not scale well. > With BLAST+ I found chunks of 1000 queries was a good > balance, while for other tools processing FASTA inputs I > used chunks of 2000 queries. > > I'll link to the latest files on GitHub, but you can browse this on > the Galaxy Tool Shed too - it also ought to show the README > text quite prominently: > > https://github.com/peterjc/galaxy_blast/tree/master/tools/ncbi_blast_plus > > In a simple tool, you would see the tag directly > in the wrapper XML file, usually near the top by convention. > e.g. > > > https://github.com/peterjc/pico_galaxy/blob/master/tools/protein_analysis/promoter2.xml > > https://github.com/peterjc/pico_galaxy/blob/master/tools/protein_analysis/tmhmm2.xml > > However, with the BLAST+ wrappers we use macros. So, > using BLASTX as an example, the wrapper is this XML file: > > > https://github.com/peterjc/galaxy_blast/blob/master/tools/ncbi_blast_plus/ncbi_blastx_wrapper.xml > > No sign of the tag directly, but it is pulled in from: > > > https://github.com/peterjc/galaxy_blast/blob/master/tools/ncbi_blast_plus/ncbi_macros.xml > > This happens via: > > ... > > ... > ncbi_macros.xml > > > ... > > This is a bit more complex, but means avoiding repeating the XML > snippet in almost all the BLAST+ wrapper files. > > Peter > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Parallelism (job splitting) for ncbi_blast_plus running through CloudMan
Peter,
We made the modification to the config file, restarted galaxy and things
seem to be working from the galaxy end. We see sub-job directories being
created in /mnt/galaxy/tmp/job_working_directory. We think all of the
required job chunks have been created (i.e. total sequences/1000 sub-job
directories now with no more being created now)
Now we have what may be a CloudMan question: our working cluster has a head
node and 4 workers. The head node is loaded up but the workers are idle. I
would have thought jobs should be pushing out to the workers but we don't
see any load on these machines.
Any advice? Thanks.
David
PS. what is the path of the file which contains the split_size="1000"
configuration?
On Tue, May 3, 2016 at 2:19 PM David Kovalic <kova...@analome.com> wrote:
> Peter,
>
> Thanks, I didn't see that, I was reading the paper and searching online.
>
> Appreciate the help, we'll give it a go!
>
> David
>
>
> On Tue, May 3, 2016 at 2:16 PM Peter Cock <p.j.a.c...@googlemail.com>
> wrote:
>
>> Hi David,
>>
>> The NCBI BLAST+ wrappers have a tag setup,
>> which becomes active if you have use_tasked_jobs = True in
>> your config/galaxy.ini file (aka universe_wsgi.ini).
>>
>> Specifically, the wrappers use this:
>>
>>
>> > split_size="1000" merge_outputs="output1" />
>>
>> This is hard coded to break up the query FASTA file into batches
>> of 1000 sequences (e.g. a transcriptome of 20k genes becomes
>> 20 jobs), which has worked nicely on our cluster.
>>
>> Separately, each job uses -num_threads "\${GALAXY_SLOTS:-8}"
>> in the command line string, i.e. uses the $GALAXY_SLOTS
>> environment variable (set via the Galaxy job configuration), or
>> if not set, defaults to using 8 threads.
>>
>> I've essentially rephrased the README file here - did you see
>> that, or does it need more information added?
>>
>> Thanks,
>>
>> Peter
>>
>>
>> On Tue, May 3, 2016 at 6:58 PM, David Kovalic <kova...@analome.com>
>> wrote:
>> > Hello,
>> >
>> > We would like to split fasta query files and run multiple concurrent
>> jobs to
>> > minimize our processing wall clock time for large jobs.
>> >
>> > After chatting with folks at GCC 2015 I understand this is possible, my
>> > problem is I cant find instructions on hos to configure
>> > CloudMan/ncbi_blast_plus to do this. For those of you who know me it
>> > probably goes without saying that I can't figure it out myself ;)
>> >
>> > Peter/Enis/others, can you help us out with this question?
>> >
>> > Thanks,
>> >
>> > David
>> >
>> >
>> > ___
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client. To manage your subscriptions to this
>> > and other Galaxy lists, please use the interface at:
>> > https://lists.galaxyproject.org/
>> >
>> > To search Galaxy mailing lists use the unified search at:
>> > http://galaxyproject.org/search/mailinglists/
>>
>
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Parallelism (job splitting) for ncbi_blast_plus running through CloudMan
Peter,
Thanks, I didn't see that, I was reading the paper and searching online.
Appreciate the help, we'll give it a go!
David
On Tue, May 3, 2016 at 2:16 PM Peter Cock <p.j.a.c...@googlemail.com> wrote:
> Hi David,
>
> The NCBI BLAST+ wrappers have a tag setup,
> which becomes active if you have use_tasked_jobs = True in
> your config/galaxy.ini file (aka universe_wsgi.ini).
>
> Specifically, the wrappers use this:
>
>
> split_size="1000" merge_outputs="output1" />
>
> This is hard coded to break up the query FASTA file into batches
> of 1000 sequences (e.g. a transcriptome of 20k genes becomes
> 20 jobs), which has worked nicely on our cluster.
>
> Separately, each job uses -num_threads "\${GALAXY_SLOTS:-8}"
> in the command line string, i.e. uses the $GALAXY_SLOTS
> environment variable (set via the Galaxy job configuration), or
> if not set, defaults to using 8 threads.
>
> I've essentially rephrased the README file here - did you see
> that, or does it need more information added?
>
> Thanks,
>
> Peter
>
>
> On Tue, May 3, 2016 at 6:58 PM, David Kovalic <kova...@analome.com> wrote:
> > Hello,
> >
> > We would like to split fasta query files and run multiple concurrent
> jobs to
> > minimize our processing wall clock time for large jobs.
> >
> > After chatting with folks at GCC 2015 I understand this is possible, my
> > problem is I cant find instructions on hos to configure
> > CloudMan/ncbi_blast_plus to do this. For those of you who know me it
> > probably goes without saying that I can't figure it out myself ;)
> >
> > Peter/Enis/others, can you help us out with this question?
> >
> > Thanks,
> >
> > David
> >
> >
> > ___
> > Please keep all replies on the list by using "reply all"
> > in your mail client. To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> > https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> > http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Parallelism (job splitting) for ncbi_blast_plus running through CloudMan
Hello, We would like to split fasta query files and run multiple concurrent jobs to minimize our processing wall clock time for large jobs. After chatting with folks at GCC 2015 I understand this is possible, my problem is I cant find instructions on hos to configure CloudMan/ncbi_blast_plus to do this. For those of you who know me it probably goes without saying that I can't figure it out myself ;) Peter/Enis/others, can you help us out with this question? Thanks, David ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Cloudman - using different worker instance types
Alexander, Thanks, awesome! Sounds like it is time for an update at my end;) David On Sun, Sep 13, 2015 at 10:31 AM Alexander Vowinkel < vowinkel.alexan...@gmail.com> wrote: > Hi David, > > that is already possible with the current CloudMan release. > You can choose "Custom Instance Type" and put whatever > you like. > > Best, > Alexander > > 2015-09-12 11:35 GMT-05:00 David Kovalic <kova...@analome.com>: > >> Hi, >> >> I want to be able to use different AWS worker instance types than are >> currently provided in the cloudman "Add worker nodes" menu. >> >> Say for example I am interested in using AWS 'r3.*' instance types as my >> workers, how can I add these to be available in the cloudman interface? >> >> Thanks, >> >> David >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Cloudman - using different worker instance types
Hi, I want to be able to use different AWS worker instance types than are currently provided in the cloudman "Add worker nodes" menu. Say for example I am interested in using AWS 'r3.*' instance types as my workers, how can I add these to be available in the cloudman interface? Thanks, David ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] SSL/LDAP configuration for CloudMan/Galaxy
Hi, We are interested in running CloudMan/Galaxy with SSL and LDAP. After searching around on the internet it seems like this is achievable and probably not too difficult, but there is no current complete “cookbook recipe” for doing so, so I thought it best to ask questions before I go ahead and break stuff :) As I understand: · ngnix needs to have the LDAP module added, as the standard CM ngnix build doesn’t include this · ngnix needs to have a custom ngnix.conf file which specifies the use of SSL and LDAP · Galaxy need to have a custom configuration universe_wsgi.ini for LDAP use By searching online I can’t clearly figure out: · How to recompile (and persist across CM cluster termination/restart) a new version of ngnix · The best way to maintain and specify a custom ngnix.conf. Is it possible to do this by placing the custom ngnix.conf in the cluster S3 bucket and adding a configuration line specifying its URL (e.g. nginx_conf_contents: https://s3.amazonaws.com/[cm bucket ID]/ngnix.conf) in persistent_data.yaml file in the CM S3 bucket? · Where do I make the modifications such that the changes to universe_wsgi.ini persist across CM cluster termination/restart? It would be great to get some experienced insight on how best to complete this configuration, and have it persist. Any guidance would be greatly appreciated. Thanks, David Kovalic ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Trouble maintaining Galaxy config on cloud instance
Hi, I am trying to change the ftp dir (and some other modifications) vs. the standard Galaxy Cloudman config. When I modify the appropriate line in /mnt/galaxy/galaxy-app/universe_wsgi.ini on my running cloud instance: my changes disappear when I terminate/restart the cluster. When I modify universe_wsgi.ini.cloud and upload to the S3 bucket the file gets overwritten by the default when I restart the cluster... I can't find anywhere on the net which file I need to modify in order to have config changes persist. Any help would be greatly appreciated. Thanks, David ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
