Re: [galaxy-dev] cheetah chaet sheet
I usually check this page: http://www.cheetahtemplate.org/docs/users_guide_html/users_guide.html#SECTION00060 M. On Fri, Jun 19, 2015 at 4:34 PM Alexander Vowinkel vowinkel.alexan...@gmail.com wrote: Hi, someone knows a cheat sheet with all the commands supported by cheetah? Best, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] package_bowtie_2_2_4 Executable Permissions - can't run bowtie as non-galaxy user
Hi David, can you open an issue at https://github.com/galaxyproject/docker-build ? Thanks, Nicola Il 18.06.2015 21:55 David Trudgian ha scritto: Installed bowtie2 from the toolshed, which brings in package_bowtie_2_2_4, which downloads bowtie binaries from depot.galaxyproject.org http://depot.galaxyproject.org/package/linux/x86_64/bowtie2/bowtie2-2.2.4-linux-x86_64.tgz Jobs submitted on our setup, running as a 'real user' (not galaxy user) via pulsar + SLURM fail with a permissions error. This is due to the permissions on some of the bowtie executables in the .tar.gz from depot being owner/group executable only. Other users cannot run bowtie2: -rwxrwx---. 1 galaxy galaxy 18K Oct 22 2014 bowtie2 If I download the tar.gz from depot manually and extract then I get: -rwxrwx--- 1 dtrudgian biohpc_admin 18K Oct 22 2014 bowtie2 Could the .tar.gz be fixed, so that the executables will work in a run as real user configuration? Is there any sensible way Galaxy could enforce permissions to guard against this issue? Cheers, -- David Trudgian Ph.D. Computational Scientist, BioHPC UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 Please contact biohpc-help@utsouthwestern with general BioHPC inquries. - UT Southwestern Medical Center The future of medicine, today. Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo. E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l’app Vai su https://www.indoona.com/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] BAM datatype - bam bai
Hi, thanks! Let me clarify myself on the multiple generation question. When I do command line, I do this: Bam,Bai - Prog1 - Bam,Bai(v2) - Prog2 - Bam,Bai(v3) What I understand, what galaxy is doing (always overwrite [1]): Bam,Bai - Prog1 - Bam,Bai(v2) - Bam(v2) - Bam,Bai(v2) - Prog2 - Bam,Bai(v3) - Bam(v3) - Bam,Bai(v3) Best, Alexander [1] https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/datatypes/binary.py#L314 2015-06-19 1:26 GMT-05:00 Björn Grüning bjoern.gruen...@gmail.com: Hi Alexander, Am 19.06.2015 um 01:44 schrieb Alexander Vowinkel: Hi, when I 'produce' a bam file in my output, I can download the bam file and a corresponding bai file. I am now wondering where does the bai file come from? From Galaxy! :) Galaxy has a concept of datatypes and metadata. In the case of BAM files, Galaxy is producing the index file to every position sorted BAM file. When is it being generated, what code is used for that? As soon as the BAM file is created. https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/datatypes/binary.py#L312 When I do multiple actions with my bam file - is the index generated every time? (overhead) No why should it? The BAM files stays the same right? So if you are using the BAM multiple time the one index is used multiple times. Where is the bai saved on disk (if it is)? You can access it with this snipped: $your_bam_file.metadata.bam_index I found that in the IUC gatk, a new bai is always generated when a gatk tool is started. This creates quite an overhead. Isn't there a better solution? Do you have an idea? Without Galaxy you also create an index once and reuse it, this is was Galaxy is doing. This concept is also used for other datatypes and improves the usability dramatically. If this was helpful, please consider to ask this question on Biostar again. I think this might be useful for others as well. Cheers, Bjoern @ref https://github.com/galaxyproject/tools-iuc/issues/194 Best, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] cloudman error somewhere
Hi Alexander,
When you build the image, did you run the image cleanup step? RabbitMQ's
database needs to be cleaned up before building the image otherwise the
application won't start properly and the AMQP connection will never be
established.
You can also post the entire CloudMan's log somewhere and I can take a look
there for other clues.
On Tue, Jun 16, 2015 at 4:07 PM, Alexander Vowinkel
vowinkel.alexan...@gmail.com wrote:
Maybe connected. this error (happening multiple times):
Exception happened during processing of request from ('127.0.0.1', 37201)
Traceback (most recent call last):
File
/home/ubuntu/.virtualenvs/CM/local/lib/python2.7/site-packages/paste/httpserver.py,
line 1068, in process_request_in_thread
self.finish_request(request, client_address)
File /usr/lib/python2.7/SocketServer.py, line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File /usr/lib/python2.7/SocketServer.py, line 651, in __init__
self.finish()
File /usr/lib/python2.7/SocketServer.py, line 710, in finish
self.wfile.close()
File /usr/lib/python2.7/socket.py, line 279, in close
self.flush()
File /usr/lib/python2.7/socket.py, line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
2015-06-16 13:59 GMT-05:00 Alexander Vowinkel
vowinkel.alexan...@gmail.com:
Hi,
I have the following problem:
First, the job scheduling didn't work.
I tried to upload a file, but the job was waiting.
Then I checked squeue, which timed out.
The slurm service was running.
Then I restarted the cluster.
Now I even have problems access the cloudman admin page.
Meaning, the page is loading endless.
The machine is not busy (CPU and mem @ htop)
In the paster.log it tries this every 4 seconds:
2015-06-16 18:35:56,547 DEBUG master:2579 Trying to setup AMQP
connection; conn = 'cm.util.comm.CMMasterComm object at 0x7f0aa26d72d0'
2015-06-16 18:35:56,547 DEBUG comm:30 Setting up a new AMQP
connection
2015-06-16 18:35:56,548 DEBUG comm:49 AMQP Connection
Failure: [Errno 111] Connection refused
Any idea what this might be?
Something seems to be really weird.
I am using the next version of CloudMan (dev).
I created an own instance with the Cloud Man playbook.
Best,
Alexander
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Custom Tool Dev Questions
The option in the GUI for selecting multiple datasets for a single data parameter - causes multiple jobs to be created as you noted. If you just place multiple=true on your data input parameter - that button will go away and all the datasets will be used for a single job. If your tool consumes pairs of datasets - you will want to expose an option where you are using a parameter like: input name=input_pairs type=data_collection collection_type=list:paired format=fastq / and then your users will have to use the data collection creator to pair off the datasets into a big list before using your tool - but the creator is actually really nice - I think it makes it very easy to build large lists of pairs. Hope this helps - happy to provide more details, -John On Fri, Jun 19, 2015 at 11:24 AM, Ryan G ngsbioinformat...@gmail.com wrote: Hi all - I have a tool which is essentially an entire NGS pipeline bundled into a perl script and associated modules. I'd like to offer this tool to users via Galaxy. I'm trying to pass a bunch of fastq files to the tool, but noticed that operating on multiple datasets creates a separate job for each dataset. I want to pass all the datasets to the tool so only one galaxy job is actually created. Is this possible, and if so, how do I go about doing this? Ryan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [spam?] Cloudman cluster not starting workers
Hi Nick, Sorry to hear you're having trouble. I just tried a couple of scenarios and they all worked as expected (e.g., with and without elastic IPs, different instance types). The main CloudMan log is located in /mnt/cm/paster.log, on both master and worker instances (if you didn't download the ssh key from cloudlaunch, you can ssh with ubuntu username and the same password as provided on the cloudlaunch form). The log is also available from the UI if you go to Admin page and then click 'Show CloudMan log' under 'System controls'. If you can share that, we can hopefully figure out what's going on. Best, Enis On Wed, Jun 17, 2015 at 11:47 AM, Nicholas Dickens nick.dick...@glasgow.ac.uk wrote: Hi All, First time post, as a quick intro, I’m have some reasonable experience with EC2 AWS for developing our own pipelines, I’m comfortable in python and *nix flavours and I am developing a completely custom galaxy in AWS for members of the WTCMP, Glasgow. In the meantime, I have thrown up a quick cloudman galaxy using the cloudstart and the cloudman 2.3 ami (ami-a7dbf6ce) in us-east-1. Auto–scaling didn’t seem to work so I’ve switched it off and added nodes manually, I tried various sizes including ‘same as master’ instances but they just don’t start – in the EC2 console I can see them and see them running. But they’re constantly pending in the /cloud interface and in the log they reboot 4 times and then terminate – apparently not responding 10:16:56 - Instance i-xx not responding after 4 reboots. Terminating instance. It’s out of the box, I editted the universe_wsgi.ini… file to disallow user registration and allow me to impersonate users but didn’t change anything else. The only other configuration I’ve done is associate an elastic IP with the master instance so I can have a more static url for a couple of test users (if I need to destroy it and start again, etc). I’m new to the system so don’t know which logs are best to check…and am I missing something obvious? It there a known bug when using elastic IPs? I’ve googled but with no joy. Thanks for your help and best wishes, Nick -- Nick Dickens DPhil BSc ARCS Bioinformatics Team Leader Wellcome Trust Centre for Molecular Parasitology B6-21 SGDB 120 University Place Glasgow G12 8TA Tel: +44 141 330 8282 http://fb.me/WTCMPbix @WTCMPbix http://www.gla.ac.uk/researchinstitutes/iii/staff/nickdickens/ http://www.gla.ac.uk/researchinstitutes/iii/staff/jeremymottram/comparativegenomicsofleishmania/ - Upozorenje - Automatskom detekcijom utvrdjeno je da tekst ove poruke podsjeca na tzv. phishing poruku. AKO SE U PORUCI TRAZI DA POSALJETE VASU IRB LOZINKU ILI DA UNESETE IRB PODATKE NA NAVEDENOM LINKU, RADI SE O NAPADU S CILJEM KRADJE I ZLOUPOTREBE PODATAKA. Centar za informatiku i racunarstvo, Institut Rudjer Boskovic - Upozorenje - ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Custom Tool Dev Questions
Ok, Great. I think I have part of this figured out. I have a conditional input as specified: conditional name=input_type param name=input_type_selector type=select label=Select input type help=Select between single and paired fastq data option value=singleSingle-End/option option value=pairedPaired-End/option option value=paired_collectionPaired Collection/option /param when value=single param name=fastq_input1 type=data format=fastq label=Select fastq dataset multiple=true help=Specify dataset with single end reads / /when when value=paired param name=fastq_input1 type=data format=fastq label=Select fastq dataset help=Specify dataset with paired-end reads / param name=fastq_input2 type=data format=fastq label=Select fastq dataset help=Specify dataset with paired-end reads / /when when value=paired_collection param name=fastq_collection type=data_collection collection_type=list:paired label=Select a paired collection help=See help section for an explanation of dataset collections/ /when /conditional In my command section, I want to test how this is passed to the tool, so I have: command #if str( $input_type.input_type_selector ) == single: echo Input Type: Single echo Input File: $input_type.fastq_input1 #elif str( $input_type.input_type_selector ) == paired: echo Input Type: Paired echo FastQ1: $input_type.fastq_input1\nFastQ2: $input_type.fastq_input2 #elif str( $input_type.input_type_selector ) == paired_collection: print Input Type: Collection\n #for $element in $input_type.$fastq_collection: echo FastQ1: $element.fastq_input1 #end for #end if /command The single and paired works as expected, however I can select multiple input for paired which I don't want to allow. A user should only be able to select a single pair of fastq files. To select multiple pairs, I have paired_collection. The second problem is in the command section, I'm trying to iterated through the pair of inputs, however my echo statement is causing a Cheetah error. I tried different incarnations of it as well. My latest (and what I thought correct) is shown above. It generates the error: Traceback (most recent call last): File /apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/jobs/runners/__init__.py, line 163, in prepare_job job_wrapper.prepare() File /apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/jobs/__init__.py, line 858, in prepare self.command_line, self.extra_filenames = tool_evaluator.build() File /apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/tools/evaluation.py, line 411, in build self.__build_command_line( ) File /apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/tools/evaluation.py, line 427, in __build_command_line command_line = fill_template( command, context=param_dict ) File /apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/util/template.py, line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File /apps/sys/galaxy/galaxy-v15.05-staging/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs2.egg/Cheetah/Template.py, line 1004, in __str__ return getattr(self, mainMethName)() File cheetah_DynamicallyCompiledCheetahTemplate_1434732603_97_56713.py, line 107, in respond AttributeError: 'dict' object has no attribute 'VFFSL' How do I iterate through the fastq inputs when they are passed as a list:paired? On Fri, Jun 19, 2015 at 11:31 AM, John Chilton jmchil...@gmail.com wrote: The option in the GUI for selecting multiple datasets for a single data parameter - causes multiple jobs to be created as you noted. If you just place multiple=true on your data input parameter - that button will go away and all the datasets will be used for a single job. If your tool consumes pairs of datasets - you will want to expose an option where you are using a parameter like: input name=input_pairs type=data_collection collection_type=list:paired format=fastq / and then your users will have to use the data collection creator to pair off the datasets into a big list before using your tool - but the creator is actually really nice - I think it makes it very easy to build large lists of pairs. Hope this helps - happy to provide more details, -John On Fri, Jun 19, 2015 at 11:24 AM, Ryan G ngsbioinformat...@gmail.com wrote: Hi all - I have a tool which is essentially an entire NGS pipeline bundled into a perl script and associated modules. I'd like to offer this tool to users via Galaxy. I'm trying to pass a bunch of fastq files to the tool, but noticed that operating on multiple datasets creates a separate job for each dataset. I want to pass all the datasets to the tool so only one galaxy job is actually created. Is this possible, and if so, how do I go about doing this? Ryan
