Re: [galaxy-dev] import Workflow from public Galaxy to local instance
For the first of the CloudMap workflows that worked. But the second failes with this error message: Failed to open URL: https://main.g2.bx.psu.edu/workflow/export_to_file?id=f444ce954bdc4953 Exception: HTTP Error 500: Internal Server Error I don't see any information in the log files. regards, Andreas On 22.05.2013 13:57, Dannon Baker wrote: On Wed, May 22, 2013 at 3:12 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de mailto:andreas.kuntz...@mdc-berlin.de wrote: But how do I get the workflow_id? For example for the workflow https://main.g2.bx.psu.edu/u/__gm2123/w/cloudmap-unmapped-__mutant-workflow-no-candidates https://main.g2.bx.psu.edu/u/gm2123/w/cloudmap-unmapped-mutant-workflow-no-candidates what would the workflow_id be? (I tried cloudmap-unmapped-mutant-__workflow-no-candidates but this did not work.) If you click on that link above, you can see the id in the url for the 'Import' link. Again, definitely not optimal -- public workflows should probably all be exposed via the API for use/import. -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] import Workflow from public Galaxy to local instance
Hi, On 21.05.2013 18:06, Dannon Baker wrote: This interaction is pretty clunky right now (I've made a trello card for improving it: https://trello.com/c/eu8XiJLK. Two options right now are: 1) Import (on main) each workflow you want to copy to your local instance, and generate the regular download link for import. (Download or Export in the workflow menu, once you've imported it) I wanted to avoid that since I have to import more then one workflow. 2) Munge the URLs yourself -- all accessible workflows are available for download via https://main.g2.bx.psu.edu/workflow/export_to_file?id=workflow_id But how do I get the workflow_id? For example for the workflow https://main.g2.bx.psu.edu/u/gm2123/w/cloudmap-unmapped-mutant-workflow-no-candidates what would the workflow_id be? (I tried cloudmap-unmapped-mutant-workflow-no-candidates but this did not work.) regards, Andreas -Dannon On Tue, May 21, 2013 at 11:56 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de mailto:andreas.kuntz...@mdc-berlin.de wrote: Hi, I want to make workflows published on usegalaxy.org http://usegalaxy.org available for users on our local instance. How do I do this? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/__bimsb/BIMSB_groups/Dieterich http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich _ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/__search/mailinglists/ http://galaxyproject.org/search/mailinglists/ -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] how to set GridEngine parameters for tools
Hi, I recently updated our local galaxy instance first time after a few month. Afterwards I noticed that the way to setup job runners has changed from single lines to an extra XML-file. I now started to write a job_conf.xml but I don't understand where I put the parameters for GridEngine. I already read http://wiki.galaxyproject.org/Admin/Config/Jobs but seem to have missed this info. For example in the old style I have lines like velveth = drmaa://-l h_vmem=25G/ In the new style I would write ... destinations destination id=cluster runner=drmaa/ /destinations tools tool id=velveth destination=cluster/ /tools ... But where do I put the -l h_vmem? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] how to set GridEngine parameters for tools
Thanks for your answer. I will try this. But I must say I don't like this new style very much. I need to create an extra destination for every paramater set. This blows up the configuration a lot. regards, Andreas On 21.05.2013 17:22, Björn Grüning wrote: Hi Andreas, try something like the following: destination id=20cores_10G runner=drmaa param id=nativeSpecification-l h_vmem=10G -pe * 20/param /destination Hope that helps, Björn Hi, I recently updated our local galaxy instance first time after a few month. Afterwards I noticed that the way to setup job runners has changed from single lines to an extra XML-file. I now started to write a job_conf.xml but I don't understand where I put the parameters for GridEngine. I already read http://wiki.galaxyproject.org/Admin/Config/Jobs but seem to have missed this info. For example in the old style I have lines like velveth = drmaa://-l h_vmem=25G/ In the new style I would write ... destinations destination id=cluster runner=drmaa/ /destinations tools tool id=velveth destination=cluster/ /tools ... But where do I put the -l h_vmem? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] import Workflow from public Galaxy to local instance
Hi, I want to make workflows published on usegalaxy.org available for users on our local instance. How do I do this? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] error when looking at tool error
.deleted AS dataset_1_deleted, dataset_1.purged AS dataset_1_purged,
dataset_1.purgable AS dataset_1_purgable, dataset_1.object_store_id AS dataset_1_object_store_id,
dataset_1.external_filename AS dataset_1_external_filename, dataset_1._extra_files_path AS
dataset_1__extra_files_path, dataset_1.file_size AS dataset_1_file_size, dataset_1.total_size AS
dataset_1_total_size \nFROM history_dataset_association LEFT OUTER JOIN dataset AS dataset_1 ON
dataset_1.id = history_dataset_association.dataset_id \nWHERE history_dataset_association.id =
%(param_1)s' {'param_1': u'e96c6ce43a3d239a'}
Looks like the id for the dataset is wrong. How can this happen and how can I
fix this?
regards, Andreas
--
Andreas Kuntzagk
SystemAdministrator
Berlin Institute for Medical Systems Biology at the
Max-Delbrueck-Center for Molecular Medicine
Robert-Roessle-Str. 10, 13125 Berlin, Germany
http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Permission denied error for velveth
Ok, while velveth seems to work now, if I put the result into velvetg I get another error: Traceback (most recent call last): File /data/galaxy/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py, line 175, in queue_job job_wrapper.prepare() File /data/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py, line 184, in prepare self.command_line = self.tool.build_command_line( param_dict ) File /data/galaxy/galaxy-dist/lib/galaxy/tools/__init__.py, line 2424, in build_command_line command_line = fill_template( self.command, context=param_dict ) File /data/galaxy/galaxy-dist/lib/galaxy/util/template.py, line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File /data/galaxy/galaxy-dist/eggs/Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg/Cheetah/Template.py, line 1004, in __str__ return getattr(self, mainMethName)() File cheetah_DynamicallyCompiledCheetahTemplate_1354790596_63_67504.py, line 84, in respond NotFound: cannot find 'files_path' while searching for 'input.files_path' Andreas On 06.12.2012 10:52, Andreas Kuntzagk wrote: Sorry, I think I had not really restarted galaxy after making the change. After I changed some logging options in universe_wsgi.ini and restarted it's suddenly working. So I can confirm your fix works. Thank you for your help. Andreas On 05.12.2012 13:40, Andreas Kuntzagk wrote: Hi, Does not work for me. I still get the same error. If you need me to run more test, just email me. regards, Andreas On 04.12.2012 18:27, John Chilton wrote: I don't have a velvet or a submit-jobs-as-user environment setup so I cannot test this, but it seems like this should have a an easy fix. Can someone who is experiencing the problem try the attached patch or has something like this been tried and there is some other problem? -John (09:43:01 AM) jmchilton: Is the velvet problem related to something specific about how velvet runs or do no tools that depend on extra_files_path work with as-user jobs? (10:10:31 AM) natefoo: they have to either write to files_path or extra_files_path, let me find the email... (10:12:24 AM) natefoo: http://dev.list.galaxyproject.org/Folder-permissions-after-cluster-run-td4657156.html#a4657219 (10:12:26 AM) mrscribe: Title: Galaxy Development List Archive - Folder permissions after cluster run (at dev.list.galaxyproject.org) (11:07:31 AM) jmchilton: Is fixing the velvet problem as simple as changing extra_files_path to files_path in the wrappers then? On Tue, Dec 4, 2012 at 9:38 AM, Oleksandr Moskalenko o...@hpc.ufl.edu wrote: On Dec 4, 2012, at 9:30 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: I see that this issue came up before by Oleksandr Moskalenko in October but was unresolved then. Is there a workaround now? regards, Andreas The velvet wrapper is not going to be usable for real user jobs until someone rewrites it to work without direct manipulation of files in the database/ tree. No work has been done on this nor any is planned as far as I know. Regards, Alex ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Permission denied error for velveth
Hi, I enabled velvet on our local install. But when I try to use velveth, I get an Permission denied: '/data/galaxy/galaxy-dist/database/files/027/dataset_27903_files' This file does not exist. Only file there is /data/galaxy/galaxy-dist/database/files/027/dataset_27903_dat regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] multithreaded tools
Hi, the four processes I saw where all called lastz and ran in parallel and consumed 100% of a core each. My guess is that the lastz_wrapper.py is responsible for this. Looking at it I see a some code regarding queuing and in the very beginning this line: WORKERS = 4 and further one the class BaseQueue which starts threads. BTW. there seems to be no way to adjust this number other than editing the source file - bad. And this get's me wondering if there are other such surprises hidden in galaxy. regards, Andreas On 26.11.2012 15:55, Bob Harris wrote: Howdy, Andreas, The four processes started for a galaxy lastz job must involve post-processing the lastz output through some other shell tool. Lastz by itself doesn't support multiple threads or processes. Bob H On Nov 26, 2012, at 3:58 AM, Andreas Kuntzagk wrote: Hi, I'm wandering how galaxy supports tools that are multithreaded or multi-process. When working with lastz I noticed that it starts 4 parallel processes. Is that always so? Can this be adjusted? What other tools also are multi-process? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] multithreaded tools
Dear Peter, As the author of several tool wrappers, I've been asking for a Galaxy wide mechanism for Galaxy to tell the tool how many threads it can use, for example via an environment variable. The value could then be set with a general default, per runner default, or even per tool using the existing runner configuration under [galaxy:tool_runners] in universe_wsgi.ini This would be a possibility. Another would be to communicate the number of threads the other way. So the tool tells the runner how many threads. And the runner knows how to handle this. I can imagine universe_wsgi.ini having such lines: ncbi_blastp_wrapper = drmaa://-V -pe smp $GALAXY_THREADS and then $GALAXY_THREADS is changed for the value given by the wrapper. Thinking again this is probably not goint to work because the runner comes first and the wrapper after. My idea was that the wrapper could decide what recources to request. So I could use lower memory settings for small mapping jobs ... In your example, and others like the BWA and BLAST+ wrappers where the tool XML is hard coded to 8 threads, you would probably want to use a custom runner in universe_wsgi.ini setting the cluster submission to request that many slots/CPUs. A list of all these wrappers on the Wiki would be nice. regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] multithreaded tools
Hi Alex, I am not sure if you can call these surprises. Well at least it surprised me :-) Didn't want to sound to negative. Some tools (which I highly appreciate) of Peter have been parallelised to get the job done more quickly. I earlier mentioned the ncbi blast+ wrappers but there the tool by itself handles the multithreading. Other tools I am aware that use a python script/wrapper to chunk up the initial query and rejoin later are tools like signalp, TMHMM and such. Usually it also involves some parsing of output to data that galaxy can subsequently handle. In the latter examples its done using python scripts, but for some of our custom tools we did it in perl, some using bash parallel, or using R. I wouldn't have a solution to getting to know this without going through the initial wrappers... While I can read Python and bash fine it becomes more complicated with perl and R. Don't know if I could easily spot from the code what the number of threads is. So maybe somebody could setup a list of these tools? regards, Andreas. Alex -Oorspronkelijk bericht- Van: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] Namens Andreas Kuntzagk Verzonden: dinsdag 27 november 2012 9:58 Aan: Bob Harris CC: galaxy-dev@lists.bx.psu.edu Onderwerp: Re: [galaxy-dev] multithreaded tools Hi, the four processes I saw where all called lastz and ran in parallel and consumed 100% of a core each. My guess is that the lastz_wrapper.py is responsible for this. Looking at it I see a some code regarding queuing and in the very beginning this line: WORKERS = 4 and further one the class BaseQueue which starts threads. BTW. there seems to be no way to adjust this number other than editing the source file - bad. And this get's me wondering if there are other such surprises hidden in galaxy. regards, Andreas On 26.11.2012 15:55, Bob Harris wrote: Howdy, Andreas, The four processes started for a galaxy lastz job must involve post-processing the lastz output through some other shell tool. Lastz by itself doesn't support multiple threads or processes. Bob H On Nov 26, 2012, at 3:58 AM, Andreas Kuntzagk wrote: Hi, I'm wandering how galaxy supports tools that are multithreaded or multi-process. When working with lastz I noticed that it starts 4 parallel processes. Is that always so? Can this be adjusted? What other tools also are multi-process? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] multithreaded tools
Hi Peter, thanks for your replies. On 27.11.2012 11:44, Peter Cock wrote: On Tue, Nov 27, 2012 at 10:38 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: Dear Peter, As the author of several tool wrappers, I've been asking for a Galaxy wide mechanism for Galaxy to tell the tool how many threads it can use, for example via an environment variable. The value could then be set with a general default, per runner default, or even per tool using the existing runner configuration under [galaxy:tool_runners] in universe_wsgi.ini This would be a possibility. Another would be to communicate the number of threads the other way. So the tool tells the runner how many threads. And the runner knows how to handle this. I can imagine universe_wsgi.ini having such lines: ncbi_blastp_wrapper = drmaa://-V -pe smp $GALAXY_THREADS and then $GALAXY_THREADS is changed for the value given by the wrapper. Thinking again this is probably not goint to work because the runner comes first and the wrapper after. My idea was that the wrapper could decide what recources to request. So I could use lower memory settings for small mapping jobs ... There is some work on dynamic job allocation you might be interested in - have you seen this thread? http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-November/011759.html This looks very promising. What I did not get from these messages is if that's already in galaxy-dist and where to put the dynamic job runner. In your example, and others like the BWA and BLAST+ wrappers where the tool XML is hard coded to 8 threads, you would probably want to use a custom runner in universe_wsgi.ini setting the cluster submission to request that many slots/CPUs. A list of all these wrappers on the Wiki would be nice. With many tools on the Tool Shed, I'm not sure how easy that would be to co-ordinate. Doing it for the core tools would be more realistic. I see the problem here. Especially since more and more tools are going into Tool Sheds. I was just looking for some way to reduce my workload ;-) -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] How to delete a toolshed tool completely or change category
Hi, In the past I installed bwa_wrapper from the toolshed. Unfortunately I must have choosen a wrong category. Currently it is not part of any tool section. According to integrated_tool_panel.xml it's supposed to be part of this section: section id=ngs:_assembly name=NGS: Assembly version= while it contains another section with the same name and a different id: section id=ngs_assembly name=NGS: Assembly version= I tried to rectify this by deleting and reinstalling but unfortunately there is no way to really reinstall and it always ends up with the same problem again. Is there a way to really uninstall? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] How to delete a toolshed tool completely or change category
Please ignore this. You can move it to a different section. Andreas On 27.11.2012 13:19, Andreas Kuntzagk wrote: Hi, In the past I installed bwa_wrapper from the toolshed. Unfortunately I must have choosen a wrong category. Currently it is not part of any tool section. According to integrated_tool_panel.xml it's supposed to be part of this section: section id=ngs:_assembly name=NGS: Assembly version= while it contains another section with the same name and a different id: section id=ngs_assembly name=NGS: Assembly version= I tried to rectify this by deleting and reinstalling but unfortunately there is no way to really reinstall and it always ends up with the same problem again. Is there a way to really uninstall? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] lastz on local galaxy failing
Hi, What I remember is that while the lastz had died the python wrapper was still hanging around for these 2 days. Maybe it's related to this builtin scheduler of the lastz wrapper. Unfortunately I don't have time now to reproduce the error conditions. regards, Andreas On 26.11.2012 15:47, Bob Harris wrote: Howdy, Andreas, Lastz memory requirements are dependent on the size of the input sequences (mainly on the size of reference sequence) and, to a lesser extent, the genome's repeat content. I'm a little confused/concerned by how this failure was indicated. When run from a console, if lastz has a memory allocation failure, a message is written to stderr (e.g. call to realloc failed to allocate 1,000,000,000 bytes) and the program exits to the shell with the status EXIT_FAILURE (an ISO C definition which I presume corresponds to a standard shell error code). Usually, if lastz isn't going to have enough memory, an allocation failure will occur early in the run as all the long term data structures are being built. This would normally be within the first five minutes. A later failure would (probably) mean that the long term structures were close to the memory limit and then alignments for one of the query sequences required enough additional memory to push us over the limit. I assume galaxy's Job did not produce output message must be based on a lack of any output to stdout. What is strange is that fact that it took 2 days to get this message. Lack of output to stdout suggests that the failure occurred before any queries were processed (strictly speaking, before any alignments were output). This should have occurred in the first few minutes of the run. Would it be possible for you to point me at the input sequences for this run, so that I can try running this via the console, and see if there's something happening in lastz that I don't understand? Bob H On Nov 26, 2012, at 3:35 AM, Andreas Kuntzagk wrote: Hi, I just noticed that this was another case of a tool that needs more then the 1GB memory that is default on our cluster. After adjusting the job_runner settings everything seems fine. regards, Andreas On 15.11.2012 09:08, Andreas Kuntzagk wrote: Hi, when I start a lastz job it submits a job to GridEngine. But this job seems to hang. The process running is python /data/galaxy/galaxy-dist/tools/sr_mapping/lastz_wrapper.py --ref_source= ... This wrapper seems to not start a lastz but lastz is in the path. After 2 days then Galaxy shows Job did not produce output Here is the paster.log from the start of the job. galaxy.jobs DEBUG 2012-11-15 09:03:07,539 (254) Working directory for job is: /data/galaxy/galaxy-dist/database/job_working_directory/000/254 galaxy.jobs.handler DEBUG 2012-11-15 09:03:07,540 dispatching job 254 to drmaa runner galaxy.jobs.handler INFO 2012-11-15 09:03:07,688 (254) Job dispatched galaxy.tools DEBUG 2012-11-15 09:03:07,980 Building dependency shell command for dependency 'lastz' galaxy.tools WARNING 2012-11-15 09:03:07,981 Failed to resolve dependency on 'lastz', ignoring galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:08,669 (254) submitting file /data/galaxy/galaxy-dist/database/pbs/galaxy_254.sh galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:08,670 (254) command is: python /data/galaxy/galaxy-dist/tools/sr_mapping/lastz_wrapper.py --ref_source=cached --source_select=pre_set --out_format=sam --input2=/data/galaxy/galaxy-dist/database/files/027/dataset_27795.dat --input1=/data/galaxy/galaxy-dist/tool-data/shared/ucsc/hg18/seq/hg18.2bit --ref_sequences=None --pre_set_options=yasra98 --identity_min=0 --identity_max=100 --coverage=0 --output=/data/galaxy/galaxy-dist/database/job_working_directory/000/254/galaxy_dataset_27800.dat --unmask=yes --lastzSeqsFileDir=/data/galaxy/galaxy-dist/tool-data; cd /data/galaxy/galaxy-dist; /data/galaxy/galaxy-dist/set_metadata.sh ./database/files /data/galaxy/galaxy-dist/database/job_working_directory/000/254 . /data/galaxy/galaxy-dist/universe_wsgi.ini /data/galaxy/galaxy-dist/database/tmp/tmpL4MMJV /data/galaxy/galaxy-dist/database/job_working_directory/000/254/galaxy.json /data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_in_HistoryDatasetAssociation_288_9ynzEN,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_kwds_HistoryDatasetAssociation_288_AEIseO,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_out_HistoryDatasetAssociation_288_RhsufT,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_results_HistoryDatasetAssociation_288_86DYsR,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/galaxy_dataset_27800.dat,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_override_HistoryDatasetAssociation_288_HMRNEG galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:08,676 run as user ['kuntzagk', '600'] galaxy.jobs DEBUG 2012-11-15 09
Re: [galaxy-dev] lastz on local galaxy failing
Hi, I just noticed that this was another case of a tool that needs more then the 1GB memory that is default on our cluster. After adjusting the job_runner settings everything seems fine. regards, Andreas On 15.11.2012 09:08, Andreas Kuntzagk wrote: Hi, when I start a lastz job it submits a job to GridEngine. But this job seems to hang. The process running is python /data/galaxy/galaxy-dist/tools/sr_mapping/lastz_wrapper.py --ref_source= ... This wrapper seems to not start a lastz but lastz is in the path. After 2 days then Galaxy shows Job did not produce output Here is the paster.log from the start of the job. galaxy.jobs DEBUG 2012-11-15 09:03:07,539 (254) Working directory for job is: /data/galaxy/galaxy-dist/database/job_working_directory/000/254 galaxy.jobs.handler DEBUG 2012-11-15 09:03:07,540 dispatching job 254 to drmaa runner galaxy.jobs.handler INFO 2012-11-15 09:03:07,688 (254) Job dispatched galaxy.tools DEBUG 2012-11-15 09:03:07,980 Building dependency shell command for dependency 'lastz' galaxy.tools WARNING 2012-11-15 09:03:07,981 Failed to resolve dependency on 'lastz', ignoring galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:08,669 (254) submitting file /data/galaxy/galaxy-dist/database/pbs/galaxy_254.sh galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:08,670 (254) command is: python /data/galaxy/galaxy-dist/tools/sr_mapping/lastz_wrapper.py --ref_source=cached --source_select=pre_set --out_format=sam --input2=/data/galaxy/galaxy-dist/database/files/027/dataset_27795.dat --input1=/data/galaxy/galaxy-dist/tool-data/shared/ucsc/hg18/seq/hg18.2bit --ref_sequences=None --pre_set_options=yasra98 --identity_min=0 --identity_max=100 --coverage=0 --output=/data/galaxy/galaxy-dist/database/job_working_directory/000/254/galaxy_dataset_27800.dat --unmask=yes --lastzSeqsFileDir=/data/galaxy/galaxy-dist/tool-data; cd /data/galaxy/galaxy-dist; /data/galaxy/galaxy-dist/set_metadata.sh ./database/files /data/galaxy/galaxy-dist/database/job_working_directory/000/254 . /data/galaxy/galaxy-dist/universe_wsgi.ini /data/galaxy/galaxy-dist/database/tmp/tmpL4MMJV /data/galaxy/galaxy-dist/database/job_working_directory/000/254/galaxy.json /data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_in_HistoryDatasetAssociation_288_9ynzEN,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_kwds_HistoryDatasetAssociation_288_AEIseO,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_out_HistoryDatasetAssociation_288_RhsufT,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_results_HistoryDatasetAssociation_288_86DYsR,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/galaxy_dataset_27800.dat,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_override_HistoryDatasetAssociation_288_HMRNEG galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:08,676 run as user ['kuntzagk', '600'] galaxy.jobs DEBUG 2012-11-15 09:03:08,676 (254) Changing ownership of working directory with: /usr/bin/sudo -E scripts/external_chown_script.py /data/galaxy/galaxy-dist/database/job_working_directory/000/254 kuntzagk 600 galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:09,148 (254) Job script for external submission is: /data/galaxy/galaxy-dist/database/pbs/254.jt_json 141.80.188.178 - - [15/Nov/2012:09:03:10 +0200] POST /galaxy/root/history_item_updates HTTP/1.1 200 - http://bbc.mdc-berlin.de/galaxy/history; Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0 galaxy.jobs.runners.drmaa INFO 2012-11-15 09:03:11,553 (254) queued as 1282605 galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:11,767 (254/1282605) state change: job is running -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] multithreaded tools
Hi, I'm wandering how galaxy supports tools that are multithreaded or multi-process. When working with lastz I noticed that it starts 4 parallel processes. Is that always so? Can this be adjusted? What other tools also are multi-process? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] lastz on local galaxy failing
Hi, when I start a lastz job it submits a job to GridEngine. But this job seems to hang. The process running is python /data/galaxy/galaxy-dist/tools/sr_mapping/lastz_wrapper.py --ref_source= ... This wrapper seems to not start a lastz but lastz is in the path. After 2 days then Galaxy shows Job did not produce output Here is the paster.log from the start of the job. galaxy.jobs DEBUG 2012-11-15 09:03:07,539 (254) Working directory for job is: /data/galaxy/galaxy-dist/database/job_working_directory/000/254 galaxy.jobs.handler DEBUG 2012-11-15 09:03:07,540 dispatching job 254 to drmaa runner galaxy.jobs.handler INFO 2012-11-15 09:03:07,688 (254) Job dispatched galaxy.tools DEBUG 2012-11-15 09:03:07,980 Building dependency shell command for dependency 'lastz' galaxy.tools WARNING 2012-11-15 09:03:07,981 Failed to resolve dependency on 'lastz', ignoring galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:08,669 (254) submitting file /data/galaxy/galaxy-dist/database/pbs/galaxy_254.sh galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:08,670 (254) command is: python /data/galaxy/galaxy-dist/tools/sr_mapping/lastz_wrapper.py --ref_source=cached --source_select=pre_set --out_format=sam --input2=/data/galaxy/galaxy-dist/database/files/027/dataset_27795.dat --input1=/data/galaxy/galaxy-dist/tool-data/shared/ucsc/hg18/seq/hg18.2bit --ref_sequences=None --pre_set_options=yasra98 --identity_min=0 --identity_max=100 --coverage=0 --output=/data/galaxy/galaxy-dist/database/job_working_directory/000/254/galaxy_dataset_27800.dat --unmask=yes --lastzSeqsFileDir=/data/galaxy/galaxy-dist/tool-data; cd /data/galaxy/galaxy-dist; /data/galaxy/galaxy-dist/set_metadata.sh ./database/files /data/galaxy/galaxy-dist/database/job_working_directory/000/254 . /data/galaxy/galaxy-dist/universe_wsgi.ini /data/galaxy/galaxy-dist/database/tmp/tmpL4MMJV /data/galaxy/galaxy-dist/database/job_working_directory/000/254/galaxy.json /data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_in_HistoryDatasetAssociation_288_9ynzEN,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_kwds_HistoryDatasetAssociation_288_AEIseO,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_out_HistoryDatasetAssociation_288_RhsufT,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_results_HistoryDatasetAssociation_288_86DYsR,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/galaxy_dataset_27800.dat,/data/galaxy/galaxy-dist/database/job_working_directory/000/254/metadata_override_HistoryDatasetAssociation_288_HMRNEG galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:08,676 run as user ['kuntzagk', '600'] galaxy.jobs DEBUG 2012-11-15 09:03:08,676 (254) Changing ownership of working directory with: /usr/bin/sudo -E scripts/external_chown_script.py /data/galaxy/galaxy-dist/database/job_working_directory/000/254 kuntzagk 600 galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:09,148 (254) Job script for external submission is: /data/galaxy/galaxy-dist/database/pbs/254.jt_json 141.80.188.178 - - [15/Nov/2012:09:03:10 +0200] POST /galaxy/root/history_item_updates HTTP/1.1 200 - http://bbc.mdc-berlin.de/galaxy/history; Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0 galaxy.jobs.runners.drmaa INFO 2012-11-15 09:03:11,553 (254) queued as 1282605 galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:11,767 (254/1282605) state change: job is running -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] tool_id for tool_shed tools
Hi, I upgraded to the latest version and again some tools moved to tool_shed. I now need to set job runners for these tools. Is there an easy way to get the tool_ids other then hovering the mouse over the tool name and write down the id? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] hg18.fa on rsync? [was: which .loc file for SAM to BAM?]
Ok, I've found the solution. sam_fa_indices.loc contained relative paths. When I changed to absolute paths it worked. regards, Murple On 02.11.2012 09:26, Andreas Kuntzagk wrote: Oh, I forgot to mention it: hg18.fa.fai is already in tool-data/shared/ucsc/hg18/sam_index/ as a link to tool-data/shared/ucsc/hg18/seq Replacing the link with the actual file did not help (not suprising). regards, Andreas On 01.11.2012 17:56, Dave Bouvier wrote: Andreas, I recommend moving hg18.fa.fai into tool-data/shared/ucsc/hg18/sam_index/, and then samtools should work. Also, if you're using picard tools, you'll want hg18.fa.fai and hg18.dict in tool-data/shared/ucsc/hg18/srma_index/, as well as links to hg18.fa in both directories. --Dave B. On 11/1/12 11:59:24.000, Andreas Kuntzagk wrote: Dave, In the meantime I found that out by myself howto generate the FASTA and also to rum samtools faidx on it. The info about all_fasta.loc was missing. But it's still not working. Let me summarize what I did so far: - tool-data/shared/ucsc/hg18/seq/ contains these files: hg18.2bit hg18.fahg18.fa.fai where hg18.2bit was downloaded from the rsync server and the other two generated from it. - tool-data/shared/ucsc/builds.txt contains this line: hg18Human Mar. 2006 (NCBI36/hg18) (hg18) - tool-data/all_fasta.loc contains this line: hg18hg18Human (Homo sapiens): hg18 tool-data/shared/ucsc/hg18/seq/hg18.fa - tool-data/sam_fa_indices.loc contains this line: indexhg18tool-data/shared/ucsc/hg18/sam_index/hg18.fa - tool-data/srma_index.loc contains this line: hg18hg18hg18tool-data/shared/ucsc/hg18/srma_index/hg18.fa So any ideas where to look further? regards, Andreas On 01.11.2012 15:36, Dave Bouvier wrote: Andreas, When setting up the rsync server, we decided that .fa files would be excluded from the listing, since the 2bit format contains the same data but takes up to 75% less space. I would recommend downloading the relevant .2bit file and converting it back to FASTA with twoBitToFa, then updating your all_fasta.loc file to point to the resulting .fa file. --Dave B. On 11/1/12 06:27:49.000, Andreas Kuntzagk wrote: Hi, It's still not working. I just noticed that the sam_index dir only contains links to some files in ../seq which is mostly empty except some 2bit files. I could not find any documentation how to obtain these data files. regards, Andreas On 31.10.2012 17:50, Carlos Borroto wrote: On Wed, Oct 31, 2012 at 11:30 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: Hi, I'm still setting up a local galaxy. Currently I'm testing the setup of NGS tools. If I try SAM to BAM for a BAM file that has hg18 set as build I get a message that Sequences are not currently available for the specified build. I guess that I have either to manipulate one of the .loc files (but which?) or have to download additional data from rsync server. (I already have the tool-data/shared/hg18 completely) The .loc file you want to modify is 'tool-data/sam_fa_indices.loc'. You can find information about this subject in the wiki[1]. Although the table there is not complete, so you could always find the right xml under 'tools' and poke inside to find a line like this one: validator type=dataset_metadata_in_file filename=sam_fa_indices.loc metadata_name=dbkey metadata_column=1 message=Sequences are not currently available for the specified build. line_startswith=index / [1]http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup And I agree, dealing with .loc files is quite cumbersome. Hope it helps, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] hg18.fa on rsync? [was: which .loc file for SAM to BAM?]
Oh, I forgot to mention it: hg18.fa.fai is already in tool-data/shared/ucsc/hg18/sam_index/ as a link to tool-data/shared/ucsc/hg18/seq Replacing the link with the actual file did not help (not suprising). regards, Andreas On 01.11.2012 17:56, Dave Bouvier wrote: Andreas, I recommend moving hg18.fa.fai into tool-data/shared/ucsc/hg18/sam_index/, and then samtools should work. Also, if you're using picard tools, you'll want hg18.fa.fai and hg18.dict in tool-data/shared/ucsc/hg18/srma_index/, as well as links to hg18.fa in both directories. --Dave B. On 11/1/12 11:59:24.000, Andreas Kuntzagk wrote: Dave, In the meantime I found that out by myself howto generate the FASTA and also to rum samtools faidx on it. The info about all_fasta.loc was missing. But it's still not working. Let me summarize what I did so far: - tool-data/shared/ucsc/hg18/seq/ contains these files: hg18.2bit hg18.fahg18.fa.fai where hg18.2bit was downloaded from the rsync server and the other two generated from it. - tool-data/shared/ucsc/builds.txt contains this line: hg18Human Mar. 2006 (NCBI36/hg18) (hg18) - tool-data/all_fasta.loc contains this line: hg18hg18Human (Homo sapiens): hg18 tool-data/shared/ucsc/hg18/seq/hg18.fa - tool-data/sam_fa_indices.loc contains this line: indexhg18tool-data/shared/ucsc/hg18/sam_index/hg18.fa - tool-data/srma_index.loc contains this line: hg18hg18hg18tool-data/shared/ucsc/hg18/srma_index/hg18.fa So any ideas where to look further? regards, Andreas On 01.11.2012 15:36, Dave Bouvier wrote: Andreas, When setting up the rsync server, we decided that .fa files would be excluded from the listing, since the 2bit format contains the same data but takes up to 75% less space. I would recommend downloading the relevant .2bit file and converting it back to FASTA with twoBitToFa, then updating your all_fasta.loc file to point to the resulting .fa file. --Dave B. On 11/1/12 06:27:49.000, Andreas Kuntzagk wrote: Hi, It's still not working. I just noticed that the sam_index dir only contains links to some files in ../seq which is mostly empty except some 2bit files. I could not find any documentation how to obtain these data files. regards, Andreas On 31.10.2012 17:50, Carlos Borroto wrote: On Wed, Oct 31, 2012 at 11:30 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: Hi, I'm still setting up a local galaxy. Currently I'm testing the setup of NGS tools. If I try SAM to BAM for a BAM file that has hg18 set as build I get a message that Sequences are not currently available for the specified build. I guess that I have either to manipulate one of the .loc files (but which?) or have to download additional data from rsync server. (I already have the tool-data/shared/hg18 completely) The .loc file you want to modify is 'tool-data/sam_fa_indices.loc'. You can find information about this subject in the wiki[1]. Although the table there is not complete, so you could always find the right xml under 'tools' and poke inside to find a line like this one: validator type=dataset_metadata_in_file filename=sam_fa_indices.loc metadata_name=dbkey metadata_column=1 message=Sequences are not currently available for the specified build. line_startswith=index / [1]http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup And I agree, dealing with .loc files is quite cumbersome. Hope it helps, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] which .loc file for SAM to BAM?
Hi, thank you for the pointer. I was only looking at this wiki page: http://wiki.g2.bx.psu.edu/Admin/Data%20Integration Maybe this should point to your page? regards, Andreas On 31.10.2012 17:50, Carlos Borroto wrote: On Wed, Oct 31, 2012 at 11:30 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: Hi, I'm still setting up a local galaxy. Currently I'm testing the setup of NGS tools. If I try SAM to BAM for a BAM file that has hg18 set as build I get a message that Sequences are not currently available for the specified build. I guess that I have either to manipulate one of the .loc files (but which?) or have to download additional data from rsync server. (I already have the tool-data/shared/hg18 completely) The .loc file you want to modify is 'tool-data/sam_fa_indices.loc'. You can find information about this subject in the wiki[1]. Although the table there is not complete, so you could always find the right xml under 'tools' and poke inside to find a line like this one: validator type=dataset_metadata_in_file filename=sam_fa_indices.loc metadata_name=dbkey metadata_column=1 message=Sequences are not currently available for the specified build. line_startswith=index / [1]http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup And I agree, dealing with .loc files is quite cumbersome. Hope it helps, Carlos -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] hg18.fa on rsync? [was: which .loc file for SAM to BAM?]
Hi, It's still not working. I just noticed that the sam_index dir only contains links to some files in ../seq which is mostly empty except some 2bit files. I could not find any documentation how to obtain these data files. regards, Andreas On 31.10.2012 17:50, Carlos Borroto wrote: On Wed, Oct 31, 2012 at 11:30 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: Hi, I'm still setting up a local galaxy. Currently I'm testing the setup of NGS tools. If I try SAM to BAM for a BAM file that has hg18 set as build I get a message that Sequences are not currently available for the specified build. I guess that I have either to manipulate one of the .loc files (but which?) or have to download additional data from rsync server. (I already have the tool-data/shared/hg18 completely) The .loc file you want to modify is 'tool-data/sam_fa_indices.loc'. You can find information about this subject in the wiki[1]. Although the table there is not complete, so you could always find the right xml under 'tools' and poke inside to find a line like this one: validator type=dataset_metadata_in_file filename=sam_fa_indices.loc metadata_name=dbkey metadata_column=1 message=Sequences are not currently available for the specified build. line_startswith=index / [1]http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup And I agree, dealing with .loc files is quite cumbersome. Hope it helps, Carlos -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] hg18.fa on rsync? [was: which .loc file for SAM to BAM?]
Dave, In the meantime I found that out by myself howto generate the FASTA and also to rum samtools faidx on it. The info about all_fasta.loc was missing. But it's still not working. Let me summarize what I did so far: - tool-data/shared/ucsc/hg18/seq/ contains these files: hg18.2bit hg18.fa hg18.fa.fai where hg18.2bit was downloaded from the rsync server and the other two generated from it. - tool-data/shared/ucsc/builds.txt contains this line: hg18Human Mar. 2006 (NCBI36/hg18) (hg18) - tool-data/all_fasta.loc contains this line: hg18hg18Human (Homo sapiens): hg18 tool-data/shared/ucsc/hg18/seq/hg18.fa - tool-data/sam_fa_indices.loc contains this line: index hg18tool-data/shared/ucsc/hg18/sam_index/hg18.fa - tool-data/srma_index.loc contains this line: hg18hg18hg18tool-data/shared/ucsc/hg18/srma_index/hg18.fa So any ideas where to look further? regards, Andreas On 01.11.2012 15:36, Dave Bouvier wrote: Andreas, When setting up the rsync server, we decided that .fa files would be excluded from the listing, since the 2bit format contains the same data but takes up to 75% less space. I would recommend downloading the relevant .2bit file and converting it back to FASTA with twoBitToFa, then updating your all_fasta.loc file to point to the resulting .fa file. --Dave B. On 11/1/12 06:27:49.000, Andreas Kuntzagk wrote: Hi, It's still not working. I just noticed that the sam_index dir only contains links to some files in ../seq which is mostly empty except some 2bit files. I could not find any documentation how to obtain these data files. regards, Andreas On 31.10.2012 17:50, Carlos Borroto wrote: On Wed, Oct 31, 2012 at 11:30 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: Hi, I'm still setting up a local galaxy. Currently I'm testing the setup of NGS tools. If I try SAM to BAM for a BAM file that has hg18 set as build I get a message that Sequences are not currently available for the specified build. I guess that I have either to manipulate one of the .loc files (but which?) or have to download additional data from rsync server. (I already have the tool-data/shared/hg18 completely) The .loc file you want to modify is 'tool-data/sam_fa_indices.loc'. You can find information about this subject in the wiki[1]. Although the table there is not complete, so you could always find the right xml under 'tools' and poke inside to find a line like this one: validator type=dataset_metadata_in_file filename=sam_fa_indices.loc metadata_name=dbkey metadata_column=1 message=Sequences are not currently available for the specified build. line_startswith=index / [1]http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup And I agree, dealing with .loc files is quite cumbersome. Hope it helps, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] which .loc file for SAM to BAM?
Sorry for my unclear sentence, I meant the page you pointed me too. regards, Andreas On 01.11.2012 14:50, Carlos Borroto wrote: Well is not my page, I'm not directly related to Galaxy. Still, I think Galaxy Project would be happy to receive updates to the wiki, maybe a complete table can be added to the page you are mentioning. On Thu, Nov 1, 2012 at 4:27 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: Hi, thank you for the pointer. I was only looking at this wiki page: http://wiki.g2.bx.psu.edu/Admin/Data%20Integration Maybe this should point to your page? regards, Andreas On 31.10.2012 17:50, Carlos Borroto wrote: On Wed, Oct 31, 2012 at 11:30 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: Hi, I'm still setting up a local galaxy. Currently I'm testing the setup of NGS tools. If I try SAM to BAM for a BAM file that has hg18 set as build I get a message that Sequences are not currently available for the specified build. I guess that I have either to manipulate one of the .loc files (but which?) or have to download additional data from rsync server. (I already have the tool-data/shared/hg18 completely) The .loc file you want to modify is 'tool-data/sam_fa_indices.loc'. You can find information about this subject in the wiki[1]. Although the table there is not complete, so you could always find the right xml under 'tools' and poke inside to find a line like this one: validator type=dataset_metadata_in_file filename=sam_fa_indices.loc metadata_name=dbkey metadata_column=1 message=Sequences are not currently available for the specified build. line_startswith=index / [1]http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup And I agree, dealing with .loc files is quite cumbersome. Hope it helps, Carlos -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] which .loc file for SAM to BAM?
Hi, I'm still setting up a local galaxy. Currently I'm testing the setup of NGS tools. If I try SAM to BAM for a BAM file that has hg18 set as build I get a message that Sequences are not currently available for the specified build. I guess that I have either to manipulate one of the .loc files (but which?) or have to download additional data from rsync server. (I already have the tool-data/shared/hg18 completely) regards, Andreas Btw. Do you have any plans to ease the pain on adding additional builds? Something simpler than having to add one line for each build*tool combo? These lines seem very redundant to me. -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Error when creating Visualization
Hi, on our local Galaxy instance I try to setup Visualization. I followed http://wiki.g2.bx.psu.edu/Learn/Visualization#Setup_for_Local_Instances But when I try to create a new visalization I get an error with this traceback: URL: http://bbc.mdc-berlin.de/galaxy/visualization/create File '/data/galaxy/galaxy-dist/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 93 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/visualization.py', line 590 in create type=visualization_type ) File '/data/galaxy/galaxy-dist/lib/galaxy/web/base/controller.py', line 343 in create_visualization revision = trans.model.VisualizationRevision( visualization=visualization, title=title, config=config, dbkey=dbkey ) File 'string', line 4 in __init__ File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/state.py', line 82 in initialize_instance return manager.events.original_init(*mixed[1:], **kwargs) File '/data/galaxy/galaxy-dist/lib/galaxy/model/__init__.py', line 2764 in __init__ self.visualization = visualization File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/attributes.py', line 150 in __set__ self.impl.set(instance_state(instance), instance_dict(instance), value, None) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/attributes.py', line 590 in set value = self.fire_replace_event(state, dict_, value, old, initiator) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/attributes.py', line 610 in fire_replace_event value = ext.set(state, value, previous, initiator or self) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/attributes.py', line 847 in set new_state, new_dict = instance_state(child), instance_dict(child) AttributeError: 'dict' object has no attribute '_sa_instance_state' Any idea what may be wrong? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error when creating Visualization
My fault. I did not set a title for it. (Anyway this should be caught I think.) regards, Andreas On 25.10.2012 13:18, Andreas Kuntzagk wrote: Hi, on our local Galaxy instance I try to setup Visualization. I followed http://wiki.g2.bx.psu.edu/Learn/Visualization#Setup_for_Local_Instances But when I try to create a new visalization I get an error with this traceback: URL: http://bbc.mdc-berlin.de/galaxy/visualization/create File '/data/galaxy/galaxy-dist/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 93 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/visualization.py', line 590 in create type=visualization_type ) File '/data/galaxy/galaxy-dist/lib/galaxy/web/base/controller.py', line 343 in create_visualization revision = trans.model.VisualizationRevision( visualization=visualization, title=title, config=config, dbkey=dbkey ) File 'string', line 4 in __init__ File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/state.py', line 82 in initialize_instance return manager.events.original_init(*mixed[1:], **kwargs) File '/data/galaxy/galaxy-dist/lib/galaxy/model/__init__.py', line 2764 in __init__ self.visualization = visualization File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/attributes.py', line 150 in __set__ self.impl.set(instance_state(instance), instance_dict(instance), value, None) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/attributes.py', line 590 in set value = self.fire_replace_event(state, dict_, value, old, initiator) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/attributes.py', line 610 in fire_replace_event value = ext.set(state, value, previous, initiator or self) File '/data/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/attributes.py', line 847 in set new_state, new_dict = instance_state(child), instance_dict(child) AttributeError: 'dict' object has no attribute '_sa_instance_state' Any idea what may be wrong? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] renaming fastq to fastqsanger
Hi, I have a similar problem with renaming fastq to fastqsanger. In my case after some time galaxy displays An error occurred setting the metadata for this dataset. You may be able to set it manually or retry auto-detection. The paster.log shows this: galaxy.jobs DEBUG 2012-10-17 14:16:44,623 (123) Changing ownership of working directory with: /usr/bin/sudo -E scripts/external_chown_script.py /data/galaxy/galaxy-dist/database/job_working_directory/000/123 galaxy 3400 galaxy.jobs DEBUG 2012-10-17 14:16:44,989 Tool did not define exit code or stdio handling; checking stderr for success 141.80.188.178 - - [17/Oct/2012:14:16:45 +0200] POST /galaxy/root/history_item_updates HTTP/1.1 200 - http://bbc.mdc-berlin.de/galaxy/history?dataset_id=f30a35c999095ed7filename=None; Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0 galaxy.datatypes.metadata DEBUG 2012-10-17 14:16:45,234 setting metadata externally failed for HistoryDatasetAssociation 110: External set_meta() not called galaxy.jobs DEBUG 2012-10-17 14:16:45,375 job 123 ended May that be related to an incorrect setup of drmaa job-runner? regards, Andreas On 03.10.2012 18:49, Pete Schmitt wrote: When attempting the change the filetype of a fastq file to fastqsanger, the process for changing the type goes on forever. This does not allow any further processing of the data. -- * Pete Schmitt Technical Director: Discovery Cluster NH INBRE Grid Computational Genetics Lab Dartmouth College, HB 6223 L24 Berry/Baker Library Hanover, NH 03755 Phone: 603-646-8109 http://discovery.dartmouth.edu http://columbia.dartmouth.edu/grid http://www.epistasis.org * This body part will be downloaded on demand. -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Which bam_to_bigwig to use
In the main toolshed I find bam_to_bigwig 2 times. Description, Version and Requirements seem the same. How am I to decide which one to use? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] problem with data libraries
Hi, I created a data libraries and imported files into it (from local path with symlinks) There are 1500 files in that directory. Now when I attempt to view the library I get this error: Error attempting to display contents of library (test lib): 'NoneType' object has no attribute 'dataset'. What is the problem here? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] run cluster jobs as real user
Hmm, I start feeling like I'm only talking to myself here :-) I finally have a prototyp running. Currently the mapping galaxy username - cluster username is only done using a dictionary in dmraa.py and should (and will) be moved to a configuration file. (Even better would be to put that into the database and should be editable by the admins but I'm afraid that is to much work for me right now.) I've added the diff and would appreciate if some expert would have a look on it and point out any glaring mistakes. There was a lot of guesswork about the inner workings of galaxy involved so I'm sure there are some bugs. regards, Andreas On 29.03.2012 16:12, Andreas Kuntzagk wrote: Hi, Ok, I was digging around for this and it seems to be a bit more involved then I expected. Problem is I don't really understand the interplay of DRMAAJobRunner and the user management. Can somebody provide me with some pointers? regards, Andreas On 28.03.2012 13:30, Andreas Kuntzagk wrote: Hi, after thinking more about this and also trying to get Apache use NIS (I failed with that :-( ) I just came up with an idea for a different solution: I don't use NIS for login into galaxy but when starting jobs I consult a mapping file (or the database) to map email addresses to know users. For unknown users I just continue to use the galaxy user. I guess a good starting point would be DRMAAJobRunner.queue_job Do you see any bigger problems coming my way? How best do I avoid to degress from the main tree to much? regards, Andreas file:///home/kuntzagk/galaxy_diff On 27.03.2012 09:57, Andreas Kuntzagk wrote: Hi, On 26.03.2012 17:14, Nate Coraor wrote: On Mar 22, 2012, at 12:30 PM, Alfonso Núñez Salgado wrote: Hi Andreas: As far as I know Galaxy is designed to manage its own users and not to be integrated in any other platform. I think that the idea is to use galaxy as another user of your system. Hi Alfonso, Andreas is referring to recent changes that allow jobs to run as users other than the single Galaxy user that started the server process. See: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster#Submitting_Jobs_as_the_Real_User Yes, that's what I wanted. On 20/03/12 16:31, Andreas Kuntzagk wrote: Hi, I (re-)installed galaxy on our server and managed to set it up to run jobs through our cluster (GridEngine). It's also running behind an Apache proxy now. But now I'm wondering how to setup to have the cluster jobs run as the real cluster users. I found the documentation on this topic a little thin. We use NIS for managing users and passwords across the cluster. How do I connect that to Galaxy? regards, Andreas Andreas, You'll need to have Apache authenticate your users, as explained here: http://wiki.g2.bx.psu.edu/Admin/Config/Apache%20Proxy For NIS without Kerberos, you'd probably need to use mod_auth_pam to authenticate through your system's PAM stack. Thanks, I will look into it. regards, Andreas --nate -- = Alfonso Núñez Salgado Unidad de Bioinformática Centro de Biologia Molecular Severo Ochoa C/Nicolás Cabrera 1 Universidad Autónoma de Madrid Cantoblanco, 28049 Madrid (Spain) Phone: (34) 91-196-4633 Fax: (34) 91-196-4420 web: http://ub.cbm.uam.es/ = ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich diff -r 26920e20157f lib/galaxy/config.py --- a/lib/galaxy/config.py Wed Jan 25 12:20:50 2012 -0500 +++ b/lib/galaxy/config.py Wed Apr 04 13:11:05 2012 +0200 @@ -76,6 +76,7 @@ self.retry_metadata_internally = string_as_bool( kwargs.get( retry_metadata_internally, True ) ) self.use_remote_user = string_as_bool( kwargs.get( use_remote_user, False ) ) self.remote_user_maildomain = kwargs.get( remote_user_maildomain, None ) +self.map_users = kwargs.get(map_users,False) self.remote_user_logout_href = kwargs.get( remote_user_logout_href, None ) self.require_login = string_as_bool( kwargs.get( require_login, False ) ) self.allow_user_creation = string_as_bool( kwargs.get( allow_user_creation, True ) ) diff -r 26920e20157f lib/galaxy/jobs/runners/drmaa.py --- a/lib/galaxy/jobs/runners/drmaa.py Wed Jan 25 12:20:50 2012 -0500 +++ b/lib/galaxy/jobs
Re: [galaxy-dev] run cluster jobs as real user
Hi, after thinking more about this and also trying to get Apache use NIS (I failed with that :-( ) I just came up with an idea for a different solution: I don't use NIS for login into galaxy but when starting jobs I consult a mapping file (or the database) to map email addresses to know users. For unknown users I just continue to use the galaxy user. I guess a good starting point would be DRMAAJobRunner.queue_job Do you see any bigger problems coming my way? How best do I avoid to degress from the main tree to much? regards, Andreas On 27.03.2012 09:57, Andreas Kuntzagk wrote: Hi, On 26.03.2012 17:14, Nate Coraor wrote: On Mar 22, 2012, at 12:30 PM, Alfonso Núñez Salgado wrote: Hi Andreas: As far as I know Galaxy is designed to manage its own users and not to be integrated in any other platform. I think that the idea is to use galaxy as another user of your system. Hi Alfonso, Andreas is referring to recent changes that allow jobs to run as users other than the single Galaxy user that started the server process. See: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster#Submitting_Jobs_as_the_Real_User Yes, that's what I wanted. On 20/03/12 16:31, Andreas Kuntzagk wrote: Hi, I (re-)installed galaxy on our server and managed to set it up to run jobs through our cluster (GridEngine). It's also running behind an Apache proxy now. But now I'm wondering how to setup to have the cluster jobs run as the real cluster users. I found the documentation on this topic a little thin. We use NIS for managing users and passwords across the cluster. How do I connect that to Galaxy? regards, Andreas Andreas, You'll need to have Apache authenticate your users, as explained here: http://wiki.g2.bx.psu.edu/Admin/Config/Apache%20Proxy For NIS without Kerberos, you'd probably need to use mod_auth_pam to authenticate through your system's PAM stack. Thanks, I will look into it. regards, Andreas --nate -- = Alfonso Núñez Salgado Unidad de Bioinformática Centro de Biologia Molecular Severo Ochoa C/Nicolás Cabrera 1 Universidad Autónoma de Madrid Cantoblanco, 28049 Madrid (Spain) Phone: (34) 91-196-4633 Fax: (34) 91-196-4420 web: http://ub.cbm.uam.es/ = ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] run cluster jobs as real user
Hi, On 26.03.2012 17:14, Nate Coraor wrote: On Mar 22, 2012, at 12:30 PM, Alfonso Núñez Salgado wrote: Hi Andreas: As far as I know Galaxy is designed to manage its own users and not to be integrated in any other platform. I think that the idea is to use galaxy as another user of your system. Hi Alfonso, Andreas is referring to recent changes that allow jobs to run as users other than the single Galaxy user that started the server process. See: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster#Submitting_Jobs_as_the_Real_User Yes, that's what I wanted. On 20/03/12 16:31, Andreas Kuntzagk wrote: Hi, I (re-)installed galaxy on our server and managed to set it up to run jobs through our cluster (GridEngine). It's also running behind an Apache proxy now. But now I'm wondering how to setup to have the cluster jobs run as the real cluster users. I found the documentation on this topic a little thin. We use NIS for managing users and passwords across the cluster. How do I connect that to Galaxy? regards, Andreas Andreas, You'll need to have Apache authenticate your users, as explained here: http://wiki.g2.bx.psu.edu/Admin/Config/Apache%20Proxy For NIS without Kerberos, you'd probably need to use mod_auth_pam to authenticate through your system's PAM stack. Thanks, I will look into it. regards, Andreas --nate -- = Alfonso Núñez Salgado Unidad de Bioinformática Centro de Biologia Molecular Severo Ochoa C/Nicolás Cabrera 1 Universidad Autónoma de Madrid Cantoblanco, 28049 Madrid (Spain) Phone: (34) 91-196-4633 Fax: (34) 91-196-4420 web: http://ub.cbm.uam.es/ = ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] login not working on local instance
Hi, For our fresh new local instance I created a user. But whenever I try to login it looks like I end up as anonymous user again. And if I click Workflows I'm told that I need to login first. The only thing that looks like an error message is this one: galaxy.web.framework DEBUG 2012-03-22 16:38:42,570 Error: this request returned None from get_history(): http://bbc.mdc-berlin.de/galaxy/workflow; I'm running behind an Apache, could that be the reason? My Apache config: RewriteRule ^/galaxy$ /galaxy/ [R] RewriteRule ^/galaxy/static/style/(.*) /opt/galaxy/galaxy-dist/static/june_2007_style/blue/$1 [L] RewriteRule ^/galaxy/static/scripts/(.*) /opt/galaxy/galaxy-dist/static/scripts/packed/$1 [L] RewriteRule ^/galaxy/static/(.*) /opt/galaxy/galaxy-dist/static/$1 [L] RewriteRule ^/galaxy/favicon.ico /opt/galaxy/galaxy-dist/static/favicon.ico [L] RewriteRule ^/galaxy/robots.txt /opt/galaxy/galaxy-dist/static/robots.txt [L] RewriteRule ^/galaxy(.*) http://database1:8080$1 [P] universe_wsgi.conf ... [filter:proxy-prefix] use = egg:PasteDeploy#prefix prefix = /galaxy ... [app:main] ... # Allow unregistered users to create new accounts (otherwise, they will have to # be created by an admin). #allow_user_creation = True # Allow administrators to delete accounts. #allow_user_deletion = False # Allow administrators to log in as other users (useful for debugging) #allow_user_impersonation = False regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] run cluster jobs as real user
Hi, I (re-)installed galaxy on our server and managed to set it up to run jobs through our cluster (GridEngine). It's also running behind an Apache proxy now. But now I'm wondering how to setup to have the cluster jobs run as the real cluster users. I found the documentation on this topic a little thin. We use NIS for managing users and passwords across the cluster. How do I connect that to Galaxy? regards, Andreas -- Andreas Kuntzagk SystemAdministrator Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for Molecular Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
