Re: [galaxy-dev] Best location for galaxy environmental variables
We are using environment modules ( http://modules.sourceforge.net/ ) on the cluster and took same approach for the galaxy install. The module files are loaded using .bashrc file of galaxy user. It's also useful for non-galaxy users/admins for running galaxy-tools from command-line by simply loading required module files in their shell environment. -- Shantanu. On Jul 20, 2011, at 10:28 AM, michael burrell (TSL) wrote: Good afternoon, More of a general discussion point, where is the best place to put the environmental variable declarations within galaxy. To date I have been placing them as exports in the run.sh, though I can not help but wonder if there is a better place, which is appropriate whatever method you use to kick off galaxy. Thanks Michael. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Handling runner-specific (SLURM) DRMAA settings (Roman Valls)
Thanks indeed for the ~/.slurm_drmaa.conf hint Mariusz, very useful !
I wonder how this approach will work out when one runs galaxy as advised
in a production setting:
http://usegalaxy.org/production
Together with the gordon patch to run drmaa as different users:
http://wiki.g2.bx.psu.edu/Events/GCC2011?action=AttachFiledo=gettarget=RunningGalaxyDRMAAJobsAsDifferentUsers.pdf
My concern is that probably the SUID code will end up clearing up all
the environment variables and/or ignoring the user's ~/.slurm_drmaa.conf
as happened to me before with screen:
http://superuser.com/questions/235760/ld-library-path-unset-by-screen
I'll jump in that Gordon patch/code eventually, but for now your
insight/fixes have been very helpful from a galaxy developer instance
perspective, thanks much again !
/Roman
On 2011-07-19 19:53, Mariusz Mamoński wrote:
The general config file allows us to set a fixed project:
default_cluster_job_runner = drmaa://-A a2010002 -p core
And even set per-tool job settings. But we would like each user to have
the ability to change those settings.
What is the least intrusive way to set per-user native (site-specific)
job manager settings ?
you may try to use an user's local DRMAA configuration file:
~/.slurm_drmaa.conf
. If multiple configuration sources are present then all
configurations are merged with values from user-defined files taking
precedence (in following order: $SLURM_DRMAA_CONF,
~/.slurm_drmaa.conf, /etc/slurm_drmaa.conf)
there you can put any user's specific settings, e.g.:
job_categories: {
default: -A a2010002 -p core,
}
Cheers,
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] workflows are broken in current galaxy-central version
Fixed on trunk, thanks for reporting! K On Thu, Jul 21, 2011 at 12:41 PM, Edward Kirton eskir...@lbl.gov wrote: URL: https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8 Module paste.exceptions.errormiddleware:*143* in __call__ Module paste.debug.prints:*98* in __call__ Module paste.wsgilib:*539* in intercept_output Module paste.recursive:*80* in __call__ Module paste.httpexceptions:*632* in __call__ Module galaxy.web.framework.base:*145* in __call__ https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8#body *=* method*(* trans*,* kwargs *)* Module galaxy.web.framework:*84* in decorator https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8#* return* func*(* self*,* trans*,* ***args*,* kwargs *)* Module galaxy.web.controllers.workflow:*686* in editor https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8#* return* trans*.*fill_template*(* workflow/editor.mako*,* stored*=*stored *,* annotation*=*self*.*get_item_annotation_str*(* trans*.*sa_session*,* trans*.*user*,* stored *)* *)* Module galaxy.web.framework:*661* in fill_template https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8#* return* self*.*fill_template_mako*(* filename*,* kwargs *)* Module galaxy.web.framework:*672* in fill_template_mako https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8#* return* template*.*render*(* data *)* Module mako.template:*296* in render https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8#* return* runtime*.*_render*(*self*,* self*.*callable_*,* args*,* data*)* Module mako.runtime:*660* in _render https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8# _kwargs_for_callable*(*callable_*,* data*)**)* Module mako.runtime:*692* in _render_context https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8# _exec_template*(*inherit*,* lclcontext*,* args*=*args*,* kwargs*=*kwargs*) * Module mako.runtime:*718* in _exec_template https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8# callable_*(*context*,* ***args*,* kwargs*)* Module _base_panels_mako:*87* in render_body https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8# __M_writer*(*unicode*(*self*.*overlay*(*visible*=*self*.*overlay_visible*) **)**)* *TypeError: render_overlay() got an unexpected keyword argument 'visible'* ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] EpiGRAPH
Hello Christoph, This is the result of a bug that I've corrected in change set 5819:0f15591f2acd, which should be available in our main Galaxy instance, as well as the distribution, within the next few weeks. Thanks for pointing this out, and very sorry for the inconvenience. Greg Von Kuster On Jul 21, 2011, at 12:58 AM, Christoph Bock wrote: Hi Greg, Anton, I received a user bug report asking why the “Send Data - EpiGRAPH” function in Galaxy doesn’t work anymore. Is this a temporary technical issue or did you do a general cleanup and removed this link? Best regards, Christoph From: christoph.sch...@unibas.ch [mailto:christoph.sch...@unibas.ch] Sent: 13 July 2011 3:56 AM To: cb...@mpi-inf.mpg.de Subject: EpiGRAPH Dear Christoph, I am trying to get an overview on existing computational solutions in the public domain to compare epigenetic maps. One interesting solution is your EpiGRAPH Software (Genome Biol 10, R14.[2009]). I have the following questions: 1) Could I get access to the most recent source code release, installation instructions (I saw the pre-print of the tutorial paper)? Any comments on current and future development activities on EpiGRAPH? 2) What is the state of the integration of EpiGRAPH into the Galaxy framework? As far as I remember EpiGRAPH was available for a short moment onhttp://main.g2.bx.psu.edu/, but now I can no longer see it? Is it still available on another mirror? thanks in advance for your help -Christoph Christoph D. Schmid, PhD Swiss Tropical and Public Health Institute Socinstrasse 57 CH-4002 Basel Switzerland skype: chimborazo93 Tel +41 61 284 8323 Fax +41 61 284 8101 www.swisstph.ch -- This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately by reply e-mail and delete this message from your system. -- Greg Von Kuster Galaxy Development Team g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Detailed SGE timing information about galaxy jobs
Hello Assaf, I've added this to the ~/contrib directory in change set 5820:c609c2574c66. Thanks very much for contributing this. Greg Von Kuster On Jul 19, 2011, at 2:45 PM, Assaf Gordon wrote: Hi all, I got around to writing the report I always needed: how much time each job actually runs and how much time it spends waiting in the SGE queue. The attached shell script produces the report, combining information from SGE's QACCT with the galaxy job/dataset information. The output contains: job, user, tool name, dbkey, total input size in bytes, waiting time in SGE queue, actual SGE executiong running time, and some other tidbits. This allows finding how much running time each user had on the cluster, how much time each tool/user spend idly waiting, and some possible correlations between tools, dbkeys, input size and running time. The script is tightly coupled with SGE and PostgreSQL, but can probably be adapted to PBS/MySQL. Hope this helps someone, -gordon collect_sge_job_timings.sh___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ Greg Von Kuster Galaxy Development Team g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] workflows are broken in current galaxy-central version
yup, works. thanks for the prompt reply On Thu, Jul 21, 2011 at 10:15 AM, Kanwei Li kan...@gmail.com wrote: Fixed on trunk, thanks for reporting! K On Thu, Jul 21, 2011 at 12:41 PM, Edward Kirton eskir...@lbl.gov wrote: URL: https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8 Module paste.exceptions.errormiddleware:*143* in __call__ Module paste.debug.prints:*98* in __call__ Module paste.wsgilib:*539* in intercept_output Module paste.recursive:*80* in __call__ Module paste.httpexceptions:*632* in __call__ Module galaxy.web.framework.base:*145* in __call__ https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8# body *=* method*(* trans*,* kwargs *)* Module galaxy.web.framework:*84* in decorator https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8#* return* func*(* self*,* trans*,* ***args*,* kwargs *)* Module galaxy.web.controllers.workflow:*686* in editor https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8#* return* trans*.*fill_template*(* workflow/editor.mako*,* stored*=* stored*,* annotation*=*self*.*get_item_annotation_str*(* trans*.* sa_session*,* trans*.*user*,* stored *)* *)* Module galaxy.web.framework:*661* in fill_template https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8#* return* self*.*fill_template_mako*(* filename*,* kwargs *)* Module galaxy.web.framework:*672* in fill_template_mako https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8#* return* template*.*render*(* data *)* Module mako.template:*296* in render https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8#* return* runtime*.*_render*(*self*,* self*.*callable_*,* args*,* data*)* Module mako.runtime:*660* in _render https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8#*** *_kwargs_for_callable*(*callable_*,* data*)**)* Module mako.runtime:*692* in _render_context https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8# _exec_template*(*inherit*,* lclcontext*,* args*=*args*,* kwargs*=*kwargs* )* Module mako.runtime:*718* in _exec_template https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8# callable_*(*context*,* ***args*,* kwargs*)* Module _base_panels_mako:*87* in render_body https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8# __M_writer*(*unicode*(*self*.*overlay*(*visible*=*self*.*overlay_visible* )**)**)* *TypeError: render_overlay() got an unexpected keyword argument 'visible' * ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Add Genome
Hello Somayyeh, These data are from the same authors/release build. At Ensembl, the same 5 core chromosomes plus an additional 78 unplaced scaffolds are available. The data authors decided not to include these scaffolds in their publication at NCBI. For a primary genome, I suggest that we use the core 5 chromosomes as this is what was published externally to NCBI by the authors. Would this also be your preference? Both options are possible. Many genomes have unmapped scaffolds included in the Galaxy genome builds, however they are usually published at NCBI if/when the genome is submitted there. Thank you! Jennifer On 7/21/11 5:41 AM, Somayyeh Fahiminiya wrote: Hi Jen, Thanks for your email and sorry to be late . Please look at : wget ftp://ftp.ensemblgenomes.org/pub/plants/current/fasta/brachypodium_distachyon/dna/Brachypodium_distachyon.Brachy1.0.dna.toplevel.fa.gz I dont know if it is more recent or the same version as yours Thanks again somayyeh On Mon, Jul 11, 2011 at 9:05 PM, Jennifer Jackson j...@bx.psu.edu mailto:j...@bx.psu.edu wrote: Hello, To confirm: the correct source would be this project, only has chromosomes 1-5, at NCBI? Or is there a preferred (more complete/recent) source? http://www.ncbi.nlm.nih.gov/__bioproject/32607 http://www.ncbi.nlm.nih.gov/bioproject/32607 Once we have the source confirmed, we can add this genome to our to-add list, Best, Jen Galaxy team On 7/7/11 6:25 AM, Somayyeh Fahiminiya wrote: Hello everyone, Could you plz add the genome of *Brachypodium distachyon* to the Galaxy. Thanks in advance _ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Jennifer Jackson http://usegalaxy.org/ http://galaxyproject.org/ -- Jennifer Jackson http://usegalaxy.org/ http://galaxyproject.org/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Per-user tools permissions
Hi, I've noticed a ticket on the wiki for per-user tool permissions: https://bitbucket.org/galaxy/galaxy-central/issue/269/use-galaxy-security-to-restrict-tool If anyone is interested, the following patch implements such thing (but requires manual code changes whenever the access needs to be changed). The method tool_allowed_for_user() can of course be extended/rewritten to read access permissions from file or from a database. Comments are welcomed, -gordon diff -r b23f64804904 -r d6b37783b25e lib/galaxy/util/gordon_tool_permissions.py --- /dev/null Thu Jan 01 00:00:00 1970 + +++ b/lib/galaxy/util/gordon_tool_permissions.py Thu Feb 03 12:29:41 2011 -0500 @@ -0,0 +1,41 @@ +import logging +import threading, random, string, re, binascii, pickle, time, datetime, math, re, os, sys, tempfile + +log = logging.getLogger(__name__) + +# Hack by gordon: +# returns True if the given user is allowed to view/execute the given tool +# very hackish and ugly, but should work. +# Called by ./lib/galaxy/web/controller/tool_runner.py in index() +# +# Added to support outside collaborators (who aren't allowed to run certain tools) +# TODO: make this much more generic (and read access list from an external file) +def tool_allowed_for_user ( trans, tool_id, tool, user, user_is_admin ): +# This hack requires users to register +if user is None: +log.warn ( tool_allowed_for_user ( tool = %s, user = None ) - blocking tool usage. ) +return False + +if user_is_admin: +log.warn (tool_allowed_for_user ( tool = %s, user = %s, user_is_admin ) - allowing tool for administrator % ( tool_id, user.email ) ) +return True + +log.warn ( checking tool_allowed_for_user ( tool = %s, user = %s ) % ( tool_id, user.email ) ) + +# block collaborator's access to import/export tools +if ( user.email.find ( x...@other.place.edu ) == 0 ): +if ( tool_id.find ( cshl_import) != -1 ) or ( tool_id.find(cshl_export) != -1 ): +log.warn ( tool_allowed_for_user ( tool = %s, user = %s ) - blocking tool usage. % ( tool_id, user.email ) ) +return False + +# Solexa watch tools: +# for now, allow only Asasf and Oliver. +# Security hole: if someone has tam/gordon/hodges as the beginning of the email, he/she will also be able to run the tool ;( +if ( tool_id.find (solwatch) != -1): +if ( user.email.find(xxx...@cshl.edu)==0 ) or ( user.email.find(xxx...@cshl.edu)==0) or ( user.email.find(gor...@cshl.edu)==0 ) : +return True +return False +# +# By default, allow access +return True + diff -r b23f64804904 -r d6b37783b25e lib/galaxy/web/controllers/tool_runner.py --- a/lib/galaxy/web/controllers/tool_runner.py Thu Feb 03 09:34:39 2011 -0500 +++ b/lib/galaxy/web/controllers/tool_runner.py Thu Feb 03 12:29:41 2011 -0500 @@ -7,6 +7,7 @@ from galaxy.tools import DefaultToolState from galaxy.tools.parameters.basic import UnvalidatedValue from galaxy.tools.actions import upload_common +from galaxy.util import gordon_tool_permissions import logging log = logging.getLogger( __name__ ) @@ -58,6 +59,18 @@ log.error( index called with tool id '%s' but no such tool exists, tool_id ) trans.log_event( Tool id '%s' does not exist % tool_id ) return Tool '%s' does not exist, kwd=%s % (tool_id, kwd) +# +# Hack By Gordon: +# Disable certain tools, if the user is an outside collaborator +# +if not util.gordon_tool_permissions.tool_allowed_for_user ( trans, tool_id, tool, trans.user, trans.user_is_admin() ) : +user_email = unregistered user +if not (trans.user is None): +user_email = trans.user.email +log.error(disabling tool %s for outside user %s % ( tool_id, user_email ) ) +trans.log_event(disabling tool %s for outside user %s % ( tool_id, user_email ) ) +return Sorry, tool '%s' is disabled for user '%s'. % ( tool_id, user_email ) +# End of Gordon Hack params = util.Params( kwd, sanitize = False ) #Sanitize parameters when substituting into command line via input wrappers #do param translation here, used by datasource tools if tool.input_translator: ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] TEMP variable in cluster install
We have configured galaxy to work with our SGE cluster using drmaa job runner interface. We are using 'unified method' for this install and both TEMP environment variable and new_file_path in universe_wsgi.ini file have been configured correctly. However, we are seeing some errors where local /tmp space on compute nodes is being referenced by the galaxy tools. Specifically we saw it mentioned in error messages from following tools: * bwa_wrapper and upload tools: 'No space left on device: /tmp...' * sam_to_bam tool: 'Error sorting alignments from /tmp/filename..' Shouldn't it be referencing TEMP environment variable or new_file_path configuration value? Is it getting overridden by TMP or TMPDIR variables in python wrapper code? Has anyone else experienced similar issue? -- Thanks, Shantanu. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] new data type.
Hi, I created an XML file for a tool. This tool has an Nexus input. When I click the input field of the web interface for this tool, it shows all kinds of data that I can select. How do I restrict the list to show only data in nexus format? I added a line in the registration part and the sniffer part of the datatypes_conf.xml and I also created a class called Nexus which inherits the class galaxy.datatypes.Sequence, but this doesn't solve the problem. Thanks, Luobin ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
