Re: [galaxy-dev] Galaxy Bare/Core and custom upload tool

[Stephan Oepen]

dear keith,

in our work on building the Linguistic Analysis Portal (LAP) at the
University of Oslo, we have had to confront the same two issues you
mention.  our general approach has been to separate LAP-specific code
(and specifications) from the Galaxy tree as much as possible.

for example, we keep our tool descriptions, tool implementations, job
runner configurations, job resource specifications, and such in a
separately versioned tree and use the various options in ‘galaxy.ini’
to point to our own variants, e.g.

  tool_path = /home/laportal/tools
  tool_config_file = /home/laportal/tools/config.xml
  job_config_file = /home/laportal/tools/jobs.xml
  job_resource_params_file = /home/laportal/tools/resources.xml

for the upload tool, we modify the JavaScript files (‘upload-row.js’
and ‘upload-view.js’) to hide UI elements (notably the Genome
selection) that do not apply to us, and we have added custom code to
the actual tool implementation to post-process incoming data.  as for
the latter changes, our modified ‘upload.py’ and ‘upload.xml’ reside
outside the Galaxy tree (with the other tools), so our only risk is
that we need to validate compatibility (or import updates) when moving
to newer Galaxy versions.  for the JavaScript changes, we keep a
separate copy of the handful of UI files (from below ‘client/galaxy/’)
that we modify outside of Galaxy and use a modified ‘GruntFile.js’
(with different ‘dest’ targets) to make grunt(1) propagate our changes
into the Galaxy tree.

maybe some of these techniques could work for you too?  oe


On Thu, Mar 31, 2016 at 10:17 PM, Suderman Keith  wrote:
> Dear Galaxy Team,
>
> Two questions.
>
> 1) Is there a bare-bones version of Galaxy available somewhere?  That is, 
> Galaxy with no tools pre-installed?  We are creating Ansible play books for 
> configuring Docker/Galaxy instances with our NLP tools installed.  Currently 
> we are using a fork of the Galaxy project with the bio tools removed, but 
> this is less than ideal when we try to update to new versions of Galaxy.
>
> Alternatively, does anyone have an automated process to delete unwanted tools 
> that we could run immediately after cloning the main Galaxy repo?
>
> 2) Is is possible to either a) modify the existing upload tool, or b) 
> implement a new upload tool?  Several options in the upload tool do not make 
> sense in our domain (e.g. selecting a genome) and we would also like to do 
> some post-processing to files after they have been uploaded.  I've looked at 
> the code for the existing upload tool and it seems quite tightly coupled to 
> the Galaxy code base.  I was hoping the "Downgrade upload tool" thread from a 
> few weeks ago was going to help, but unfortunately for me (and good for the 
> user) they found a solution that didn't involve installing a new upload tool.
>
> However, could I follow the approach suggested in that thread?  I.E. go back 
> in time and find an old upload.[py | xml] and use that as a starting point 
> for my own custom upload tool?  How far back in time would I have to go to 
> find an upload.py that was a standalone tool?
>
> Cheers,
> Keith
>
> --
> Research Associate
> Department of Computer Science
> Vassar College
> Poughkeepsie, NY
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] per-tool job resource defaults

[Stephan Oepen]

dear colleagues,

at the university of oslo, we develop a galaxy-based portal for
natural language processing (LAP: Language Analysis Portal).  jobs are
submitted to a compute cluster via DRMAA and SLURM.  current
development is against the galaxy release of march 2015.

i am wondering about fine-grained control of job resources.  our goal
is that most users need not look past the ‘Use default job resource
parameters’ toggle in the job configuration dialogue.

as i understand it, i think we can populate the ‘nativeSpecification’
parameter in ‘job_conf.xml’ with SLURM-specific command-line options
to set defaults, for example the project, maximum run-time, number of
cores, memory per core, and such.  i assume these defaults will be
combined with and overwritten by ‘custom’ job resource parameters, in
case any are specified in the job configuration dialogue?

i tried to track the flow of information from
‘lib/galaxy/jobs/runners/drmaa.py’ via
‘scripts/drmaa_external_runner.py’ into the drmaa-python egg, but i
could not easily work out where the merging of ‘nativeSpecification’
and custom resource parameters happens; presumably at about the same
time as an actual job script file is created, for submission to SLURM?
 could someone point me in the right direction here?

—more importantly, maybe: we would like to establish per-tool resource
defaults.  for example, some of our tools require substantially more
memory per core than others.  i cannot easily find a way of
associating resource default with individual tools.  i looked at the
tool configuration syntax, ‘job_conf.xml.sample_advanced’, and
‘job_resource_params_conf.xml.sample’, as well as at the following
documentation pages:

  https://wiki.galaxyproject.org/Admin/Config/Jobs
  https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster

i am hoping i am overlooking something :-).  is there a way to define
job resource defaults on a per-tool basis?

with warmest thanks in advance, oe
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/