On Mon, Nov 24, 2014 at 12:56 PM, Dan Tenenbaum <dtene...@fredhutch.org> wrote:
> ----- Original Message -----
>> From: "John Chilton" <jmchil...@gmail.com>
>> To: "Dan Tenenbaum" <dtene...@fredhutch.org>
>> Cc: firstname.lastname@example.org
>> Sent: Monday, November 24, 2014 9:30:50 AM
>> Subject: Re: [galaxy-dev] more control over the "docker" command
>> On Mon, Nov 24, 2014 at 10:54 AM, Dan Tenenbaum
>> <dtene...@fredhutch.org> wrote:
>> > Hi,
>> > Thanks for supporting the running of docker containers in Galaxy.
>> > I have two requests for more control over the docker command that
>> > is run.
>> > According to https://github.com/apetkau/galaxy-hackathon-2014 , the
>> > docker command that is run when a docker-enabled tool is run might
>> > look something like this:
>> > command is: sudo docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
>> > /home/aaron/Projects/galaxy-central:/home/aaron/Proje
>> > cts/galaxy-central:ro -v
>> > /home/aaron/Projects/galaxy-central/tools/docker:/home/aaron/Projects/galaxy-central/tools/docker:ro
>> > -v /home/aaron/Projects/galaxy-central/datab
>> > ase/job_working_directory/000/6:/home/aaron/Projects/galaxy-central/database/job_working_directory/000/6:rw
>> > -v /home/aaron/Projects/galaxy-central/database/files:/home/aa
>> > ron/Projects/galaxy-central/database/files:rw -w
>> > /home/aaron/Projects/galaxy-central/database/job_working_directory/000/6
>> > --net none busybox:ubuntu-14.04
>> > /home/aaron/Projects/galaxy-central/database/job_working_directory/000/6/container.sh;
>> > return_code=$?; if [ -f
>> > /home/aaron/Projects/galaxy-central/database/job_working_directory/000/6/wo
>> > rking_file ] ; then cp
>> > /home/aaron/Projects/galaxy-central/database/job_working_directory/000/6/working_file
>> > /home/aaron/Projects/galaxy-central/database/files/000/dataset_10.dat
>> > ; fi; sh -c "exit $return_code"
>> > I'd like to be able to specify extra flags to be included in the
>> > command. In my case I'd like to include "--link server:server"
>> > because I want to link this container with another container that
>> > contains a long-running server process (I can describe my use case
>> > in greater detail if desired).
>> > Can there be a way to do this in my tool wrapper?
>> I certainly don't mind adding more options - we've added a bunch as
>> different interesting things have been attempted (e.g. docker in
>> docker). I would like to keep the tool wrapper very general though
>> expose these options in the job_conf.xml instead - since for the most
>> part they seem like they will be related to the tool's deployment and
>> not the abstract command being produced. I have added this
>> functionality with the following changeset -
>> Simply add <param id="docker_run_extra_arguments">--link
>> server:server</param> to your job destination. Let me know if this
>> doesn't work out.
> Thanks very much for the quick turnaround!
> I think it will work out. I am not an expert in job_conf.xml. My main concern
> is being able to pass extra arguments to some docker-enabled tools but not to
> others, is that possible?
Many job destinations can map to the same underlying compute resource
(cluster, server, etc...) - so I would just duplicate your job
destination definition and customize each one for the different tool
and then your tool section - make sure a given tool maps to the
correct destination. There will be some unfortunate duplication - but
it shouldn't be too bad unless you are doing a bunch of very cool
things with dozens of different tools.
>> > Secondly, rather than giving my galaxy user passwordless sudo, I'd
>> > like to add it to the docker group, then I can run docker commands
>> > without prepending sudo. This seems a lot safer. Could this be
>> > exposed, maybe by a sudo="false" attribute in the tool wrapper?
>> This is currently possible - in your job_conf.xml just add to the job
>> destination parameter:
>> <param id="docker_sudo">false</param>
>> Checkout the admittedly huge file config/job_conf.xml.sample_advanced
>> for a full list of the supported options. The docker related options
>> are all near each other and apply to any built-in job runner.
>> I suspect tot running with sudo really should be the default and I am
>> growing frustrated it is not. I am unsure if I should "fix" it now
>> break the couple sites doing Docker in Galaxy or just leave it and
>> have everyone who wants to turn sudo off disable it explicitly.
> Great, thanks. Actually, as was pointed out here (and as I realize now),
> adding the galaxy user to the docker group is not necessarily safer
> (though it superficially looks like it because the word 'sudo' disappears).
> So I think perhaps docker_sudo should remain true by default.
>> At any rate - keep us informed about how things are going - I would
> I will let you know how it goes.
>> love to hear about what you are attempting to do with the linked
>> persistent docker container :).
> Briefly, our tool is an R script that loads several R packages and then
> performs a calculation based on user input.
> The problem is that it can take several minutes just to load the packages
> required to perform the calculation,
> but the calculation itself only takes a couple of seconds. Fast performance
> is important to us, so we set up
> an instance of R that already has the packages loaded and we "serve" it using
> the Rserve package. So now the galaxy tool just has to send an expression to
> Rserve which can evaluate it quickly and reduce the tool running time from
> minutes to seconds. We are already using this setup in a non-docker context
> and your change now (I hope) enables us to do this with Docker.
Very cool, good luck!
>> > Thanks,
>> > Dan
>> > ___________________________________________________________
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client. To manage your subscriptions to this
>> > and other Galaxy lists, please use the interface at:
>> > https://lists.galaxyproject.org/
>> > To search Galaxy mailing lists use the unified search at:
>> > http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
To search Galaxy mailing lists use the unified search at: