Re: [galaxy-dev] Unable to run simple Docker tool
Hi;
The problem is definitively Galaxy related, and the cause seems to be that
my program, when invoked in the Docker image from Galaxy, it's unable to
resolve the service host. The weird thing is that it can resolve it when
invoked within Docker, so there must be a problem with how Galaxy invokes
the program.
I have reproduced the problem by implementing a new client from scratch,
based on Apache Http Components, that simply POSTs the input file, received
from Galaxy, to the service. The exception, this time, is more informative:
Exception in thread main java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.net.UnknownHostException: sadiframework.org: unknown error
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at
org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:259)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:159)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at
es.eurohelp.sadi.client.SynchronousClient.main(SynchronousClient.java:61)
... 5 more
Any ideas?
Thanks
The client:
/**
* Copyright (c) 2015 Mikel Egaña Aranguren
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see http://www.gnu.org/licenses/.
*
*/
package es.eurohelp.sadi.client;
import java.io.File;
import java.io.IOException;
import org.apache.http.Header;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.FileEntity;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.util.EntityUtils;
/**
* @author Mikel Egaña Aranguren
* @version
* @date 2015 abu 29
*/
public class SynchronousClient {
/**
* @param args
* @throws IOException
* @throws ClientProtocolException
*/
public static void main(String[] args) throws ClientProtocolException,
IOException {
String service = args[0].trim();
String rdf_input = args[1];
System.out.println(service + -- + rdf_input);
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(service);
FileEntity input_file = new FileEntity(new File(rdf_input));
post.setEntity(input_file);
System.out.println(EntityUtils.toString(post.getEntity()));
HttpResponse response = client.execute(post);
Header[] headers = response.getAllHeaders();
for(Header header : headers){
System.out.println(header.getName() +--+header.getValue());
}
String response_entity_string =
EntityUtils.toString(response.getEntity());
post.releaseConnection();
Re: [galaxy-dev] Unable to run simple Docker tool
Hi again;
I can execute Docker tools from Galaxy (thanks!), however now I have a
further problem that must be Galaxy-related and I can't debug straight
away. My Galaxy tool is a simple java jar that gets a service URL and an
RDF file as inputs [1], invokes the service by POSTing the RDF to the URL
and then prints out the output of the service as another RDF file. My jar
(pasted at the end) works in my file system, works in the docker image, but
when the Docker image is executed by Galaxy it throws an exception. I
understand that the exception is local to the application and no one will
be familiar with the SADI libraries, but perhaps someone has a hint since
the problem clearly involves the way Galaxy invokes the program in the
Docker image.
As mentioned, the manual execution of the program within Docker works fine
(it prints out the url and file path, a warning from the service, and the
output RDF):
root@df69a0a2b06b:/# java -jar sadi/test-param.jar
http://sadiframework.org/examples/hello input.rdf
http://sadiframework.org/examples/hello--input.rdf
WARN [main] (RDFDefaultErrorHandler.java:36) -
http://sadiframework.org/examples/hello(line 2 column 48): {W119} A
processing instruction is in RDF content. No processing was done.
rdf:RDF
xmlns:rdf=http://www.w3.org/1999/02/22-rdf-syntax-ns#;
xmlns:hello=http://sadiframework.org/examples/hello.owl#;
rdf:Description rdf:about=http://example.com/guy;
hello:greeting
rdf:datatype=http://www.w3.org/2001/XMLSchema#string;Hello,
Guy Incognito!/hello:greeting
rdf:type rdf:resource=
http://sadiframework.org/examples/hello.owl#GreetedIndividual/
/rdf:Description
/rdf:RDF
root@df69a0a2b06b:/#
However, when I execute the program in Galaxy, the program starts fine (the
URL and file path are printed out [2]) but the following exception is
raised when the program is trying to invoke the service, as if the service
were unreachable:
org.sadiframework.client.ServiceConnectionException: sadiframework.org
at
org.sadiframework.client.ServiceImpl.loadServiceModel(ServiceImpl.java:96)
at
org.sadiframework.client.ServiceFactory.createService(ServiceFactory.java:23)
at
es.eurohelp.sadi.client.NoInferenceClient.main(NoInferenceClient.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Any hints?
Thanks
[1] The program (Maven dependencies from
http://mvnrepository.com/artifact/org.sadiframework/sadi-client/0.3.0):
package es.eurohelp.sadi.client;
import java.io.InputStream;
import java.util.Collection;
import org.sadiframework.SADIException;
import org.sadiframework.client.Service;
import org.sadiframework.client.ServiceFactory;
import com.hp.hpl.jena.rdf.model.Model;
import com.hp.hpl.jena.rdf.model.ModelFactory;
import com.hp.hpl.jena.rdf.model.Resource;
import com.hp.hpl.jena.util.FileManager;
public class NoInferenceClient {
public static void main(String[] args) {
String url = args[0].trim();
String input = args[1];
System.out.println(url + -- + input);
try {
Service service = ServiceFactory.createService(url);
Model model = ModelFactory.createOntologyModel();
InputStream in = FileManager.get().open(input);
model.read(in, null);
CollectionResource inputs =
service.discoverInputInstances(model);
Model output = service.invokeService(inputs);
output.write(System.out, RDF/XML);
output.close();
} catch (SADIException e) {
e.printStackTrace();
}
}
}
[2]
http://sadiframework.org/examples/hello--/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/files/000/dataset_5.dat
2015-08-19 18:21 GMT+02:00 Mikel Egaña Aranguren
mikel.egana.arangu...@gmail.com:
2015-08-19 15:50 GMT+02:00 John Chilton jmchil...@gmail.com:
I don't know to be honest - a couple of things to verify.
I wrote a bunch of debug stuff right away at the end of the e-mail and
you can try if I am wrong, but after I wrote I realized the problem.
You are not using sudo for running docker in your examples - this is
paused because sudo is waiting on a password and you don't have
passwordless sudo setup probably.
I would just add param id=docker_sudofalse/param to your job
conf destination and this should work.
Yes, this worked.
Thanks!
Regards
The other stuff to try:
Does the tool work without using Docker - if you just place test-io.sh
on Galaxy's PATH. If yes, I would set cleanup_job = never in
galaxy.ini and try again. Once it dies, grab this part of the command
line:
sudo docker run -e
Re: [galaxy-dev] Unable to run simple Docker tool
2015-08-19 15:50 GMT+02:00 John Chilton jmchil...@gmail.com: I don't know to be honest - a couple of things to verify. I wrote a bunch of debug stuff right away at the end of the e-mail and you can try if I am wrong, but after I wrote I realized the problem. You are not using sudo for running docker in your examples - this is paused because sudo is waiting on a password and you don't have passwordless sudo setup probably. I would just add param id=docker_sudofalse/param to your job conf destination and this should work. Yes, this worked. Thanks! Regards The other stuff to try: Does the tool work without using Docker - if you just place test-io.sh on Galaxy's PATH. If yes, I would set cleanup_job = never in galaxy.ini and try again. Once it dies, grab this part of the command line: sudo docker run -e GALAXY_SLOTS=$GALAXY_SLOTS -v /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy:/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy:ro -v /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/tools/catDocker:/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/tools/catDocker:ro -v /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2:/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2:rw -v /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/files:/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/files:rw -w /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2 --net none --rm -u 1001 mikeleganaaranguren/busybox-galaxy-test-io:v1 /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2/tool_script.sh and try to debug the problem outside of Galaxy. Maybe using docker logs for instance. If the logs don't reveal anything try dropping some of the command-line arguments and see if that is the problem - e.g. --net none or -u 1001. -John On Tue, Aug 18, 2015 at 6:32 PM, Mikel Egaña Aranguren mikel.egana.arangu...@gmail.com wrote: Hi; I'm trying to develop a Docker based tool, as suggested by a peer-reviewer who might be reading this :P However, I'm having trouble with even the most basic setting, and I don't know what might be wrong, so any help will be much appreciated. I have developed a very simple docker image and corresponding Galaxy tool, so that I get it working before starting with the actual tool, but when I execute it through Galaxy it simply stays executing forever, instead of failing or terminating. My image simply executes a shell script that reads the content of a file and concatenates a string to it. The image: FROM busybox:ubuntu-14.04 MAINTAINER Mikel Egaña Aranguren mikel.egana.arangu...@gmail.com RUN mkdir /sadi COPY test-io.sh /sadi/ RUN chmod a+x /sadi/test-io.sh ENV PATH $PATH:/sadi The test-io.sh script within the image: #!/bin/sh cat $1 echo AAA Invoking the container and executing the script through a normal shell works fine: REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE mikeleganaaranguren/busybox-galaxy-test-io v1 9c2b8bdade1d54 minutes ago 5.609 MB docker run -i -t mikeleganaaranguren/busybox-galaxy-test-io:v1 BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash) Enter 'help' for a list of built-in commands. ~ # ~ # echo BBB test ~ # test-io.sh test BBB AAA This is the tool file I'm using in Galaxy: tool id=SADIBUSYBOX name=SADIBUSYBOX descriptionIO/description requirements container type=dockermikeleganaaranguren/busybox-galaxy-test-io:v1/container /requirements command test-io.sh $input $output /command inputs param name=input type=data label=Dataset/ /inputs outputs data format=txt name=output / /outputs help /help /tool And my job_conf.xml: ?xml version=1.0? !-- A sample job config that explicitly configures job running the way it is configured by default (if there is no explicit config). -- job_conf plugins plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner workers=4/ /plugins handlers handler id=main/ /handlers destinations default=docker_local destination id=local runner=local/ destination id=docker_local runner=local param id=docker_enabledtrue/param /destination /destinations /job_conf As I said, when I execute the tool in Galaxy, it simply executes forever, it stays in a yellow state, till I kill the Galaxy server. The log says: 127.0.0.1 - - [18/Aug/2015:19:08:00 +0200] GET /tool_runner?tool_id=SADIBUSYBOX HTTP/1.1 200 - http://127.0.0.1:8080/ Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0 galaxy.tools.actions INFO 2015-08-18 19:08:07,178 Handled output
[galaxy-dev] Unable to run simple Docker tool
Hi; I'm trying to develop a Docker based tool, as suggested by a peer-reviewer who might be reading this :P However, I'm having trouble with even the most basic setting, and I don't know what might be wrong, so any help will be much appreciated. I have developed a very simple docker image and corresponding Galaxy tool, so that I get it working before starting with the actual tool, but when I execute it through Galaxy it simply stays executing forever, instead of failing or terminating. My image simply executes a shell script that reads the content of a file and concatenates a string to it. The image: FROM busybox:ubuntu-14.04 MAINTAINER Mikel Egaña Aranguren mikel.egana.arangu...@gmail.com RUN mkdir /sadi COPY test-io.sh /sadi/ RUN chmod a+x /sadi/test-io.sh ENV PATH $PATH:/sadi The test-io.sh script within the image: #!/bin/sh cat $1 echo AAA Invoking the container and executing the script through a normal shell works fine: REPOSITORY TAG IMAGE IDCREATED VIRTUAL SIZE mikeleganaaranguren/busybox-galaxy-test-io v1 9c2b8bdade1d54 minutes ago 5.609 MB docker run -i -t mikeleganaaranguren/busybox-galaxy-test-io:v1 BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash) Enter 'help' for a list of built-in commands. ~ # ~ # echo BBB test ~ # test-io.sh test BBB AAA This is the tool file I'm using in Galaxy: tool id=SADIBUSYBOX name=SADIBUSYBOX descriptionIO/description requirements container type=dockermikeleganaaranguren/busybox-galaxy-test-io:v1/container /requirements command test-io.sh $input $output /command inputs param name=input type=data label=Dataset/ /inputs outputs data format=txt name=output / /outputs help /help /tool And my job_conf.xml: ?xml version=1.0? !-- A sample job config that explicitly configures job running the way it is configured by default (if there is no explicit config). -- job_conf plugins plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner workers=4/ /plugins handlers handler id=main/ /handlers destinations default=docker_local destination id=local runner=local/ destination id=docker_local runner=local param id=docker_enabledtrue/param /destination /destinations /job_conf As I said, when I execute the tool in Galaxy, it simply executes forever, it stays in a yellow state, till I kill the Galaxy server. The log says: 127.0.0.1 - - [18/Aug/2015:19:08:00 +0200] GET /tool_runner?tool_id=SADIBUSYBOX HTTP/1.1 200 - http://127.0.0.1:8080/; Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0 galaxy.tools.actions INFO 2015-08-18 19:08:07,178 Handled output (77.825 ms) galaxy.tools.actions INFO 2015-08-18 19:08:07,280 Verified access to datasets (12.387 ms) galaxy.tools.execute DEBUG 2015-08-18 19:08:07,307 Tool [SADIBUSYBOX] created job [2] (217.508 ms) 127.0.0.1 - - [18/Aug/2015:19:08:07 +0200] POST /api/tools HTTP/1.1 200 - http://127.0.0.1:8080/tool_runner?tool_id=SADIBUSYBOX; Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0 127.0.0.1 - - [18/Aug/2015:19:08:07 +0200] GET /api/histories/f2db41e1fa331b3e/contents HTTP/1.1 200 - http://127.0.0.1:8080/; Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0 galaxy.jobs DEBUG 2015-08-18 19:08:07,714 (2) Working directory for job is: /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2 galaxy.jobs.handler DEBUG 2015-08-18 19:08:07,721 (2) Dispatching to local runner galaxy.jobs DEBUG 2015-08-18 19:08:07,839 (2) Persisting job destination (destination id: docker_local) galaxy.jobs.runners DEBUG 2015-08-18 19:08:07,849 Job [2] queued (128.176 ms) galaxy.jobs.handler INFO 2015-08-18 19:08:07,897 (2) Job dispatched galaxy.jobs.command_factory INFO 2015-08-18 19:08:08,163 Built script [/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2/tool_script.sh] for tool command[/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2/tool_script.sh] galaxy.jobs.runners DEBUG 2015-08-18 19:08:08,322 (2) command is: sudo docker inspect mikeleganaaranguren/busybox-galaxy-test-io:v1 /dev/null 21 [ $? -ne 0 ] sudo docker pull mikeleganaaranguren/busybox-galaxy-test-io:v1 /dev/null 21 sudo docker run -e GALAXY_SLOTS=$GALAXY_SLOTS -v /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy:/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy:ro -v /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/tools/catDocker:/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/tools/catDocker:ro -v /home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2:/home/mikel/UPV-EHU/SADI-Docker-Galaxy/galaxy/database/job_working_directory/000/2:rw -v
