[galaxy-dev] new datasets appear in database/files/000/

[Jochen Bick]

Hi,

today we found out that some of our files in your galaxy instance are
new in "database/files/000/" which is the first folder of your database
files. Could some please explain me how galaxy generates new file names?
and how this is possible?

Thanks in advance Jochen

-- 
ETH Zurich
*Jochen Bick*
Animal Physiology
Institute of Agricultural Sciences
Postal address: Universitätstrasse 2 / LFW B 58.1
8092 Zurich, Switzerland
Office: Eschikon 27
8315 Lindau, Switzerland

Phone +41 52 354 92 06
jochen.b...@usys.ethz.ch 
www.ap.ethz.ch

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Re: Galaxy jobs on Apache Mesos with chronos.py runner

[John Chilton]

Hi Pietro,

Thanks for trying that runner!

Some background here regarding the volume directory issue:
https://github.com/galaxyproject/galaxy/pull/3946
https://github.com/galaxyproject/galaxy/pull/3946/files/2099c09f6ab5a8f5951d01cd6fa67681618e2dda#r112457231

It looks to be a known and documented limitation of this runner that
it only allows one directory to be mounted in.

I don't think the devteam really maintains this runner or has any
throughput to - so you'll probably have to patch the runner to support
multiple volumes if you need it to work :(. I assume the underlying
API it leverages would allow that. The Kubernetes runner supports
multiple volumes for instance. If you figure that out, we'd love a PR
though.

Sorry I don't have better news and can't be more helpful.

-John

On Tue, May 14, 2019 at 8:56 AM Pietro Mandreoli
 wrote:
>
>
> Dear experts,
>
> I’m trying to run Galaxy (version 18.05) jobs with Apache MESOS (version 
> 1.5.0). I configured a NFS between galaxy and mesos cluster nodes, thus 
> sharing the directories /home/galaxy and /path/to/galaxy/database with the 
> directories (citations, compiled_templates, files, ftp, home, 
> job_working_directory, object_store_cache, tmp).
>
> Then I have changed the job_conf.xml accordingly: ( 
> https://gist.github.com/pmandreoli/6ffba03193717393a2322586686f9aed ).
>
> It works fine with my very simple test wrapper here:  
> https://gist.github.com/pmandreoli/ce120612afd0ac9ee80ce70c90e7d324
>
>
> Then I enabled mulled containers in the galaxy.yml file in order to test the 
> configuration using fastQC (version 0.72, owner devteam).
>
> In this case the job was correctly executed on the mesos node (using the 
> container quay.io/biocontainers/fastqc:0.11.8--1) but the results were not 
> linked in the history (please see the attached fig1).
>
>
>
> [fig1. history panel screenshot for fastQC job on chronos destination]
>
>
> You can have a look to the output here:
>
> https://gist.github.com/pmandreoli/bbbeb2eab5c1d1772872220c01678e15
>
> I checked the docker run command on chronos and the working directory is 
> located on /root/working.
>
>
> In order to check if the problems is related to my job_conf.xml configuration 
> and not to the mulled containers i changed the job_conf.xml to run jobs 
> locally ( https://gist.github.com/pmandreoli/484566b2c548d39d8bddb5aa54461ecc 
> )
>
> and i have run the same tool (fastQC 0.72). In this case everything was fine.
>
>
> Is my job_conf.xml configuration correct?
>
>
> I would also like to ask if it is possible to add more than one volume to the 
> docker container run on MESOS in order to add the location for the reference 
> data, located on my mesos slave node in /cvmfs. I tried to modify the 
> job_conf.xml block in this way
>
> /export/galaxy/database/,/cvmfs/
>
> but the job failed. The docker run command sent to chronos was indeed wrong: 
> “-v /export/galaxy/database/,/cvmfs/:/export/galaxy/database/,/cvmfs/:rw “.
>
> This is probably due to the definition of the “volumes” field in the 
> chronos.py runner:  
> https://github.com/galaxyproject/galaxy/blob/3b3b52f013ac8c6b5bf8a4765f9fe9c50d3cebbb/lib/galaxy/jobs/runners/chronos.py#L103
>
> which, if I understand well, is allowing to use only one path. Did I miss 
> something?
>
>
> Any suggestion and correction is, of course, more than welcome.
>
>
> Best regards,
>
>
> Pietro Mandreoli
>
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Galaxy jobs on Apache Mesos with chronos.py runner

[Pietro Mandreoli]

Dear experts, 

I’m trying to run Galaxy (version 18.05) jobs with Apache MESOS (version 
1.5.0). I configured a NFS between galaxy and mesos cluster nodes, thus sharing 
the directories /home/galaxy and /path/to/galaxy/database with the directories 
(citations, compiled_templates, files, ftp, home, job_working_directory, 
object_store_cache, tmp). 

Then I have changed the job_conf.xml accordingly: ( 
https://gist.github.com/pmandreoli/6ffba03193717393a2322586686f9aed ). 

It works fine with my very simple test wrapper here:  
https://gist.github.com/pmandreoli/ce120612afd0ac9ee80ce70c90e7d324   


Then I enabled mulled containers in the galaxy.yml file in order to test the 
configuration using fastQC (version 0.72, owner devteam). 

In this case the job was correctly executed on the mesos node (using the 
container quay.io/biocontainers/fastqc:0.11.8--1) but the results were not 
linked in the history (please see the attached fig1). 

 

[fig1. history panel screenshot for fastQC job on chronos destination] 


You can have a look to the output here: 

https://gist.github.com/pmandreoli/bbbeb2eab5c1d1772872220c01678e15   

I checked the docker run command on chronos and the working directory is 
located on /root/working.   


In order to check if the problems is related to my job_conf.xml configuration 
and not to the mulled containers i changed the job_conf.xml to run jobs locally 
( https://gist.github.com/pmandreoli/484566b2c548d39d8bddb5aa54461ecc ) 

and i have run the same tool (fastQC 0.72). In this case everything was fine. 


Is my job_conf.xml configuration correct? 


I would also like to ask if it is possible to add more than one volume to the 
docker container run on MESOS in order to add the location for the reference 
data, located on my mesos slave node in /cvmfs. I tried to modify the 
job_conf.xml block in this way 

/export/galaxy/database/,/cvmfs/ 

but the job failed. The docker run command sent to chronos was indeed wrong: 
“-v /export/galaxy/database/,/cvmfs/:/export/galaxy/database/,/cvmfs/:rw “. 

This is probably due to the definition of the “volumes” field in the chronos.py 
runner:  
https://github.com/galaxyproject/galaxy/blob/3b3b52f013ac8c6b5bf8a4765f9fe9c50d3cebbb/lib/galaxy/jobs/runners/chronos.py#L103
 

which, if I understand well, is allowing to use only one path. Did I miss 
something? 


Any suggestion and correction is, of course, more than welcome. 


Best regards, 


Pietro Mandreoli 


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/