from:"Sven E. Templer"

Re: [galaxy-dev] Galaxy Slurm job_conf.xml

2015-10-17 Thread Sven E Templer

Hi Simon,

we use slurm via drmaa. Find configurations here:

https://github.com/mpg-age-bioinformatics/galaxy-admin/tree/master/server/config
https://github.com/mpg-age-bioinformatics/galaxy-admin/tree/master/system

Best,
Sven

> On 15 Oct 2015, at 08:38, Björn Grüning  wrote:
> 
> Hi Simon,
> 
> the Galaxy Docker Image is using Slurm you can look at it here:
> 
> https://github.com/bgruening/docker-galaxy-stable
> 
> Hope this helps,
> Bjoern
> 
> Am 15.10.2015 um 06:38 schrieb Simon Gladman:
>> Hi there all,
>> 
>> Has anyone had success setting up job_conf.xml for slurm based clusters? I
>> am having trouble getting the GALAXY_SLOTS env var set for any jobs. I've
>> tried looking at destinations in the job_conf.xml but can't get
>> SLURM_NTASKS to be set correctly so the slurm run script can translate that
>> into GALAXY_SLOTS..
>> 
>> Does anyone have a toy example of a job_conf.xml for slurm that they
>> wouldn't mind sharing?
>> 
>> Cheers,
>> 
>> Simon.
>> 
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>> 
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/
>> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] package_picard_1_136 installation error

2015-10-09 Thread Sven E. Templer

Hello list,

installing the dependency package_picard_1_136 results in the following
error:

---

  File
"/beegfs/common/galaxy/home/galaxy-15.07/lib/tool_shed/galaxy_install/install_manager.py",
line 142, in install_and_build_package_via_fabric
tool_dependency = self.install_and_build_package( tool_shed_repository,
tool_dependency, actions_dict )
  File
"/beegfs/common/galaxy/home/galaxy-15.07/lib/tool_shed/galaxy_install/install_manager.py",
line 100, in install_and_build_package
initial_download=True )
  File
"/beegfs/common/galaxy/home/galaxy-15.07/lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py",
line 32, in execute_step
initial_download=initial_download )
  File
"/beegfs/common/galaxy/home/galaxy-15.07/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py",
line 685, in execute_step
checksums = self.get_dict_checksums( action_dict )
  File
"/beegfs/common/galaxy/home/galaxy-15.07/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py",
line 223, in url_download
archive = CompressedFile( file_path )
  File
"/beegfs/common/galaxy/home/galaxy-15.07/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py",
line 89, in extract
print "foo"

[Errno 2] No such file or directory:
'./database/tmp/tmp-toolshed-mtdBC1_TE/picard-tools-1.136/picard-tools-1.136/'

---

Checking the directory
'./database/tmp/tmp-toolshed-mtdBC1_TE/picard-tools-1.136' returns:

$ ls -a ./database/tmp/tmp-toolshed-mtdBC1_TE/picard-tools-1.136
.  ..  htsjdk-1.136.jar  libIntelDeflater.so  picard-lib.jar  picard.jar

The same thing happens with 'package_picard_1_126_0', whereas
'package_picard_1_56_0' installs fine.
Is there an issue with '.' (in 126 and
136) vs 'picard-tools-1.56'?
I changed this in my local xml but it had no effect.

Any ideas?

Sven
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Sharing galaxy on CentOS7/Slurm/HPC/Proftp/Postgres/Apache/LDAP setup configurations

2015-10-07 Thread Sven E. Templer

Hello together,

not being an easy task to set up galaxy on a HPC cluster, satisfying user
and the IT department needs,
I would like to share our configuration (based on 15.07) to make it easier
for others to dive into this.

All configuration files and an installation manual is available at:
https://github.com/mpg-age-bioinformatics/galaxy-admin

Feedback is welcome,
Sven
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Sharing galaxy on CentOS7/Slurm/HPC/Proftp/Postgres/Apache/LDAP setup configurations

2015-10-07 Thread Sven E. Templer

Dear Hans-Rudolf,

thank you very much pointing to that ressource!
I will have a look into it and try to contribute, too.

Best,
Sven.

On 7 October 2015 at 10:16, Hans-Rudolf Hotz <h...@fmi.ch> wrote:

> Hi Sven
>
> Nice resource - thank you very much for sharing!
>
>
> Please allow me to suggest that you copy or link your documents to the
> 'Galaxy Community Log Board':
>
> https://wiki.galaxyproject.org/Community/Logs
>
>
> This makes sharing and especially finding your documents much easier
>
> Regards, Hans-Rudolf
>
>
>
>
>
> On 10/07/2015 09:57 AM, Sven E. Templer wrote:
>
>> Hello together,
>>
>> not being an easy task to set up galaxy on a HPC cluster, satisfying
>> user and the IT department needs,
>> I would like to share our configuration (based on 15.07) to make it
>> easier for others to dive into this.
>>
>> All configuration files and an installation manual is available at:
>> https://github.com/mpg-age-bioinformatics/galaxy-admin
>>
>> Feedback is welcome,
>> Sven
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>http://galaxyproject.org/search/mailinglists/
>>
>>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Running Galaxy on a cluster with SLURM ?

2015-09-23 Thread Sven E. Templer

Hi Carlos,

next steps might include:

* install slurm drmaa

http://apps.man.poznan.pl/trac/slurm-drmaa

e.g.

curl -#o slurm-drmaa-1.0.7.tar.gz
http://apps.man.poznan.pl/trac/slurm-drmaa/downloads/9
tar -xf slurm-drmaa-1.0.7.tar.gz
cd slurm-drmaa-1.0.7
p=$(which srun)
p=${p%/bin/srun}/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH="$p"
./configure #CFLAGS="-g -O0"
make
sudo make install
# rm -r slurm-drmaa*
# Make the config file writeable by group_galaxy:
sudo touch /etc/slurm_drmaa.conf
sudo cp /etc/slurm_drmaa.conf /etc/slurm_drmaa.conf.original
sudo chown :group_galaxy /etc/slurm_drmaa.conf
sudo chmod g+w /etc/slurm_drmaa.conf
# Test the drmaa-run binary by submitting a small job:
export DRMAA_LIBRARY_PATH=/usr/local/lib/libdrmaa.so
echo 'echo "Test executed on host $(hostname) by user $USER"' > test.drmaa
drmaa-run bash test.drmaa

* configure /etc/slurm_drmaa.conf [
https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#DRMAA], e.g.

job_categories: {
  default: "-J galaxy -p galaxy_partition",
},

* configure config/job_conf.xml [
https://wiki.galaxyproject.org/Admin/Config/Jobs], e.g.





/usr/local/lib/libdrmaa.so









  
  
--ntasks=12







  



* Have fun!

Sven

On 21 September 2015 at 21:14, Carlos Lijeron 
 wrote:

> Hello everyone,
>
> I¹m setting up Galaxy to run on our cluster which uses SLURM as the job
> manager.   I am wondering if any of you have any expertise in this area,
> ideas to share or lessons learned.   I carried out the following steps
> listed below, but I can¹t make the final connection just yet.  Any
> pointers will be greatly appreciated.
>
> 1. Setup Galaxy to run on the head node
> 2. Setup MySQL and it¹s running well with Galaxy
> 3. Setup the drmaa-python library inside our Galaxy folder
>
>
> Main questions:
>
> What would be the next step to take?
>
> Where to setup Galaxy to use an exclusive pool of resources (called
> partition in SLURM).
>
>
>
> Thank you all !
>
>
> Carlos.
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/



On 21 September 2015 at 21:14, Carlos Lijeron 
wrote:

> Hello everyone,
>
> I¹m setting up Galaxy to run on our cluster which uses SLURM as the job
> manager.   I am wondering if any of you have any expertise in this area,
> ideas to share or lessons learned.   I carried out the following steps
> listed below, but I can¹t make the final connection just yet.  Any
> pointers will be greatly appreciated.
>
> 1. Setup Galaxy to run on the head node
> 2. Setup MySQL and it¹s running well with Galaxy
> 3. Setup the drmaa-python library inside our Galaxy folder
>
>
> Main questions:
>
> What would be the next step to take?
>
> Where to setup Galaxy to use an exclusive pool of resources (called
> partition in SLURM).
>
>
>
> Thank you all !
>
>
> Carlos.
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Error when Installing package_galaxy_utils_1_0

2015-09-18 Thread Sven E. Templer

Dear Leslie,

I solved the problem by using a new python installation (we use the module
system to define PATH, PYTHONPATH, PYTHONUSERBASE and PYTHONHOME), without
any site libraries installed.
Before in the installation I had those packages (in
/software/Python/2.7.9/lib/python2.7/site-packages/) installed:

---

Bio  biopython-1.65-py2.7.egg-info
 matplotlibpathtools-0.1.2-py2.7.egg
 pyparsing.pysetuptools
BioSQL   curl
matplotlib-1.4.3-py2.7-nspkg.pth  pbr
   pyparsing.pyc   setuptools-18.3.1.dist-info
Cython-0.22-py2.7-linux-x86_64.egg   dateutil
matplotlib-1.4.3-py2.7.egg-info   pbr-1.8.0.dist-info
   pysam   singledispatch-3.4.0.3-py2.7.egg
DCC-0.3.1rc0-py2.7.egg
distribute-0.7.3-py2.7.egg-infomock
 pip  pysam-0.8.2.1-py2.7.egg-info
 six-1.9.0-py2.7.egg
DCC-0.3.2-py2.7.egg  easy-install.pth
mock-1.3.0.dist-info  pip-7.1.2.dist-info
   python_dateutil-2.4.2.dist-info sklearn
HTSeq-0.6.1p1-py2.7-linux-x86_64.egg easy_install.py
 mpl_toolkits  pkg_resources
 pytzwatchdog-0.8.3-py2.7.egg
MACS2easy_install.pyc
nose  poretools-0.5.1-py2.7.egg
   pytz-2015.4.dist-info   wget-2.2-py2.7.egg-info
MACS2-2.1.0.20140616-py2.7.egg-info  easy_install.pyo
nose-1.3.7.dist-info  pybedtools
rpy2-2.5.6-py2.7-linux-x86_64.egg   wget.py
MACS2-2.1.0.20150316-py2.7-linux-x86_64.egg  funcsigs
numpy
pybedtools-0.6.9-py2.7.egg-info  ruffus
 wget.pyc
PyYAML-3.11-py2.7-linux-x86_64.egg   funcsigs-0.4.dist-info
numpy-1.10.0.dev0_63e8bab-py2.7.egg-info  pycurl.so
   ruffus-2.6.3-py2.7.egg-info
README
h5py-2.4.0-py2.7-linux-x86_64.egg  numpy-1.9.2-py2.7.egg-info
 pylab.py scikit_learn-0.16.1-py2.7.egg-info
_markerlib   httplib2
pandaspylab.pyc
   scipy
argh-0.26.1-py2.7.egghttplib2-0.8-py2.7.egg-info
 pandas-0.16.0-py2.7.egg-info  pyparsing-2.0.3.dist-info
 scipy-0.15.1-py2.7.egg-info

---

I guess setuptools (v 18.3.1) is the cause of the incompatibility with the
distribute package that galaxy_sequencing_utils downloads.
Hope this helps!

Best,
Sven.

On 15 September 2015 at 10:20, Sven E. Templer <sven.temp...@gmail.com>
wrote:

> Hi Leslie,
>
> I face the same problem:
>
> ---
>
> Downloading
> http://pypi.python.org/packages/source/d/distribute/distribute-0.6.38.tar.gz
> Extracting in /tmp/tmplcqzT1
> Now working in /tmp/tmplcqzT1/distribute-0.6.38
> Building a Distribute egg in
> /beegfs/common/galaxy/home/server/database/tmp/tmp-toolshed-mtdqsavtj/galaxy_sequence_utils-1.0.0
> Traceback (most recent call last):
>   File "setup.py", line 248, in 
> scripts = scripts,
>   File "/software/Python/2.7.9/lib/python2.7/distutils/core.py", line 111,
> in setup
> _setup_distribution = dist = klass(attrs)
>   File "/tmp/tmplcqzT1/distribute-0.6.38/setuptools/dist.py", line 225, in
> __init__
> _Distribution.__init__(self,attrs)
>   File "/software/Python/2.7.9/lib/python2.7/distutils/dist.py", line 287,
> in __init__
> self.finalize_options()
>   File "/tmp/tmplcqzT1/distribute-0.6.38/setuptools/dist.py", line 257, in
> finalize_options
> ep.require(installer=self.fetch_build_egg)
>   File "/tmp/tmplcqzT1/distribute-0.6.38/pkg_resources.py", line 2027, in
> require
> working_set.resolve(self.dist.requires(self.extras),env,installer))
>   File "/tmp/tmplcqzT1/distribute-0.6.38/pkg_resources.py", line 2237, in
> requires
> dm = self._dep_map
>   File "/tmp/tmplcqzT1/distribute-0.6.38/pkg_resources.py", line 2466, in
> _dep_map
> self.__dep_map = self._compute_dependencies()
>   File "/tmp/tmplcqzT1/distribute-0.6.38/pkg_resources.py", line 2499, in
> _compute_dependencies
> common = frozenset(reqs_for_extra(None))
>   File "/tmp/tmplcqzT1/distribute-0.6.38/pkg_resources.py", line 2496, in
> reqs_for_extra
> if req.marker_fn(override={'extra':extra}):
>   File "/tmp/tmplcqzT1/distribute-0.6.38/_markerlib/markers.py", line 109,
> in marker_fn
> return eval(compiled_marker, environment)
>   File "", line 1, in 
> NameError: name 'sys_platform' is not defined
>
> /beegfs/common/galaxy/home/server/database/tmp/tmp-toolshed-mtdqsavtj/galaxy_sequence_utils-1.0.0/distribut

Re: [galaxy-dev] Error when Installing package_galaxy_utils_1_0

2015-09-15 Thread Sven E. Templer

Hi Leslie,

I face the same problem:

---

Downloading
http://pypi.python.org/packages/source/d/distribute/distribute-0.6.38.tar.gz
Extracting in /tmp/tmplcqzT1
Now working in /tmp/tmplcqzT1/distribute-0.6.38
Building a Distribute egg in
/beegfs/common/galaxy/home/server/database/tmp/tmp-toolshed-mtdqsavtj/galaxy_sequence_utils-1.0.0
Traceback (most recent call last):
  File "setup.py", line 248, in 
scripts = scripts,
  File "/software/Python/2.7.9/lib/python2.7/distutils/core.py", line 111,
in setup
_setup_distribution = dist = klass(attrs)
  File "/tmp/tmplcqzT1/distribute-0.6.38/setuptools/dist.py", line 225, in
__init__
_Distribution.__init__(self,attrs)
  File "/software/Python/2.7.9/lib/python2.7/distutils/dist.py", line 287,
in __init__
self.finalize_options()
  File "/tmp/tmplcqzT1/distribute-0.6.38/setuptools/dist.py", line 257, in
finalize_options
ep.require(installer=self.fetch_build_egg)
  File "/tmp/tmplcqzT1/distribute-0.6.38/pkg_resources.py", line 2027, in
require
working_set.resolve(self.dist.requires(self.extras),env,installer))
  File "/tmp/tmplcqzT1/distribute-0.6.38/pkg_resources.py", line 2237, in
requires
dm = self._dep_map
  File "/tmp/tmplcqzT1/distribute-0.6.38/pkg_resources.py", line 2466, in
_dep_map
self.__dep_map = self._compute_dependencies()
  File "/tmp/tmplcqzT1/distribute-0.6.38/pkg_resources.py", line 2499, in
_compute_dependencies
common = frozenset(reqs_for_extra(None))
  File "/tmp/tmplcqzT1/distribute-0.6.38/pkg_resources.py", line 2496, in
reqs_for_extra
if req.marker_fn(override={'extra':extra}):
  File "/tmp/tmplcqzT1/distribute-0.6.38/_markerlib/markers.py", line 109,
in marker_fn
return eval(compiled_marker, environment)
  File "", line 1, in 
NameError: name 'sys_platform' is not defined
/beegfs/common/galaxy/home/server/database/tmp/tmp-toolshed-mtdqsavtj/galaxy_sequence_utils-1.0.0/distribute-0.6.38-py2.7.egg
Traceback (most recent call last):
  File "setup.py", line 9, in 
use_setuptools()
  File
"/beegfs/common/galaxy/home/server/database/tmp/tmp-toolshed-mtdqsavtj/galaxy_sequence_utils-1.0.0/distribute_setup.py",
line 152, in use_setuptools
return _do_download(version, download_base, to_dir, download_delay)
  File
"/beegfs/common/galaxy/home/server/database/tmp/tmp-toolshed-mtdqsavtj/galaxy_sequence_utils-1.0.0/distribute_setup.py",
line 132, in _do_download
_build_egg(egg, tarball, to_dir)
  File
"/beegfs/common/galaxy/home/server/database/tmp/tmp-toolshed-mtdqsavtj/galaxy_sequence_utils-1.0.0/distribute_setup.py",
line 123, in _build_egg
raise IOError('Could not build the egg.')
IOError: Could not build the egg.

---

It might be an issue that distribute is not maintained any more (v0.6.38 is
used, 0.7.3 was latest version in 2013, then migrated to setuptools).
If you probably have updated setuptools (mine is version 18.3.1), it might
cause the error.

I tried to modify setup.py to use version 0.6.49 of distribute, but got the
same error.

Best,
Sven

On 10 September 2015 at 18:33, Leslie Herrera  wrote:

> Hello:
>
> My lab is currently in the process of setting Galaxy up on our server.  I
> have been installing the appropriate tools, but I am having difficulty
> installing the package_galaxy_utils_1_0.  The error message is below.  If
> this is something that is occuring on our end, would you please send
> instuctions on how to fix it?  If I have sent this email to wrong address
> or you need more information, please let me know.  Thank you very much for
> your help.
>
>
> Downloading
> http://pypi.python.org/packages/source/d/distribute/distribute-0.6.38.tar.gz
> Extracting in /tmp/tmp3dIIS_
> Now working in /tmp/tmp3dIIS_/distribute-0.6.38
>
> Building a Distribute egg in 
> /scratch/tmp-toolshed-mtdKjroZR/galaxy_sequence_utils-1.0.0
> Traceback (most recent call last):
>   File "setup.py", line 248, in 
> scripts = scripts,
>   File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
> _setup_distribution = dist = klass(attrs)
>
>   File "/tmp/tmp3dIIS_/distribute-0.6.38/setuptools/dist.py", line 225, in 
> __init__
> _Distribution.__init__(self,attrs)
>   File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
> self.finalize_options()
>
>   File "/tmp/tmp3dIIS_/distribute-0.6.38/setuptools/dist.py", line 257, in 
> finalize_options
> ep.require(installer=self.fetch_build_egg)
>
>   File "/tmp/tmp3dIIS_/distribute-0.6.38/pkg_resources.py", line 2027, in 
> require
> working_set.resolve(self.dist.requires(self.extras),env,installer))
>
>   File "/tmp/tmp3dIIS_/distribute-0.6.38/pkg_resources.py", line 2237, in 
> requires
> dm = self._dep_map
>
>   File "/tmp/tmp3dIIS_/distribute-0.6.38/pkg_resources.py", line 2466, in 
> _dep_map
> self.__dep_map = self._compute_dependencies()
>
>   File "/tmp/tmp3dIIS_/distribute-0.6.38/pkg_resources.py", line 2499, in 
> _compute_dependencies
> common =

Re: [galaxy-dev] Galaxy Slurm job_conf.xml

[galaxy-dev] package_picard_1_136 installation error

[galaxy-dev] Sharing galaxy on CentOS7/Slurm/HPC/Proftp/Postgres/Apache/LDAP setup configurations

Re: [galaxy-dev] Sharing galaxy on CentOS7/Slurm/HPC/Proftp/Postgres/Apache/LDAP setup configurations

Re: [galaxy-dev] Running Galaxy on a cluster with SLURM ?

Re: [galaxy-dev] Error when Installing package_galaxy_utils_1_0

Re: [galaxy-dev] Error when Installing package_galaxy_utils_1_0

7 matches

Site Navigation

Mail list logo

Footer information