I have followed the instructions on how to setup a local cluster closely 
(http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster).  Frankly, the 
(Galaxy Configuration) section was not clear for me.  I am not sure if those 
outlined steps should be applied to the server's universe_wsgi.ini or to the 
nodes'?  So I might have overlooked some steps there but here is a summary of 
what I am doing at the server:

All my nodes have been configured so the galaxy user can ssh/scp between nodes 
without pwd.  Also the galaxy is a sudo in the galaxy (server).

Torque has been configured and tested between the nodes.  So the cluster is 
working fine.

In universal_wsgi.ini file at the server node






Pbs_application_server = galaxyhost (server)


Pbs_dataset_server= galaxyhost (server) ##This is the same like 

Also, ln -s /nfsexport/galaxy_stage /usr/local/galaxy/galaxy-dis/database/tmp

outputs_to_working_directory= False (if I changed this to True, the galaxy will 
not start)

After restarting the galaxy at the server node.  The job seems to be submitted 
and its status is "R".  When I "top" the processes on the node where the job 
was sent to, I see two processes; ssh and scp ran by the galaxy server.  This 
tells me something is being copied over to the node.  But I am not sure what 
and to where?

After while the job status changed to "W".

Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
68.ngsgalaxy01             ...x...@idtdna.com galaxy                 0 W batch

Here is what I say from the log when the job is sent.
galaxy.jobs DEBUG 2013-03-15 15:34:54,183 (341) Working directory for job is: 
galaxy.jobs.handler DEBUG 2013-03-15 15:34:54,183 dispatching job 341 to pbs 
galaxy.jobs.handler INFO 2013-03-15 15:34:54,231 (341) Job dispatched
galaxy.tools DEBUG 2013-03-15 15:34:54,309 Building dependency shell command 
for dependency 'samtools'
galaxy.jobs.runners.pbs DEBUG 2013-03-15 15:34:54,391 (341) submitting file 
galaxy.jobs.runners.pbs DEBUG 2013-03-15 15:34:54,391 (341) command is: 
PACKAGE_BASE=/usr/local/galaxy/software/samtools/0.1.16; export PACKAGE_BASE; . 
/usr/local/galaxy/software/samtools/0.1.16/env.sh; samtools flagstat 
"/usr/local/galaxy/galaxy-dist/database/files/000/dataset_319.dat" > 
galaxy.jobs.runners.pbs DEBUG 2013-03-15 15:34:54,394 (341) queued in default 
queue as 70.ngsgalaxy01.idtdna.com
galaxy.jobs.runners.pbs DEBUG 2013-03-15 15:34:54,966 
(341/70.ngsgalaxy01.idtdna.com) PBS job state changed from N to R

Here is the log when the ssh/scp on the node is finished.

galaxy.jobs.runners.pbs DEBUG 2013-03-15 15:37:00,815 
(341/70.ngsgalaxy01.idtdna.com) PBS job state changed from R to W

Here is the log when I qdel that job

galaxy.jobs.runners.pbs WARNING 2013-03-15 15:39:20,016 Exit code  was invalid. 
Using 0.
galaxy.jobs DEBUG 2013-03-15 15:39:20,033 (341) Changing ownership of working 
directory with: /usr/bin/sudo -E scripts/external_chown_script.py 
/usr/local/galaxy/galaxy-dist/database/job_working_directory/000/341 galaxy 
galaxy.jobs ERROR 2013-03-15 15:39:20,071 (341) Failed to change ownership of 
/usr/local/galaxy/galaxy-dist/database/job_working_directory/000/341, failing
Traceback (most recent call last):
  File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 336, 
in finish
  File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 909, 
in reclaim_ownership
    self._change_ownership( self.galaxy_system_pwent[0], str( 
self.galaxy_system_pwent[3] ) )
  File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 895, 
in _change_ownership
    assert p.returncode == 0
galaxy.datatypes.metadata DEBUG 2013-03-15 15:39:20,160 Cleaning up external 
metadata files
galaxy.jobs.runners.pbs WARNING 2013-03-15 15:39:20,172 Unable to cleanup: 
[Errno 2] No such file or directory: 
galaxy.jobs.runners.pbs WARNING 2013-03-15 15:39:20,173 Unable to cleanup: 
[Errno 2] No such file or directory: 
galaxy.jobs.runners.pbs WARNING 2013-03-15 15:39:20,173 Unable to cleanup: 
[Errno 2] No such file or directory: 
'/usr/local/galaxy/galaxy-dist/database/pbs/341.ec' - - [15/Mar/2013:15:39:22 -0500] "GET 
/api/histories/5a1cff6882ddb5b2 HTTP/1.0" 200 - ""; 
"Mozilla/5.0 (Windows NT 5.2; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0" - - [15/Mar/2013:15:39:22 -0500] "GET 
/api/histories/5a1cff6882ddb5b2/contents?ids=bbbfa414ae315caf HTTP/1.0" 200 - 
""; "Mozilla/5.0 (Windows NT 5.2; WOW64; rv:18.0) 
Gecko/20100101 Firefox/18.0"

Is there anything I am not doing or doing wrong?


Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:


Reply via email to