Hello Devs,

Now that the galaxy process has crashed, I'll send you more information
regarding the logs.

galaxy.jobs.runners.local DEBUG 2015-08-04 17:13:20,303 (2337) executing
job script:
/Users/galaxy/galaxy-dist/database/job_working_directory/002/2337/galaxy_2337.sh
galaxy.jobs DEBUG 2015-08-04 17:13:20,487 (2337) Persisting job destination
(destination id: multicore6)
galaxy.jobs DEBUG 2015-08-04 17:13:25,181 (2468) Working directory for job
is: /Users/galaxy/galaxy-dist/database/job_working_directory/002/2468
galaxy.jobs.handler DEBUG 2015-08-04 17:13:25,190 (2468) Dispatching to
local runner
galaxy.jobs DEBUG 2015-08-04 17:13:25,466 (2468) Persisting job destination
(destination id: local)
galaxy.jobs.runners DEBUG 2015-08-04 17:13:25,485 Job [2468] queued
(295.181 ms)
galaxy.jobs.handler INFO 2015-08-04 17:13:25,490 (2468) Job dispatched
galaxy.jobs DEBUG 2015-08-04 17:13:25,538 (2469) Working directory for job
is: /Users/galaxy/galaxy-dist/database/job_working_directory/002/2469
galaxy.jobs.handler DEBUG 2015-08-04 17:13:25,545 (2469) Dispatching to
local runner
galaxy.jobs DEBUG 2015-08-04 17:13:25,777 (2469) Persisting job destination
(destination id: local)
galaxy.jobs.runners DEBUG 2015-08-04 17:13:25,798 Job [2469] queued
(252.823 ms)
galaxy.jobs.handler INFO 2015-08-04 17:13:25,803 (2469) Job dispatched
galaxy.jobs.runners.local DEBUG 2015-08-04 17:14:09,922 execution finished:
/Users/galaxy/galaxy-dist/database/job_working_directory/002/2337/galaxy_2337.sh
galaxy.datatypes.metadata DEBUG 2015-08-04 17:14:10,476 loading metadata
from file for: HistoryDatasetAssociation 5117
galaxy.jobs.runners.local DEBUG 2015-08-04 17:14:10,589 execution finished:
/Users/galaxy/galaxy-dist/database/job_working_directory/002/2318/galaxy_2318.sh
galaxy.datatypes.metadata DEBUG 2015-08-04 17:14:10,919 loading metadata
from file for: HistoryDatasetAssociation 5116
galaxy.datatypes.metadata DEBUG 2015-08-04 17:14:11,500 loading metadata
from file for: HistoryDatasetAssociation 5115
galaxy.datatypes.metadata DEBUG 2015-08-04 17:14:11,519 loading metadata
from file for: HistoryDatasetAssociation 5084
galaxy.datatypes.metadata DEBUG 2015-08-04 17:14:12,270 loading metadata
from file for: HistoryDatasetAssociation 5083
galaxy.datatypes.metadata DEBUG 2015-08-04 17:14:12,804 loading metadata
from file for: HistoryDatasetAssociation 5082
galaxy.jobs INFO 2015-08-04 17:14:13,406 Collecting job metrics for
<galaxy.model.Job object at 0x117e5b4d0>
galaxy.jobs DEBUG 2015-08-04 17:14:13,537 job 2337 ended (finish() executed
in (3614.250 ms))
galaxy.datatypes.metadata DEBUG 2015-08-04 17:14:13,580 Cleaning up
external metadata files
galaxy.jobs INFO 2015-08-04 17:14:14,001 Collecting job metrics for
<galaxy.model.Job object at 0x11b03b410>
galaxy.jobs DEBUG 2015-08-04 17:14:14,068 job 2318 ended (finish() executed
in (3449.205 ms))
galaxy.datatypes.metadata DEBUG 2015-08-04 17:14:14,112 Cleaning up
external metadata files
galaxy.jobs DEBUG 2015-08-04 17:14:19,625 (2319) Working directory for job
is: /Users/galaxy/galaxy-dist/database/job_working_directory/002/2319
galaxy.jobs.handler DEBUG 2015-08-04 17:14:19,630 (2319) Dispatching to
local runner
galaxy.jobs DEBUG 2015-08-04 17:14:20,000 (2319) Persisting job destination
(destination id: local)
galaxy.jobs.runners DEBUG 2015-08-04 17:14:20,018 Job [2319] queued
(387.377 ms)
galaxy.jobs.handler INFO 2015-08-04 17:14:20,021 (2319) Job dispatched
galaxy.jobs DEBUG 2015-08-04 17:14:20,061 (2320) Working directory for job
is: /Users/galaxy/galaxy-dist/database/job_working_directory/002/2320
galaxy.jobs.handler DEBUG 2015-08-04 17:14:20,067 (2320) Dispatching to
local runner
galaxy.jobs DEBUG 2015-08-04 17:14:20,254 (2320) Persisting job destination
(destination id: local)
galaxy.jobs.runners DEBUG 2015-08-04 17:14:20,268 Job [2320] queued
(201.349 ms)
galaxy.jobs.handler INFO 2015-08-04 17:14:20,272 (2320) Job dispatched
galaxy.jobs DEBUG 2015-08-04 17:14:20,696 (2338) Working directory for job
is: /Users/galaxy/galaxy-dist/database/job_working_directory/002/2338
galaxy.jobs.handler DEBUG 2015-08-04 17:14:20,701 (2338) Dispatching to
local runner
galaxy.jobs DEBUG 2015-08-04 17:14:20,891 (2338) Persisting job destination
(destination id: local)
galaxy.jobs.runners DEBUG 2015-08-04 17:14:20,941 Job [2338] queued
(239.991 ms)
galaxy.jobs.handler INFO 2015-08-04 17:14:20,945 (2338) Job dispatched
galaxy.jobs DEBUG 2015-08-04 17:14:20,982 (2339) Working directory for job
is: /Users/galaxy/galaxy-dist/database/job_working_directory/002/2339
galaxy.jobs.handler DEBUG 2015-08-04 17:14:20,989 (2339) Dispatching to
local runner
galaxy.jobs DEBUG 2015-08-04 17:14:21,456 (2339) Persisting job destination
(destination id: local)
galaxy.jobs.runners DEBUG 2015-08-04 17:14:21,468 Job [2339] queued
(478.726 ms)
galaxy.jobs.handler INFO 2015-08-04 17:14:21,472 (2339) Job dispatched
ERROR LINE: run.sh: line 81: 21118 Abort trap: 6           python
./scripts/paster.py serve $GALAXY_CONFIG_FILE $@




When trying to restart run.sh, this is what we see in the log.
Starting server in PID 57249.
Traceback (most recent call last):
  File "./scripts/paster.py", line 37, in <module>
    serve.run()
  File "/Users/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py",
line 1049, in run
    invoke(command, command_name, options, args[1:])
  File "/Users/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py",
line 1055, in invoke
    exit_code = runner.run(args)
  File "/Users/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py",
line 220, in run
    result = self.command()
  File "/Users/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py",
line 670, in command
    serve()
  File "/Users/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py",
line 654, in serve
    server(app)
  File "/Users/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py",
line 292, in server_wrapper
    **context.local_conf)
  File "/Users/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py",
line 97, in fix_call
    val = callable(*args, **kw)
  File
"/Users/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpserver.py",
line 1342, in server_runner
    serve(wsgi_app, **kwargs)
  File
"/Users/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpserver.py",
line 1291, in serve
    request_queue_size=request_queue_size)
  File
"/Users/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpserver.py",
line 1134, in __init__
    request_queue_size=request_queue_size)
  File
"/Users/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpserver.py",
line 1113, in __init__
    request_queue_size=request_queue_size)
  File
"/Users/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpserver.py",
line 360, in __init__
    HTTPServer.__init__(self, server_address, RequestHandlerClass)
  File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py",
line 408, in __init__
    self.server_bind()
  File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py",
line 108, in server_bind
    SocketServer.TCPServer.server_bind(self)
  File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py",
line 419, in server_bind
    self.socket.bind(self.server_address)
  File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py",
line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 48] Address already in use
galaxy.jobs.handler INFO 2015-08-04 17:19:32,844 sending stop signal to
worker thread
galaxy.jobs.handler INFO 2015-08-04 17:19:32,845 job handler queue stopped
galaxy.jobs.runners INFO 2015-08-04 17:19:32,845 LocalRunner: Sending stop
signal to 2 worker threads
galaxy.jobs.runners INFO 2015-08-04 17:19:32,857 LocalRunner: Sending stop
signal to 8 worker threads
galaxy.jobs.handler INFO 2015-08-04 17:19:32,899 sending stop signal to
worker thread
galaxy.jobs.handler INFO 2015-08-04 17:19:32,912 job handler stop queue
stopped

After we kill the running galaxy tool processes, we are able to
successfully start run.sh again.  We migrated from a previous version, so
could this be caused by outdated eggs or .xml files?

Thank you,
-Hans


On Tue, Aug 4, 2015 at 2:03 PM, Hans Vasquez-Gross <
havasquezgr...@ucdavis.edu> wrote:

> Hi Bjorn and Hans,
>
> We are running Galaxy on our local webserver, so there is no job
> scheduler.  Instead, we are using the localJobRunner configuration in the
> job_conf.xml
>
> <?xml version="1.0"?>
> <job_conf>
>     <plugins>
>         <plugin id="local" type="runner"
> load="galaxy.jobs.runners.local:LocalJobRunner" workers="8"/>
>         <plugin id="multilocal" type="runner"
> load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/>
>     </plugins>
>     <destinations default="local">
>         <destination id="local" runner="local"/>
>         <destination id="multicore6" runner="multilocal">
>           <param id="local_slots">6</param>
>         </destination>
>     </destinations>
>     <tools>
>     <tool id="bowtie2" destination="multicore6" />
>     <tool id="spades"  destination="multicore6" />
>     <tool id="bbmap_1"  destination="multicore6" />
>     <tool id="iuc_pear"  destination="multicore6" />
>     <tool id="abyss-pe"  destination="multicore6" />
>     <tool id="fastq_groomer_parallel"  destination="multicore6" />
>     </tools>
>
>     <handlers>
>         <handler id="main"/>
>     </handlers>
> </job_conf>
>
> Also, normally, we run galaxy in the daemon mode, but recently to help
> debug this issue, we have been running galaxy in interactive mode in a
> screen session.
>
> @Hans - The processes run under the galaxy user and everything seems to
> run fine.  I am trying to get a concrete example, but run.sh usually
> crashes during our trimming/assembly steps for NGS data.  Sometimes these
> workflows run to completion, but sometimes they crash the run.sh process.
> When run.sh crashes, the individual running programs keep running as the
> galaxy user.  We cannot restart galaxy until we manually kill those running
> processes.  Here are what the running processes look like on the system.
>
> galaxy         21162  99.4  0.0  2432784    616 s014  R     2:08PM
> 1425:10.57 seqtk seq -q 0 -X 255 -l 0 -Q 33 -s 11 -f 1.0 -L 0 -1
> /Users/galaxy/data_galaxy/test_BACs/all-merged.interleaved.fq
> galaxy         21159  99.4  0.0  2432784    616 s014  R     2:08PM
> 1425:16.72 seqtk seq -q 0 -X 255 -l 0 -Q 33 -s 11 -f 1.0 -L 0 -2
> /Users/galaxy/data_galaxy/test_BACs/all-merged.interleaved.fq
> galaxy         21118  92.5  0.5  3015188 367852 s014  R+    2:07PM
> 791:21.54 python ./scripts/paster.py serve universe_wsgi.ini
> galaxy         21160   0.0  0.0  2433640   1044 s014  S     2:08PM
> 0:00.01 /bin/sh
> /Users/galaxy/galaxy-dist/database/job_working_directory/002/2180/galaxy_2180.sh
> galaxy         21157   0.0  0.0  2433640   1044 s014  S     2:08PM
> 0:00.01 /bin/sh
> /Users/galaxy/galaxy-dist/database/job_working_directory/002/2179/galaxy_2179.sh
> galaxy         21113   0.0  0.0  2433640   1000 s014  S+    2:07PM
> 0:00.00 sh run.sh
>
>
> Thank you for the help,
> -Hans
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to