Nothing comes to mind immediately, can you post a link to your
job_conf.xml file?

What is your `cleanup_job` option set to in universe_wsgi.ini? Can you
try changing it to 'never' and see if the jobs continue?

-John

On Tue, Aug 5, 2014 at 5:54 PM, Langhorst, Brad <langho...@neb.com> wrote:
> I’m not aware of any recent changes to the configuration of galaxy…
>
> It seems that jobs are not being distributed at the moment.
>
>
> I have 2 job handlers...
>
> It seems that when I restart them only a single job is queued, it has an
> error  and then sticks on "Cleaning up external metadata files” forever.
>
> any ideas?
> I have 700+ jobs in the queue and would like to get them running asap...
>
>
> one hanlder just stays at
> Starting server in PID 4868.
> serving on http://127.0.0.1:8090
>
> here’s the log from the other.
>
> Starting server in PID 4873.
> serving on http://127.0.0.1:8091
> galaxy.jobs DEBUG 2014-08-05 17:34:08,388 (125206) Working directory for job
> is: /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206
> galaxy.jobs.handler DEBUG 2014-08-05 17:34:08,410 (125206) Dispatching to
> gridengine runner
> galaxy.jobs DEBUG 2014-08-05 17:34:08,479 (125206) Persisting job
> destination (destination id: gridengine)
> galaxy.jobs.handler INFO 2014-08-05 17:34:08,498 (125206) Job dispatched
> galaxy.tools.deps DEBUG 2014-08-05 17:34:08,862 Building dependency shell
> command for dependency 'bedtools2'
> galaxy.tools.deps WARNING 2014-08-05 17:34:08,865 Failed to resolve
> dependency on 'bedtools2', ignoring
> galaxy.jobs.runners DEBUG 2014-08-05 17:34:08,966 (125206) command is:
> bedtools --version >
> /mnt/galaxy/data/galaxy/tmp/GALAXY_VERSION_STRING_125206 2>&1; bedtools
> intersect -v -a /mnt/galaxy/data/galaxy/user-data/000/170/dataset_170995.dat
> -b /mnt/galaxy/data/galaxy/user-data/000/169/dataset_169046.dat >
> "/mnt/galaxy/data/galaxy/user-data/000/170/dataset_170999.dat";
> return_code=$?; cd /mnt/galaxy/data/galaxy/galaxy-dist;
> /mnt/galaxy/data/galaxy/galaxy-dist/set_metadata.sh
> /mnt/galaxy/data/galaxy/user-data
> /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206 .
> /mnt/galaxy/data/galaxy/galaxy-dist/universe_wsgi.ini
> /mnt/galaxy/data/galaxy/tmp/tmpLCBPvF
> /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/galaxy.json
> /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_in_HistoryDatasetAssociation_194826_qIy33T,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_kwds_HistoryDatasetAssociation_194826_SJhToq,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_out_HistoryDatasetAssociation_194826_4Lf956,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_results_HistoryDatasetAssociation_194826_YzEZK8,,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_override_HistoryDatasetAssociation_194826_19nZAm;
> sh -c "exit $return_code"
> galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:08,994 (125206) submitting
> file
> /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/galaxy_125206.sh
> galaxy.jobs.runners.drmaa INFO 2014-08-05 17:34:09,023 (125206) queued as
> 1539118
> galaxy.jobs DEBUG 2014-08-05 17:34:09,048 (125206) Persisting job
> destination (destination id: gridengine)
> galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:09,451 (125206/1539118)
> state change: job is queued and active
> galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:11,510 (125206/1539118)
> state change: job is running
> galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:31,208 (125206/1539118)
> state change: job finished normally
> galaxy.jobs DEBUG 2014-08-05 17:34:31,440 setting dataset state to ERROR
> galaxy.jobs INFO 2014-08-05 17:34:31,652 Collecting job metrics for
> <galaxy.model.Job object at 0x7f42e030f850>
> galaxy.jobs DEBUG 2014-08-05 17:34:31,664 job 125206 ended
> galaxy.datatypes.metadata DEBUG 2014-08-05 17:34:31,665 Cleaning up external
> metadata files
>
>
> --
> Bradley W. Langhorst, Ph.D.
> Applications and Product Development Scientist
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to