Howdy, Andreas,
Lastz memory requirements are dependent on the size of the input
sequences (mainly on the size of reference sequence) and, to a lesser
extent, the genome's repeat content.
I'm a little confused/concerned by how this failure was indicated.
When run from a console, if lastz has a memory allocation failure, a
message is written to stderr (e.g. "call to realloc failed to allocate
1,000,000,000 bytes") and the program exits to the shell with the
status EXIT_FAILURE (an ISO C definition which I presume corresponds
to a standard shell error code).
Usually, if lastz isn't going to have enough memory, an allocation
failure will occur early in the run as all the long term data
structures are being built. This would normally be within the first
five minutes. A later failure would (probably) mean that the long
term structures were close to the memory limit and then alignments for
one of the query sequences required enough additional memory to push
us over the limit.
I assume galaxy's "Job did not produce output" message must be based
on a lack of any output to stdout. What is strange is that fact that
it took 2 days to get this message. Lack of output to stdout suggests
that the failure occurred before any queries were processed (strictly
speaking, before any alignments were output). This should have
occurred in the first few minutes of the run.
Would it be possible for you to point me at the input sequences for
this run, so that I can try running this via the console, and see if
there's something happening in lastz that I don't understand?
Bob H
On Nov 26, 2012, at 3:35 AM, Andreas Kuntzagk wrote:
Hi,
I just noticed that this was another case of a tool that needs more
then the 1GB memory that is default on our cluster. After adjusting
the job_runner settings everything seems fine.
regards, Andreas
On 15.11.2012 09:08, Andreas Kuntzagk wrote:
Hi,
when I start a lastz job it submits a job to GridEngine. But this
job seems to hang.
The process running is
python /data/galaxy/galaxy-dist/tools/sr_mapping/lastz_wrapper.py --
ref_source= ...
This wrapper seems to not start a lastz but lastz is in the path.
After 2 days then Galaxy shows "Job did not produce output"
Here is the paster.log from the start of the job.
galaxy.jobs DEBUG 2012-11-15 09:03:07,539 (254) Working directory
for job is:
/data/galaxy/galaxy-dist/database/job_working_directory/000/254
galaxy.jobs.handler DEBUG 2012-11-15 09:03:07,540 dispatching job
254 to drmaa runner
galaxy.jobs.handler INFO 2012-11-15 09:03:07,688 (254) Job dispatched
galaxy.tools DEBUG 2012-11-15 09:03:07,980 Building dependency
shell command for dependency 'lastz'
galaxy.tools WARNING 2012-11-15 09:03:07,981 Failed to resolve
dependency on 'lastz', ignoring
galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:08,669 (254)
submitting file
/data/galaxy/galaxy-dist/database/pbs/galaxy_254.sh
galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:08,670 (254)
command is: python
/data/galaxy/galaxy-dist/tools/sr_mapping/lastz_wrapper.py --
ref_source=cached
--source_select=pre_set --out_format=sam
--input2=/data/galaxy/galaxy-dist/database/files/027/
dataset_27795.dat
--input1="/data/galaxy/galaxy-dist/tool-data/shared/ucsc/hg18/seq/
hg18.2bit"
--ref_sequences="None" --pre_set_options=yasra98 --
identity_min=0
--identity_max=100 --coverage=0
--output=/data/galaxy/galaxy-dist/database/job_working_directory/
000/254/galaxy_dataset_27800.dat
--unmask=yes --lastzSeqsFileDir=/data/galaxy/galaxy-dist/
tool-data; cd
/data/galaxy/galaxy-dist; /data/galaxy/galaxy-dist/
set_metadata.sh ./database/files
/data/galaxy/galaxy-dist/database/job_working_directory/000/254 .
/data/galaxy/galaxy-dist/universe_wsgi.ini /data/galaxy/galaxy-dist/
database/tmp/tmpL4MMJV
/data/galaxy/galaxy-dist/database/job_working_directory/000/254/
galaxy.json
/data/galaxy/galaxy-dist/database/job_working_directory/000/254/
metadata_in_HistoryDatasetAssociation_288_9ynzEN,/data/galaxy/
galaxy-dist/database/job_working_directory/000/254/
metadata_kwds_HistoryDatasetAssociation_288_AEIseO,/data/galaxy/
galaxy-dist/database/job_working_directory/000/254/
metadata_out_HistoryDatasetAssociation_288_RhsufT,/data/galaxy/
galaxy-dist/database/job_working_directory/000/254/
metadata_results_HistoryDatasetAssociation_288_86DYsR,/data/galaxy/
galaxy-dist/database/job_working_directory/000/254/
galaxy_dataset_27800.dat,/data/galaxy/galaxy-dist/database/
job_working_directory/000/254/
metadata_override_HistoryDatasetAssociation_288_HMRNEG
galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:08,676 run as user
['kuntzagk', '600']
galaxy.jobs DEBUG 2012-11-15 09:03:08,676 (254) Changing ownership
of working directory with:
/usr/bin/sudo -E scripts/external_chown_script.py
/data/galaxy/galaxy-dist/database/job_working_directory/000/254
kuntzagk 600
galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:09,148 (254) Job
script for external submission is:
/data/galaxy/galaxy-dist/database/pbs/254.jt_json
141.80.188.178 - - [15/Nov/2012:09:03:10 +0200] "POST /galaxy/root/
history_item_updates HTTP/1.1"
200 - "http://bbc.mdc-berlin.de/galaxy/history"; "Mozilla/5.0 (X11;
Ubuntu; Linux x86_64; rv:16.0)
Gecko/20100101 Firefox/16.0"
galaxy.jobs.runners.drmaa INFO 2012-11-15 09:03:11,553 (254) queued
as 1282605
galaxy.jobs.runners.drmaa DEBUG 2012-11-15 09:03:11,767
(254/1282605) state change: job is running
--
Andreas Kuntzagk
SystemAdministrator
Berlin Institute for Medical Systems Biology at the
Max-Delbrueck-Center for Molecular Medicine
Robert-Roessle-Str. 10, 13125 Berlin, Germany
http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
