Dear Marco,
Sun Grid Engine has quite a few versions that differ in capabilities and
default
options, so succeeding with parallel training will take a while of
experimenting
and looking much deeper into the (mess of) scripts to match it with your
particular environment. The basic schema is:
1. mert-moses.pl calls qsubwrapper.pl to start a command (e.g.
filter-model-fiven-input.pl)
2. qsubwrapper.pl creates a temporary bash/csh script (containing a call to the
command) and calls qsub to run the script on the grid.
3. someone (qsub or qsubwrapper or mert-moses.pl) has to wait until all jobs are
finished.
4. maybe someone (qsubwrapper or the script itself) might delete the temporary
script just to make debugging harder ;-)
(I think there is one more layer if moses is the process to be launched:
mert-moses.pl -> moses-parallel -> qsubwrapper -> qsub -> bash/csh script ->
moses, but this might have been the case in an older version.)
With so many steps, anything can go wrong. Given the languages used and number
of people who wrote bits and pieces of the process, failure detection alone is
hard to achieve.
Start with learning how to use the grid manually on command line, and how to
diagnose if a job was actually launched. (qstat will show you jobs running, add
your own debugging prints to the scripts to learn where it got and where it
failed).
One possible reason is that in your particular setup 'qsub' is nonblocking (or
ignores the '-sync' option available in SGE 6+). qsubwrapper might try to
provide a workaround using an additional dummy job marked as depending on the
jobs submitted, but this is maybe not switched on, or it fails.
I can only wish you good luck with debugging, and warn you to expect, e.g.:
- race conditions caused by NFS; e.g. that you'll already see the dummy waiting
job to produce its confirming timestamp file but you'll still not see the
outputs of the jobs that finished before; or you will see many 'zero's in the
output files, because NFS has not written the data yet
- funny behaviour of SGE that will run your script in e.g. csh instead of bash
(sometimes even if you say you want bash), so that redirection syntax will be
misunderstood (csh does not usen 2> to redirect stderr).
Best, Ondrej.
marco turchi wrote:
> Dear experts,
> I'm trying to run the mert-moses script on a Sun Grid... I have added
> the flag --job.
> The scripts submits the jobs, but then I get this error:
>
> Executing: mkdir -p Dev
> Executing: /usr/local/moses/moses/scripts//generic/qsub- wrapper.pl
> -command='/usr/local/moses/moses/scripts//training/filter-model-given-input.pl
>
> ./filtered
> /exports/gpfs/enxmt/Moses/WorkingFolder/testFolder/0/TM/model/moses.ini
> /exports/gpfs/enxmt/Moses/DevCorpus/dev2006.Tok.Lower.es'
> -queue-parameter="-l mem_free= 0.5G -hard" -stdout=filterphrases.out
> -stderr=filterphrases.err
> filtering the phrase tables... Thu Nov 22 15:42:22 GMT 2007
> which: no pawd in
> (/usr/local/moses/scripts:/usr/local/moses/moses/misc/:/usr/local/moses/moses/scripts/training:/usr/local/moses/bin:/usr/local/moses/srilm-src/bin/i686-m64:/usr/local/moses/moses/scripts:/usr/local/ansys_inc/shared_files/licensing/lic_admin:/usr/local/ansys_inc/v110/ansys/bin:/opt/clearspeed/csx600_m512_le/bin:/usr/local/lib64/R/bin:/usr/local/grace/bin:/usr/local/matlab74/bin:/usr/local/maple11/Maple_Network_Tools/FLEXlm:/usr/local/maple11/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/molpro-serial/bin:.:/usr/local/Cluster-Apps/torque/2.1.7//bin:/usr/local/Cluster-Apps/torque/2.1.7//sbin:/usr/local/Cluster-Apps/maui/3.2.6p19//bin:/usr/local/Cluster-Apps/maui/3.2.6p19//sbin:/usr/local/Cluster-Apps/cluster-tools/bin:/usr/local/Cluster-Apps/cluster-tools/sbin:/usr/local/Cluster-Apps/installer-tools/sbin:/exports/gpfs/enxmt/bin)
>
>
> Executing: mkdir -p
> /exports/gpfs/enxmt/Moses/WorkingFolder/testFolder/0/Dev/tmp22028
> Executing: qsub -V -o
> /exports/gpfs/enxmt/Moses/WorkingFolder/testFolder/0/Dev/out.job22028 -e
> /exports/gpfs/enxmt/Moses/WorkingFolder/testFolder/0/Dev/err.job22028 -N
> WR22028
> /exports/gpfs/enxmt/Moses/WorkingFolder/testFolder/0/Dev/job22028.csh >&
> /exports/gpfs/enxmt/Moses/WorkingFolder/testFolder/0/Dev/job22028.log
> check_exit_status of submitted job
> Can't read
> /exports/gpfs/enxmt/Moses/WorkingFolder/testFolder/0/Dev/out.job22028 at
> /usr/local/moses/moses/scripts//generic/qsub-wrapper.pl line 198.
> Exit code: 2
> Failed to submit filtering of tables to the queue (via
> /usr/local/moses/moses/scripts//generic/qsub- wrapper.pl) at
> /usr/local/moses/moses/scripts/training/mert-moses.pl line 459.
>
> I do not know too much about parallel machines, but it seems that the
> single job runs but it does not do anything...
> the files err.job22028 and filterphrases.err do not contain any
> particular errors.
>
> thanks
> Marco
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
--
Ondrej Bojar (mailto:[EMAIL PROTECTED] / [EMAIL PROTECTED])
http://www.cuni.cz/~obo
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
