The temp space challenge with train-model.perl is not only how much temp
space is needed, but also where are the temp files created?
train-model.perl seems unpredictable about where temp files are placed.
In general, the /tmp folder goes unused unless you specify it with the
"--temp-dir" option. Even when specified, only the sort functions use
the configuration it. Without setting the option,e sort temp files go to
the "--model-dir" folder, which by default is "--root-dir / model".
Finally, if not specified, --root-dir defaults to the current working
directory (i.e. cwd or ".").
That said, the temp folder for extraction step 3 is different. Its temp
files default to a newly-created "tmp" folder under the parent directory
of the path value set with "--extract-file". If "--extract-file" is not
set, the default is "--model-dir / extract", which also follows the
"--model-dir" path defined above.
Because of this complexity, we advise our customers to use hardware
configuration that relies on one large root folder partition... i.e. 1
or 2 TB mounted as "/" root without any other mounts. This gives Moses
one contiguous pool of temp storage independent of the configuration
complexities. I acknowledge this recommendation violates normal system
administrators' configurations, but without significant changes to
train-model.perl, this is our best recommendation.
On 02/27/2015 04:28 AM, Barry Haddow wrote:
Hi Alexander
From the error logs, it looks as though alignment went fine, the
training pipeline reports 24860460 lines of aligned bitext. Since the
extract files were empty, I'd suggest that extraction crashed, and the
most likely is that it ran out of disk. I'm not sure what happened to
the error messages.
For 25M sentence pairs, the final phrase table could easily be 30G and
the intermediate files are larger. You probably need more like 500G to
be safe.
I would follow Tom's advice and start with a much smaller corpus to
see how the process works. Also, for the full corpus, you could look
in to fast_align (https://github.com/clab/fast_align) for alignment as
it is much faster than mgiza (e.g. 2 days versus 2 weeks), and use EMS
for large jobs since it's much easier to restart a failed step.
cheers - Barry
On 26/02/15 15:06, Александр Паньшин wrote:
Hi Barry!
Here you can download training.out
https://www.dropbox.com/s/d0f0n99x4wbw3mo/training.out.gz?dl=1
I have about 50 Gb of free space in working dir.
2015-02-25 17:19 GMT+07:00 Barry Haddow <[email protected]
<mailto:[email protected]>>:
Hi Alexander,
It looks like something went wrong at the extract stage. If you
could make your training.out available then we can look for clues.
Could the system have run out of disk space, either in the
working directory or in /tmp? A lot of space is required to build
the extract files and phrase tables.
cheers - Barry
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
