Chaitu chaitu,
You can reduce the in-memory size of the phrase table and language
model by binarizing them so they are read directly from disk. This
slows translation time slightly but takes much less memory.
If you know ahead of time what test set you are translating, you can
also filter the phrase table so it contains only the source phrases
present in your input.
Check out the "Binary Phrase Tables with On-demand loading" and
"Efficient Language Model Handling" sections on this web page: http://www.statmt.org/moses/?n=Moses.AdvancedFeatures
The "filter-model-given-input.pl" and "filter-and-binarize-model-given-
input.pl" scripts included in the moses scripts directory help with
filtering and binarizing the phrase table. There's an example of
calling filter-model-given-input in the step-by-step instructions for
building a Europarl system here: http://www.statmt.org/wmt08/baseline.html
Hope this helps,
-Josh
On 21 Feb 2008, at 20:41, chaitu chaitu wrote:
Hello all,
While running the moses for testing, I used the following command
and got the following output.
The problem is that Before the translation the machine runs out of
memory.
I have the following file sizes for phrase table and europarl.lm:
phrase-table.0-0.gz is about 797M (or 835919639 bytes) and
working-dir/lm/europarl.lm is about 250M.
I am using a machine with 4G of RAM
I am not sure what I can do now?
This is what I have done to run moses
moses -f working-dir/model/moses.ini < working-dir/model/in
Defined parameters (per moses.ini or switch):
config: working-dir/model/moses.ini
distortion-file: 0-0 msd-bidirectional-fe 6 /home/moses/
webapps/working-dir/model/reordering-table.msd-bidirectional-fe.
0.5.0-0.gz
distortion-limit: 6
input-factors: 0
lmodel-file: 0 0 5 working-dir/lm/europarl.lm
mapping: 0 T 0
ttable-file: 0 0 5 /home/moses/webapps/working-dir/model/
phrase-table.0-0.gz
ttable-limit: 20 0
weight-d: 0.3 0.3 0.3 0.3 0.3 0.3 0.3
weight-l: 0.5000
weight-t: 0.2 0.2 0.2 0.2 0.2
weight-w: -1
Loading lexical distortion models...
have 1 models
Creating lexical reordering...
weights: 0.300 0.300 0.300 0.300 0.300 0.300
Loading table into memory...done.
Created lexical orientation reordering
Start loading LanguageModel working-dir/lm/europarl.lm : [500.000]
seconds
Finished loading LanguageModels : [520.000] seconds
Start loading PhraseTable /home/moses/webapps/working-dir/model/
phrase-table.0-0.gz : [520.000] seconds
Any help would be GREATLY appreciated
Thanking you all in anticipation
Forgot the famous last words? Access your message archive online.
Click here._______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
