I've seen mention of filtering the model before translation. Many of the phrase in your TM may not even exist in the document you wish to translate. Why bother loading them into memory?
Quoting Hubert Crépy <[EMAIL PROTECTED]>: > -- apologies if this is a duplicate, I'm having trouble posting to the > list -- > > I just built (as an exercise) a fr-en model based on the first 500K > sentences of Europarl. > It appears the generated model is just way too big to load on my Windows > machine: while trying to load, I see memory/swap going up and up and up > to just about 3Gb, then the process aborts. > > /STATMT/working-dir/model > $ ls -l > total 1343280 > -rw-r--r-- 41328421 Feb 27 09:26 aligned.0.en > -rw-r--r-- 47262588 Feb 27 09:25 aligned.0.fr > -rw-r--r-- 43062565 Feb 27 09:18 aligned.grow-diag-final-and > -rw-r--r-- 211396458 Feb 27 10:40 extract.0-0.gz > -rw-r--r-- 209218625 Feb 27 10:50 extract.0-0.inv.gz > -rw-r--r-- 162287272 Feb 27 10:28 extract.0-0.o.gz > -rw-r--r-- 22179434 Feb 27 09:29 lex.0-0.f2n > -rw-r--r-- 22179434 Feb 27 09:29 lex.0-0.n2f > -rw-r--r-- 949 Feb 28 13:01 moses.ini > -rw-r--r-- 447528428 Feb 28 05:33 phrase-table.0-0.gz > -rw-r--r-- 169060574 Feb 28 12:56 > reordering-table.msd-bidirectional-fe.0.5.0-0.gz > > From what I read here, maybe using binary versions of the files would help? > Where do I find information on how to use "the binary translation table > format" ? > Any other suggestions to reduce the footprint? > > Thanks in advance. > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
