Hi,
Why are you running 100 MERT iterations as opposed to, say, 20? And
whether that amount of time is normal depends on how much RAM you have.
Kenneth
On 04/25/2015 03:31 PM, liling tan wrote:
> Dear Moses devs/users,
>
> @Marcin, thanks for the tip on the trie, I'll try out the trie.
>
> About the 100 MERT iterations, when i tried to run mert-moses.pl
> <http://mert-moses.pl> on that target language with 71GB of binarized
> language model on a 3000 line dev set, it took more than one day to tune
> using 10 threads. Is that normal?
>
> For a different experiment with a 38GB binarized language model, it took
> max 4-5 hours to tune with 10 threads on a 3000 lines dev set. (all the
> phrase-tables and rerodering-tables are binarized)
>
> I ran mert-moses.pl <http://mert-moses.pl> with only the model directory
> and the path to moses.ini.
>
> Regards,
> Liling
>
>
> binarizing like this gives you a lot smaller file:
>
> build_binary trie -a 22 -b 8 -q 8 lm.arpa.gz lm.kenlm
>
> This uses quantization, in theory that could cause quality loss, but I
> never saw that happen. Remove "-b 8 -q 8" if you are afraid of that, the
> file will be larger, but still a lot smaller than what you have. That's
> about all I do. You said "100 MERT iterations" ... what do you mean by
> that? Also the LM uses memory mapping in shared memory, so running
> several moses instances in parallel does not use additional memory due
> to the LM, similar for the phrase table.
>
>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
