Hello, we've the same problem, and we will try to split the dev in
subpart to run moses in parallel.
Alexandre.
J.Tiedemann wrote:
> Hello Moses users and developers,
>
>
> I'm facing problems with memory requirements and decoding speed when
> running a factored model on Europarl data. I trained a model with
> lemma and POS factors with about 1 million sentence pairs but running
> moses always fails after some sentences because of memory allocation
> errors (terminate called after throwing an instance of
> 'std::bad_alloc')
>
> I use 3 translation factors and 2 generation factors together with
> lexicalized reordering models. I already tried to reduce memory usage
> by compiling phrase and reordering tables to binary formats and by
> switching to IRSTLM with binary LMs. I also added
> '[use-persistent-cache] 0' to my config file but still moses allocates
> between 2 and 4GB of internal memory and after about 20 test sentences
> the process crashes. This also means that I cannot run mert on any
> tuning data. Anyway, the decoding also becomes so slow that tuning
> would probably not be feasible for my data (one sentence takes between
> 200 and 2000 seconds to translate).
>
> I'm just wondering what other moses users experienced with factored
> models and what I should expect when training on rather large data. Is
> there any other trick I could try to get at least a result back for my
> test set? Do I just need more memory? By the way, filtering the phrase
> tables according to input data didn't work for me either (still too
> big to fit into memory). What are the limits and what are the system
> requirements?
>
> I also wonder if the cache can be controlled somehow to get a
> reasonable decoding speed without running out of memory so quickly.
> With caching switched on I cannot even run more than a couple of
> sentences.
>
> Using the latest release improved the situation a little bit but I
> still run out of memory. Any help would be greatly appreciated. I'm
> just curious to see the results with a factorized model compared to
> the baseline approach with plain text only.
>
> cheers,
>
> Jörg
>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
--
Alexandre Allauzen
Univ. Paris XI, LIMSI-CNRS
Tel : 01.69.85.80.64 (80.88)
Bur : 114 LIMSI Bat. 508
[EMAIL PROTECTED]
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
