Hello Moses users and developers,
I'm facing problems with memory requirements and decoding speed when running a factored model on Europarl data. I trained a model with lemma and POS factors with about 1 million sentence pairs but running moses always fails after some sentences because of memory allocation errors (terminate called after throwing an instance of 'std::bad_alloc') I use 3 translation factors and 2 generation factors together with lexicalized reordering models. I already tried to reduce memory usage by compiling phrase and reordering tables to binary formats and by switching to IRSTLM with binary LMs. I also added '[use-persistent-cache] 0' to my config file but still moses allocates between 2 and 4GB of internal memory and after about 20 test sentences the process crashes. This also means that I cannot run mert on any tuning data. Anyway, the decoding also becomes so slow that tuning would probably not be feasible for my data (one sentence takes between 200 and 2000 seconds to translate). I'm just wondering what other moses users experienced with factored models and what I should expect when training on rather large data. Is there any other trick I could try to get at least a result back for my test set? Do I just need more memory? By the way, filtering the phrase tables according to input data didn't work for me either (still too big to fit into memory). What are the limits and what are the system requirements? I also wonder if the cache can be controlled somehow to get a reasonable decoding speed without running out of memory so quickly. With caching switched on I cannot even run more than a couple of sentences. Using the latest release improved the situation a little bit but I still run out of memory. Any help would be greatly appreciated. I'm just curious to see the results with a factorized model compared to the baseline approach with plain text only. cheers, Jörg _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
