Hi, factored decoding with 5 steps, as you are trying, is certainly a challenge. We are aware of these problems, and it is an interesting problem to make the factored translation option extension more efficient.
-phi On Sun, Feb 24, 2008 at 11:10 PM, J.Tiedemann <[EMAIL PROTECTED]> wrote: > > Hello Moses users and developers, > > > I'm facing problems with memory requirements and decoding speed when > running a factored model on Europarl data. I trained a model with > lemma and POS factors with about 1 million sentence pairs but running > moses always fails after some sentences because of memory allocation > errors (terminate called after throwing an instance of > 'std::bad_alloc') > > I use 3 translation factors and 2 generation factors together with > lexicalized reordering models. I already tried to reduce memory usage > by compiling phrase and reordering tables to binary formats and by > switching to IRSTLM with binary LMs. I also added > '[use-persistent-cache] 0' to my config file but still moses allocates > between 2 and 4GB of internal memory and after about 20 test sentences > the process crashes. This also means that I cannot run mert on any > tuning data. Anyway, the decoding also becomes so slow that tuning > would probably not be feasible for my data (one sentence takes between > 200 and 2000 seconds to translate). > > I'm just wondering what other moses users experienced with factored > models and what I should expect when training on rather large data. Is > there any other trick I could try to get at least a result back for my > test set? Do I just need more memory? By the way, filtering the phrase > tables according to input data didn't work for me either (still too > big to fit into memory). What are the limits and what are the system > requirements? > > I also wonder if the cache can be controlled somehow to get a > reasonable decoding speed without running out of memory so quickly. > With caching switched on I cannot even run more than a couple of > sentences. > > Using the latest release improved the situation a little bit but I > still run out of memory. Any help would be greatly appreciated. I'm > just curious to see the results with a factorized model compared to > the baseline approach with plain text only. > > cheers, > > Jörg > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
