Hieu, Thanks for your help. -- Felipe
El 16/06/10 01:29, Hieu Hoang escribió: > Hi Felipe, > > The on-demand table uses prefix tree like Zens& Ney. I would have just > reused the existing code but couldn't read it, so just reimplemented it. > > Abby Levenberg has implemented dynamic suffix array, but I think only > for phrase-based, not hierarchical. > > i don't know of any other work being done in moses to compress make the > suffix array or phrase table, or make it run faster, so it's still wide > open. > > On 15/06/2010 20:43, Felipe Sánchez Martínez wrote: >> Hello all, >> >> I wonder which is the method implemented in Moses for on-demand loading >> of the rule table when hierarchical phrase-based models are used. Is >> this the same method used for the phrase table in phrase-based SMT, i.e. >> the use of a prefix tree (trie) as describe by Zens& Ney (2007)? >> >> Zens& Ney (2007): "Efficient Phrase Table Representation for >> Machine Translation with Applications to Online MT and Speech >> Translation" >> >> In the literature I have found papers describing the use of suffix >> arrays both for phrase-based SMT (Callison-Burch, Bannard& Schoeder, >> 2005; Zhang& Vogel 2005), and for hierarchical phrase-based SMT (Lopez, >> 2008; Schwartz& Callison-Burch, 2010), but all these methods store the >> parallel corpus and compute the required probabilities on the fly. >> >> Callison-Burch, Bannard& Schoeder (2005): "Scaling Phrase-Based >> Statistical Machine Translation to Large Corpora and Longer >> Phrases" >> >> Zhang& Vogel (2005): "An Efficient Phrase-to-Phrase Alignment Model >> for Arbitrarily Long Phrases and Large Corpora. >> >> Lopez (2008): "Tera-Scale Translation Models via Pattern Matching" >> >> Schwartz& Callison-Burch (2010): "Hierarchical Phrase-Based Grammar >> Extraction in Joshua" >> >> >> In addition, I would also like to know if Moses implements any >> compression technique to save memory or disk space or if it just >> identifies each word by an integer (32 bits), and which data structure >> uses Moses to store the phrase table in memory. >> >> If I am missing some work I have not cited, please let me know. I >> appreciate your help. >> >> Thank you very much in advance. >> Kind regards. >> > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support -- Felipe Sánchez Martínez Departamento de Lenguajes y Sistemas Informáticos Universidad de Alicante, E-03071 Alicante (Spain) Tel.: +34 965 903 400, ext: 2966 Fax: +34 965 909 326 http://www.dlsi.ua.es/~fsanchez _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
