Hi Felipe, The on-demand table uses prefix tree like Zens & Ney. I would have just reused the existing code but couldn't read it, so just reimplemented it.
Abby Levenberg has implemented dynamic suffix array, but I think only for phrase-based, not hierarchical. i don't know of any other work being done in moses to compress make the suffix array or phrase table, or make it run faster, so it's still wide open. On 15/06/2010 20:43, Felipe Sánchez Martínez wrote: > Hello all, > > I wonder which is the method implemented in Moses for on-demand loading > of the rule table when hierarchical phrase-based models are used. Is > this the same method used for the phrase table in phrase-based SMT, i.e. > the use of a prefix tree (trie) as describe by Zens& Ney (2007)? > > Zens& Ney (2007): "Efficient Phrase Table Representation for > Machine Translation with Applications to Online MT and Speech > Translation" > > In the literature I have found papers describing the use of suffix > arrays both for phrase-based SMT (Callison-Burch, Bannard& Schoeder, > 2005; Zhang& Vogel 2005), and for hierarchical phrase-based SMT (Lopez, > 2008; Schwartz& Callison-Burch, 2010), but all these methods store the > parallel corpus and compute the required probabilities on the fly. > > Callison-Burch, Bannard& Schoeder (2005): "Scaling Phrase-Based > Statistical Machine Translation to Large Corpora and Longer > Phrases" > > Zhang& Vogel (2005): "An Efficient Phrase-to-Phrase Alignment Model > for Arbitrarily Long Phrases and Large Corpora. > > Lopez (2008): "Tera-Scale Translation Models via Pattern Matching" > > Schwartz& Callison-Burch (2010): "Hierarchical Phrase-Based Grammar > Extraction in Joshua" > > > In addition, I would also like to know if Moses implements any > compression technique to save memory or disk space or if it just > identifies each word by an integer (32 bits), and which data structure > uses Moses to store the phrase table in memory. > > If I am missing some work I have not cited, please let me know. I > appreciate your help. > > Thank you very much in advance. > Kind regards. > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
