Hi, Since the topic of huge reordering tables appeared I just wanted to let you now that I am working at the moment on a much smaller reordering table and phrase table representation. The reordering-table is already working and could be used within Moses without much problems. For instance, I have a reduction from 114G to 5.2G for the reordering table. Right now however, it needs to be kept fully in RAM. Some memory-mapping capabilities are planned. It also needs quite a lot memory during construction time, but I will do something with that, too.
If someone wants to take a look or share some comments, it's in the WIPO branch, LexicalReorderingDictionary.* (class LexicalReoderingDictionaryHashed). Best, Marcin W dniu 03.02.2012 05:50, Wang Pidong pisze: > Thanks a lot! > I am going to increase my disk space and try again. > > Best wishes! > Pidong > > On 3 February 2012 12:11, Hieu Hoang <[email protected] > <mailto:[email protected]>> wrote: > > yes, that's typical. > > as a comparison, my 3.9GB (text, gz) reordering table takes up 40GB > when binarized. > > On Fri, Feb 3, 2012 at 10:49 AM, Wang Pidong <[email protected] > <mailto:[email protected]>> wrote: > > Hi all, > > I am now trying to binarize a big lexical reordering table, > *reordering-table.wbe-msd-bidirectional-fe.gz*, whose size is > 2.4GB after gzip. > After about 6-hour running of the *processLexicalTable* tool, > the incomplete binarized table (about 68GB) used up all my disk > space. > > Does anyone have the experience of binarizing big lexical > reordering table? > Is my case typical? do I need to increase my disk space then try > again? > > Thank you in advance! > > Best wishes! > Pidong > > > > > _______________________________________________ > Moses-support mailing list > [email protected] <mailto:[email protected]> > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > > -- > Wang Pidong > > Department of Computer Science > School of Computing > National University of Singapore > > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
