Hi everybody, I am happy to announce that a new compact binary phrase table implementation is now available in Moses.
Some figures: A 22.1 GB standard binary phrase table (ca. 220M phrase pairs) can be stored in 2.9 GB. A corresponding 46.6 GB standard binary lexical reordering table fits into 1.4 GB. No pruning is involved, compression is (nearly) lossless. Due to reduced I/O requirements, phrase pair retrieving speed from disk is significantly improved. Both, phrase tables and reordering tables can be accessed on-disk or loaded into memory which eliminates disk I/O completely. Basic usage is quite easy, it can be used as a drop-in replacement for the standard binary phrase table. Best compression performance, can be achieved for phrase tables that contain phrase-internal alignment information. Just checkout the newest version of Moses from its github repository and proceed with the instructions on the Moses website: Compact Phrase Table http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc5 Compact Lexical Reordering Table http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc6 If you have questions do not hesitate to write me an email. For more technical information on advanced options or citing see: @article{junczys_mtm_2012, author = {Marcin Junczys-Dowmunt}, title = {Phrasal Rank-Encoding: Exploiting Phrase Redundancy and Translational Relations for Phrase Table Compression}, journal = {The Prague Bulletin of Mathematical Linguistics}, volume = {98}, year = {2012}, note = {Draft version for the Machine Translation Marathon 2012}, file = "http://amu.edu.pl/~junczys/dokuwiki/lib/exe/fetch.php?cache=&media=wiki:mjd2012pbml98_draft.pdf";, } @inproceedings{junczys_eamt_2012, author = {Marcin Junczys-Dowmunt}, title = {A Phrase Table without Phrases: Rank Encoding for Better Phrase Table Compression}, booktitle = {16th Annual Conference of the European Association for Machine Translation (EAMT)}, year = {2012}, address = {Trento, Italy}, url = "http://hltshare.fbk.eu/EAMT2012/html/Papers/57.pdf";, pages = {245--252}, } Happy testing and compressing, Best, Marcin _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
