Hi Hieu, > 1. does the phrase table/reordering model have to be sorted before > they are compacted with your tool?
Yes, in the same way as the training script sorts phrase tables. I need to add this information or maybe run a check during creation. > 2. on a 2.5GB compressed phrase table (trained on europarl), the > compacted tool peak memory usuage is 10GB+. Is this expected? This is expected. The tool uses memory mapped files for temporary data between passes, these show up in top as used memory, but it is possible to build on machines that have much less memory than that. You just need to have that space in your temporary folder. If I remember correctly, I had no problems to compact huge phrase tables on a laptop, despite crazy memory usage beyond 60 GB displayed by top. > 3. I tried compacting a 22 line phrase table (attached) and got a > segfault. The command I ran was > ~/workspace/github/moses-smt/bin/processPhraseTableMin -in > phrase-table -out compact/phrase-table -nscores 1 -threads 2 > I'll try & debug it myself but if you can see an obvious problem, > please let me know Hm, maybe the -nscores 1 is the problem. Never tried a phrase table with only one score. It works fine with 6 scores (tested). I will take a look. > On 05/08/2012 15:45, Marcin Junczys-Dowmunt wrote: >> Hi everybody, >> I am happy to announce that a new compact binary phrase table >> implementation is now available in Moses. >> >> Some figures: A 22.1 GB standard binary phrase table (ca. 220M phrase >> pairs) can be stored in 2.9 GB. A corresponding 46.6 GB standard binary >> lexical reordering table fits into 1.4 GB. No pruning is involved, >> compression is (nearly) lossless. >> >> Due to reduced I/O requirements, phrase pair retrieving speed from disk >> is significantly improved. Both, phrase tables and reordering tables can >> be accessed on-disk or loaded into memory which eliminates disk I/O >> completely. >> >> Basic usage is quite easy, it can be used as a drop-in replacement for >> the standard binary phrase table. Best compression performance, can be >> achieved for phrase tables that contain phrase-internal alignment >> information. >> >> Just checkout the newest version of Moses from its github repository and >> proceed with the instructions on the Moses website: >> >> Compact Phrase Table >> http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc5 >> >> Compact Lexical Reordering Table >> http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc6 >> >> If you have questions do not hesitate to write me an email. >> >> For more technical information on advanced options or citing see: >> >> @article{junczys_mtm_2012, >> author = {Marcin Junczys-Dowmunt}, >> title = {Phrasal Rank-Encoding: Exploiting Phrase Redundancy and >> Translational Relations for Phrase Table Compression}, >> journal = {The Prague Bulletin of Mathematical Linguistics}, >> volume = {98}, >> year = {2012}, >> note = {Draft version for the Machine Translation Marathon 2012}, >> file = >> "http://amu.edu.pl/~junczys/dokuwiki/lib/exe/fetch.php?cache=&media=wiki:mjd2012pbml98_draft.pdf";, >> >> } >> >> @inproceedings{junczys_eamt_2012, >> author = {Marcin Junczys-Dowmunt}, >> title = {A Phrase Table without Phrases: Rank Encoding for >> Better Phrase Table Compression}, >> booktitle = {16th Annual Conference of the European Association >> for Machine Translation (EAMT)}, >> year = {2012}, >> address = {Trento, Italy}, >> url = "http://hltshare.fbk.eu/EAMT2012/html/Papers/57.pdf";, >> pages = {245--252}, >> } >> >> Happy testing and compressing, >> Best, >> Marcin >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> > > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > -- dr inż. Marcin Junczys-Dowmunt Uniwersytet im. Adama Mickiewicza Wydział Matematyki i Informatyki ul. Umultowska 87 61-614 Poznań _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
