Hi, OK, I definitely need to add format checks :) If you do not provide alignment information, use
-encoding None The compression will be slightly less effective, but still about 4-5 times smaller than the Moses binary. Try to change your format to (with a single space at the end): source phrase ||| target phrase ||| score1 score2 score3 score4 score5 score6 ||| ||| Also do not forget to sort with "LC_ALL=C sort" Tell me if that works, Best, Marcin W dniu 06.08.2012 22:00, Lane Schwartz pisze: > Is there a special flag I need to pass to processPhraseTableMin if I > provide an input phrase table that lacks alignment info? > > I'm providing a phrase table that was using a custom in-house script > (not using Moses's training scripts), that has the following format. I > get a segfault when I attempt to train with this: > > source phrase ||| target phrase ||| score1 score2 score3 score4 score5 score6 > > $ processPhraseTableMin -in /path/to/my_plaintext_pt -out phrase-table > -nscores 6 > > This results in a segfault. If you like I can provide more log info. > > Cheers, > Lane > > > On Sun, Aug 5, 2012 at 10:45 AM, Marcin Junczys-Dowmunt > <[email protected]> wrote: >> Hi everybody, >> I am happy to announce that a new compact binary phrase table >> implementation is now available in Moses. >> >> Some figures: A 22.1 GB standard binary phrase table (ca. 220M phrase >> pairs) can be stored in 2.9 GB. A corresponding 46.6 GB standard binary >> lexical reordering table fits into 1.4 GB. No pruning is involved, >> compression is (nearly) lossless. >> >> Due to reduced I/O requirements, phrase pair retrieving speed from disk >> is significantly improved. Both, phrase tables and reordering tables can >> be accessed on-disk or loaded into memory which eliminates disk I/O >> completely. >> >> Basic usage is quite easy, it can be used as a drop-in replacement for >> the standard binary phrase table. Best compression performance, can be >> achieved for phrase tables that contain phrase-internal alignment >> information. >> >> Just checkout the newest version of Moses from its github repository and >> proceed with the instructions on the Moses website: >> >> Compact Phrase Table >> http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc5 >> >> Compact Lexical Reordering Table >> http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc6 >> >> If you have questions do not hesitate to write me an email. >> >> For more technical information on advanced options or citing see: >> >> @article{junczys_mtm_2012, >> author = {Marcin Junczys-Dowmunt}, >> title = {Phrasal Rank-Encoding: Exploiting Phrase Redundancy and >> Translational Relations for Phrase Table Compression}, >> journal = {The Prague Bulletin of Mathematical Linguistics}, >> volume = {98}, >> year = {2012}, >> note = {Draft version for the Machine Translation Marathon 2012}, >> file = >> "http://amu.edu.pl/~junczys/dokuwiki/lib/exe/fetch.php?cache=&media=wiki:mjd2012pbml98_draft.pdf";, >> } >> >> @inproceedings{junczys_eamt_2012, >> author = {Marcin Junczys-Dowmunt}, >> title = {A Phrase Table without Phrases: Rank Encoding for >> Better Phrase Table Compression}, >> booktitle = {16th Annual Conference of the European Association >> for Machine Translation (EAMT)}, >> year = {2012}, >> address = {Trento, Italy}, >> url = "http://hltshare.fbk.eu/EAMT2012/html/Papers/57.pdf";, >> pages = {245--252}, >> } >> >> Happy testing and compressing, >> Best, >> Marcin >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support > > > -- dr inż. Marcin Junczys-Dowmunt Uniwersytet im. Adama Mickiewicza Wydział Matematyki i Informatyki ul. Umultowska 87 61-614 Poznań _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
