Is there a special flag I need to pass to processPhraseTableMin if I provide an input phrase table that lacks alignment info?
I'm providing a phrase table that was using a custom in-house script (not using Moses's training scripts), that has the following format. I get a segfault when I attempt to train with this: source phrase ||| target phrase ||| score1 score2 score3 score4 score5 score6 $ processPhraseTableMin -in /path/to/my_plaintext_pt -out phrase-table -nscores 6 This results in a segfault. If you like I can provide more log info. Cheers, Lane On Sun, Aug 5, 2012 at 10:45 AM, Marcin Junczys-Dowmunt <[email protected]> wrote: > Hi everybody, > I am happy to announce that a new compact binary phrase table > implementation is now available in Moses. > > Some figures: A 22.1 GB standard binary phrase table (ca. 220M phrase > pairs) can be stored in 2.9 GB. A corresponding 46.6 GB standard binary > lexical reordering table fits into 1.4 GB. No pruning is involved, > compression is (nearly) lossless. > > Due to reduced I/O requirements, phrase pair retrieving speed from disk > is significantly improved. Both, phrase tables and reordering tables can > be accessed on-disk or loaded into memory which eliminates disk I/O > completely. > > Basic usage is quite easy, it can be used as a drop-in replacement for > the standard binary phrase table. Best compression performance, can be > achieved for phrase tables that contain phrase-internal alignment > information. > > Just checkout the newest version of Moses from its github repository and > proceed with the instructions on the Moses website: > > Compact Phrase Table > http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc5 > > Compact Lexical Reordering Table > http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc6 > > If you have questions do not hesitate to write me an email. > > For more technical information on advanced options or citing see: > > @article{junczys_mtm_2012, > author = {Marcin Junczys-Dowmunt}, > title = {Phrasal Rank-Encoding: Exploiting Phrase Redundancy and > Translational Relations for Phrase Table Compression}, > journal = {The Prague Bulletin of Mathematical Linguistics}, > volume = {98}, > year = {2012}, > note = {Draft version for the Machine Translation Marathon 2012}, > file = > "http://amu.edu.pl/~junczys/dokuwiki/lib/exe/fetch.php?cache=&media=wiki:mjd2012pbml98_draft.pdf";, > } > > @inproceedings{junczys_eamt_2012, > author = {Marcin Junczys-Dowmunt}, > title = {A Phrase Table without Phrases: Rank Encoding for > Better Phrase Table Compression}, > booktitle = {16th Annual Conference of the European Association > for Machine Translation (EAMT)}, > year = {2012}, > address = {Trento, Italy}, > url = "http://hltshare.fbk.eu/EAMT2012/html/Papers/57.pdf";, > pages = {245--252}, > } > > Happy testing and compressing, > Best, > Marcin > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support -- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- R.A. Heinlein, "Time Enough For Love" _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
