hi marcin,
thanks for this. i'm just spinning it up now and have a few questions:
1. does the phrase table/reordering model have to be sorted before
they are compacted with your tool?
2. on a 2.5GB compressed phrase table (trained on europarl), the
compacted tool peak memory usuage is 10GB+. Is this expected?
3. I tried compacting a 22 line phrase table (attached) and got a
segfault. The command I ran was
~/workspace/github/moses-smt/bin/processPhraseTableMin -in
phrase-table -out compact/phrase-table -nscores 1 -threads 2
I'll try & debug it myself but if you can see an obvious problem,
please let me know
On 05/08/2012 15:45, Marcin Junczys-Dowmunt wrote:
Hi everybody,
I am happy to announce that a new compact binary phrase table
implementation is now available in Moses.
Some figures: A 22.1 GB standard binary phrase table (ca. 220M phrase
pairs) can be stored in 2.9 GB. A corresponding 46.6 GB standard binary
lexical reordering table fits into 1.4 GB. No pruning is involved,
compression is (nearly) lossless.
Due to reduced I/O requirements, phrase pair retrieving speed from disk
is significantly improved. Both, phrase tables and reordering tables can
be accessed on-disk or loaded into memory which eliminates disk I/O
completely.
Basic usage is quite easy, it can be used as a drop-in replacement for
the standard binary phrase table. Best compression performance, can be
achieved for phrase tables that contain phrase-internal alignment
information.
Just checkout the newest version of Moses from its github repository and
proceed with the instructions on the Moses website:
Compact Phrase Table
http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc5
Compact Lexical Reordering Table
http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc6
If you have questions do not hesitate to write me an email.
For more technical information on advanced options or citing see:
@article{junczys_mtm_2012,
author = {Marcin Junczys-Dowmunt},
title = {Phrasal Rank-Encoding: Exploiting Phrase Redundancy and
Translational Relations for Phrase Table Compression},
journal = {The Prague Bulletin of Mathematical Linguistics},
volume = {98},
year = {2012},
note = {Draft version for the Machine Translation Marathon 2012},
file =
"http://amu.edu.pl/~junczys/dokuwiki/lib/exe/fetch.php?cache=&media=wiki:mjd2012pbml98_draft.pdf";,
}
@inproceedings{junczys_eamt_2012,
author = {Marcin Junczys-Dowmunt},
title = {A Phrase Table without Phrases: Rank Encoding for
Better Phrase Table Compression},
booktitle = {16th Annual Conference of the European Association
for Machine Translation (EAMT)},
year = {2012},
address = {Trento, Italy},
url = "http://hltshare.fbk.eu/EAMT2012/html/Papers/57.pdf";,
pages = {245--252},
}
Happy testing and compressing,
Best,
Marcin
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
der ||| the ||| 0.3 ||| |||
das ||| the ||| 0.4 ||| |||
das ||| it ||| 0.1 ||| |||
das ||| this ||| 0.1 ||| |||
die ||| the ||| 0.3 ||| |||
ist ||| is ||| 1.0 ||| |||
ist ||| 's ||| 1.0 ||| |||
das ist ||| it is ||| 0.2 ||| |||
das ist ||| this is ||| 0.8 ||| |||
es ist ||| it is ||| 0.8 ||| |||
es ist ||| this is ||| 0.2 ||| |||
ein ||| a ||| 1.0 ||| |||
ein ||| an ||| 1.0 ||| |||
klein ||| small ||| 0.8 ||| |||
klein ||| little ||| 0.8 ||| |||
kleines ||| small ||| 0.2 ||| |||
kleines ||| little ||| 0.2 ||| |||
haus ||| house ||| 1.0 ||| |||
alt ||| old ||| 0.8 ||| |||
altes ||| old ||| 0.2 ||| |||
gibt ||| gives ||| 1.0 ||| |||
es gibt ||| there is ||| 1.0 ||| |||
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
