thanks christof,
i think a lot of people will find this feature very useful. I've checked
it in
http://mosesdecoder.svn.sourceforge.net/viewvc/mosesdecoder?view=revision&revision=3637
On 22/10/2010 00:01, Christof Pintaske wrote:
Hi,
"train-model.perl" with the parameter "-phrase-word-alignment" adds
word-for-word alignment information to the phrase table. Unfortunately
this information get's lost when converting the textual phrase-table
into a binary format with "processPhraseTable". Using
"processPhraseTable -alignment-info" was meant to store the alignment
information in the binary table as well. This functionality is broken
since the format for the word alignment information changed and
currently no word alignment information is stored in the binary phrase
tables. Being required to use the textual file limits the size of the
phrase-table in respect to the memory on the server.
The attached patch provides the missing changes. It stores new-style
alignment information with the target candidates in the
"phrase-table.binphr.tgtdata.wa" file and reads them out
correspondingly (It doesn't split the alignment information into
source and target alignment as in the old implementation/format. It
keeps it in a format supported by
"TargetPhrase::SetAlignmentInfo(std::string&)).
I tested the change with valgrind for both "moses" and
"processPhraseTable" in a smaller moses translation system without any
complaints. And both the translation and the alignment file that gets
produced with "moses -use-alignment-info -print-alignment-info -T
<File>" are identical, regardless of text or binary phrase-table. The
patch should not change the behavior for phrase-tables without
word-alignment.
I hope you find the patch useful and hopefully it can be committed to
repo. Of course, please let me know if any modifications are necessary
or desirable.
best regards
Christof
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
