Hi Miguel, If you just want to use language model over word-classes, here's a set of instructions to do that
http://www.statmt.org/moses/?n=Advanced.Models#ntoc4 Cheers, Nadir On Tue, Mar 31, 2015 at 7:25 PM, <[email protected]> wrote: > Send Moses-support mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.mit.edu/mailman/listinfo/moses-support > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Moses-support digest..." > > > Today's Topics: > > 1. Re: Unicode Issues when Using Compact Phrase Table, Binaries > vs. Own Build (????????? ????? (Ventsislav Zhechev)) > 2. Segmentation fault when loading a generation model in the > decoder (Miguel Gra?a) > 3. Requirement of sorting tables before compacting: unclear > documentation (Eleftherios Avramidis) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 31 Mar 2015 11:34:12 +0200 > From: "????????? ????? (Ventsislav Zhechev)" > <[email protected]> > Subject: Re: [Moses-support] Unicode Issues when Using Compact Phrase > Table, Binaries vs. Own Build > To: [email protected] > Message-ID: > <[email protected]> > Content-Type: text/plain; charset=utf-8 > > Hi, > > Any clue what systems could be messed up? On Ubuntu I complied boost 1.57, > cmph and Moses right out of the box, so I don?t see what I could have done > wrong there. > > I just checked and the gzip phrase tables are proper UTF-8. I even ran the > processPhraseTableMin binary from the website on the Ubuntu machine and still > got the same results. That is, if I query the compact phrase table with the > queryPhraseTableMin binary from the website, UTF-8 is recognised and I get > results; if I use queryPhraseTableMin that I complied on the same system, > UTF-8 is not recognised and I get no results. > > Does anyone have an idea what could influence the compilation of Moses in a > way that would prevent it from properly reading UTF-8? > Especially given that the Moses binaries for MacOS X from the website don?t > seem to read UTF-8 properly (at least on my machine), and I didn?t compile > those. > > > Cheers, > > Ventzi > >> 30.03.2015 ?., ? 11:08, [email protected] ???????(?): >> >> Date: Mon, 30 Mar 2015 11:08:13 +0200 >> From: Marcin Junczys-Dowmunt <[email protected]> >> Subject: Re: [Moses-support] Unicode Issues when Using Compact Phrase >> Table, Binaries vs. Own Build >> To: [email protected] >> Message-ID: <[email protected]> >> Content-Type: text/plain; charset="utf-8" >> >> Hi, >> the phrase-table and as far as I know Moses in general are >> unicode-agnostic, as long as you use utf-8. Input is handled as raw byte >> sequences, most of the time there are numeric identifiers only. >> Sounds more like a couple of messed up systems on your side, especially >> the part where self-compiled systems work or don't work. Cannot give you >> much more insight, unfortunately. >> Best, >> Marcin > > > > > ------------------------------ > > Message: 2 > Date: Tue, 31 Mar 2015 12:34:36 +0200 > From: Miguel Gra?a <[email protected]> > Subject: [Moses-support] Segmentation fault when loading a generation > model in the decoder > To: [email protected] > Message-ID: > <CANBbWMPsjPs6vzK-pcO02q8YN8LOvjzNzAdv5DtJmURG=en...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello, > > I am trying to configure Moses to use a language model over word classes. > To do this, I've added a generation model, which is basically just a > mapping from a word to a word class and a generation step after the > translation step. > > The issue is that while loading the generation model I get a segmentation > fault. I am using a custom word mapping file, so there might be issues with > the structure of the file. I have tried to follow the layout as shown in > your factored training tutorial. A line in the mapping file is split into 4 > fields: <target_word> <word_class> 1 0 > > The output of the command "./mosesdecoder/bin/moses -i > /work/smt2/graca/iwslt/2013/de-en/translation/pbt.all.wclm.hrm.moses.0/f-demo > -f moses.ini -v 3 > out" is given below: > > I have also tried not including the generation model and the language model > over word classes and in that case I do not obtain a segmentation fault. > > Regards, > Miguel > -------------------------------------- > > Defined parameters (per moses.ini or switch): > config: moses.ini > distortion-limit: 6 > feature: UnknownWordPenalty WordPenalty PhrasePenalty > PhraseDictionaryOnDisk name=TranslationModel0 num-features=4 > path=/work/smt2/graca/iwslt/2013/de-en/training/pbt.moses/model/filtered/phrase-table.bin > input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 > num-features=6 type=hier-msd-bidirectional-fe-allff input-factor=0 > output-factor=0 > path=/work/smt2/graca/iwslt/2013/de-en/training/pbt.moses/model/filtered/reordering-table.hier-msd-bidirectional-fe > Distortion KENLM name=LM0 factor=0 path=lm order=4 KENLM name=LM1 factor=1 > path=wclm order=7 Generation name=GEN path=e.classes.ext input-factor=0 > output-factor=1 > input-factors: 0 > input-file: > /work/smt2/graca/iwslt/2013/de-en/translation/pbt.all.wclm.hrm.moses.0/f-demo > mapping: 0 T 0 0 G 1 > verbose: 3 > weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 > TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 > 0.3 Distortion0= 0.3 LM0= 0.5 LM1= 0.5 GEN= 1 > input type is: text input > line=UnknownWordPenalty > FeatureFunction: UnknownWordPenalty0 start: 0 end: 0 > line=WordPenalty > FeatureFunction: WordPenalty0 start: 1 end: 1 > line=PhrasePenalty > FeatureFunction: PhrasePenalty0 start: 2 end: 2 > line=PhraseDictionaryOnDisk name=TranslationModel0 num-features=4 > path=/work/smt2/graca/iwslt/2013/de-en/training/pbt.moses/model/filtered/phrase-table.bin > input-factor=0 output-factor=0 > FeatureFunction: TranslationModel0 start: 3 end: 6 > DecodeFeature: > line=LexicalReordering name=LexicalReordering0 num-features=6 > type=hier-msd-bidirectional-fe-allff input-factor=0 output-factor=0 > path=/work/smt2/graca/iwslt/2013/de-en/training/pbt.moses/model/filtered/reordering-table.hier-msd-bidirectional-fe > FeatureFunction: LexicalReordering0 start: 7 end: 12 > Initializing Lexical Reordering Feature.. > line=Distortion > FeatureFunction: Distortion0 start: 13 end: 13 > line=KENLM name=LM0 factor=0 path=lm order=4 > FeatureFunction: LM0 start: 14 end: 14 > line=KENLM name=LM1 factor=1 path=wclm order=7 > FeatureFunction: LM1 start: 15 end: 15 > line=Generation name=GEN path=e.classes.ext input-factor=0 output-factor=1 > FeatureFunction: GEN start: 16 end: 16 > DecodeFeature: > Loading UnknownWordPenalty0 > Loading WordPenalty0 > Loading PhrasePenalty0 > Loading LexicalReordering0 > Loading table into memory...done. > Loading Distortion0 > Loading LM0 > Loading LM1 > Loading GEN > ./queueTrans.sh: line 1: 13276 Segmentation fault (core dumped) > ./mosesdecoder/bin/moses -i > /work/smt2/graca/iwslt/2013/de-en/translation/pbt.all.wclm.hrm.moses.0/f-demo > -f moses.ini -v 3 > out > ------------------------------------------ > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mailman.mit.edu/mailman/private/moses-support/attachments/20150331/134215e7/attachment-0001.htm > > ------------------------------ > > Message: 3 > Date: Tue, 31 Mar 2015 16:37:58 +0200 > From: Eleftherios Avramidis <[email protected]> > Subject: [Moses-support] Requirement of sorting tables before > compacting: unclear documentation > To: [email protected] > Message-ID: <[email protected]> > Content-Type: text/plain; charset=utf-8; format=flowed > > Hi, > > there is an unclear piece of documentation concerning the required > "sorting" of compact phrase and reordering tables > http://www.statmt.org/moses/?n=Advanced.RuleTables > > "Phrase tables are required to be sorted as above." but there is no > sorting mentioned "above". It appears twice. Can somebody clarify this > to me, so that I can proceed with table processing; and could you also > possibly fix the documentation? > > best > Lefteris > > > ------------------------------ > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > > End of Moses-support Digest, Vol 101, Issue 84 > ********************************************** _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
