Re: [Moses-support] differences between moses and moses2 output

hiya Hieu Hoang http://www.hoang.co.uk/hieu On 13 October 2016 at 15:08, Vito Mandorino < vito.mandor...@linguacustodia.com> wrote: > We haven't checked the probingpt + minlexr speedup yet, however we have > found some further differences in the output with respect to the standard > Moses

Re: [Moses-support] differences between moses and moses2 output

We haven't checked the probingpt + minlexr speedup yet, however we have found some further differences in the output with respect to the standard Moses decoder. It happens sometimes that the order of replacement of placeholders with actual numbers is not the good one. For instance : moses2

Re: [Moses-support] differences between moses and moses2 output

Yes I modified the line in the moses.ini . My comparison was with respect to probingPT + minlexr reordering model (rather than .gz reordering model) 2016-10-07 16:25 GMT+02:00 Hieu Hoang : > weird. it should be a massive speedup (~500%). You have to change the > moses.ini

Re: [Moses-support] differences between moses and moses2 output

yep, it should give you a big speedup compared to probingpt + minlexr model Hieu Hoang http://www.hoang.co.uk/hieu On 7 October 2016 at 16:21, Vito Mandorino < vito.mandor...@linguacustodia.com> wrote: > Yes I modified the line in the moses.ini . My comparison was with respect > to probingPT +

Re: [Moses-support] differences between moses and moses2 output

weird. it should be a massive speedup (~500%). You have to change the moses.ini file slightly [feature] LexicalReordering … path=reordering-table.msd-bidirectional-fe.0.5.0-0.gz to [feature] LexicalReordering … property-index=0 Hieu Hoang http://www.hoang.co.uk/hieu On 7 October 2016

Re: [Moses-support] differences between moses and moses2 output

Yes, that worked for me as well, thank you. There is a little improvement in speed but not that much actually (about 5% faster using 30 threads). 2016-10-04 11:44 GMT+02:00 Hieu Hoang : > yes - the script expects the files to be gzipped. > It runs ok for me. I executed this:

Re: [Moses-support] differences between moses and moses2 output

yes - the script expects the files to be gzipped. It runs ok for me. I executed this: MOSES_DIR=~/workspace/github/mosesdecoder.perf $MOSES_DIR/scripts/generic/binarize4moses2.perl --phrase-table=phrase-table.gz --lex-ro=reordering-table.wbe-msd-bidirectional-fe.gz

Re: [Moses-support] differences between moses and moses2 output

if guessed you ran binarize4moses2.perl so please give me the phrase-table and reordering model, and the exact command you ran Hieu Hoang http://www.hoang.co.uk/hieu On 3 October 2016 at 15:20, Vito Mandorino < vito.mandor...@linguacustodia.com> wrote: > I have managed to replicate the issue on

Re: [Moses-support] differences between moses and moses2 output

I have managed to replicate the issue on a smaller corpus. Do you need the training corpus, the tables (phrase- and reordering-), or all of them? Vito 2016-09-30 13:30 GMT+02:00 Hieu Hoang : > wow, that looks like a seriously problem. > > I've not seen this before. If you

Re: [Moses-support] differences between moses and moses2 output

I tried the following command: perl /home/Moses/mosesdecoder/scripts/generic/binarize4moses2.perl --phrase-table=/home/vito/phrase-table.sorted --lex-ro=/home/vito/reordering-table.sorted --output-dir=/home/vito/integrated_phrase-reordering/ --num-lex-scores=6 but it gets stuck with the

Re: [Moses-support] differences between moses and moses2 output

use the script: scripts/generic/binarize4moses2.perl It takes as input the (text) phrase-table and the (text) lexro model. It will give you the probing pt which contains the info for both. To use this script, Moses MUST be compiled with the flag --with-cmph. Also, the program in

Re: [Moses-support] differences between moses and moses2 output

you will get another big speedup fromm integrating the lexro into the pt Hieu Hoang http://www.hoang.co.uk/hieu On 29 September 2016 at 15:03, Vito Mandorino < vito.mandor...@linguacustodia.com> wrote: > Yes the model includes a lexicalised reordering model but is not > integrated into the

Re: [Moses-support] differences between moses and moses2 output

Ok thank you, I'll check that. Do you know how to perform the integration? Juxtaposing the 4 phrase-table scores and the 6 reordering scores before calling CreateProbingPT2 would be enough? (I have used the CreateProbingPT2 binary and not CreateProbingPT so far) Vito 2016-09-29 16:07 GMT+02:00

Re: [Moses-support] differences between moses and moses2 output

Yes the model includes a lexicalised reordering model but is not integrated into the probingPT. The size of the LM is 1.8G. 2016-09-29 15:59 GMT+02:00 Hieu Hoang : > ps. how big is your LM? > > Hieu Hoang > http://www.hoang.co.uk/hieu > > On 29 September 2016 at 14:58, Hieu

Re: [Moses-support] differences between moses and moses2 output

great, thanks. Do you use the lexicalised reordering model, and is it integrated into the phrase-table in Moses2? There is latency in communicating with the server. As Moses2 is much faster now, the client can't feed it fast enough. You should see that moses2 command line will max out the CPU,

Re: [Moses-support] differences between moses and moses2 output

ps. how big is your LM? Hieu Hoang http://www.hoang.co.uk/hieu On 29 September 2016 at 14:58, Hieu Hoang wrote: > great, thanks. Do you use the lexicalised reordering model, and is it > integrated into the phrase-table in Moses2? > > There is latency in communicating with

Re: [Moses-support] differences between moses and moses2 output

Yes, here are some data: Average source sentence length: 29 tokens Phrase-table size, probingPT: 11G Phrase-table size, compact phrase-table: 2.1G Translation time Moses2 with 32 threads: 1m36.511s Translation time Moses with 32 threads: 6m14.248s Translation time Moses2 with 32 threads in

Re: [Moses-support] differences between moses and moses2 output

cool. do you have any indications of speed, especially when using multiple threads? model sizes and average input sentence length are also relevant. ___ Moses-support mailing list Moses-support@mit.edu

Re: [Moses-support] differences between moses and moses2 output

Now it works! Thanks. On 6000 test sentences the Moses2 output is now actually 100% identical to the standard Moses output. Vito 2016-09-28 16:12 GMT+02:00 Hieu Hoang : > hi Vito, > > please git pull and try decoding again. I've just pushed a fix >

Re: [Moses-support] differences between moses and moses2 output

hi Vito, please git pull and try decoding again. I've just pushed a fix https://github.com/hieuhoang/mosesdecoder/commit/0005e98b2674906162ce7945c5edd6a42c9ca418 Basically, I've changed changed the behavious of the pugi call so that it doesn't unescape the words Hieu Hoang

Re: [Moses-support] differences between moses and moses2 output

ah ok. do you have a moses.ini and example input sentence to go with that. pugixml.cpp is used to parse the input sentence for XML markups for placeholders, forced-translation etc. You shouldn't change the code for pugixml 'cos it's an imported library that we don't control and we may reimport in

Re: [Moses-support] differences between moses and moses2 output

We are able to replicate the issue with the probingPT version of this phrase-table: ||| ||| 1 1 1 1 ||| 0-0 ||| 1 1 1 ||| ||| ||| ||| 1 1 1 1 ||| 0-0 ||| 1 1 1 ||| ||| ||| ||| 1 1 1 1 ||| 0-0 ||| 1 1 1 ||| ||| ||| ||| 1 1 1 1 ||| 0-0 ||| 1 1 1 ||| ||| ||| ||| 1 1 1 1 ||| 0-0 ||| 1 1 1

Re: [Moses-support] differences between moses and moses2 output

Can you make your model files available for download? Moses and Moses2 aren't guaranteed to give exactly the same answer. However, they should be the same quality overall Hieu Hoang http://www.hoang.co.uk/hieu On 28 September 2016 at 09:53, Vito Mandorino < vito.mandor...@linguacustodia.com>