if guessed you ran binarize4moses2.perl so please give me the phrase-table and reordering model, and the exact command you ran
Hieu Hoang http://www.hoang.co.uk/hieu On 3 October 2016 at 15:20, Vito Mandorino < [email protected]> wrote: > I have managed to replicate the issue on a smaller corpus. Do you need the > training corpus, the tables (phrase- and reordering-), or all of them? > > Vito > > 2016-09-30 13:30 GMT+02:00 Hieu Hoang <[email protected]>: > >> wow, that looks like a seriously problem. >> >> I've not seen this before. If you can make the data file available for >> download, it would be much appreciated >> >> Hieu Hoang >> http://www.hoang.co.uk/hieu >> >> On 30 September 2016 at 09:11, Vito Mandorino < >> [email protected]> wrote: >> >>> I tried the following command: >>> >>> perl /home/Moses/mosesdecoder/scripts/generic/binarize4moses2.perl >>> --phrase-table=/home/vito/phrase-table.sorted >>> --lex-ro=/home/vito/reordering-table.sorted >>> --output-dir=/home/vito/integrated_phrase-reordering/ --num-lex-scores=6 >>> >>> but it gets stuck with the following message: >>> >>> Reading phrase table finished, writing remaining files to disk. >>> terminate called after throwing an instance of >>> 'util::ProbingSizeException' >>> what(): ./util/probing_hash_table.hh:150 in >>> util::ProbingHashTable<EntryT, HashT, EqualT, ModT>::Entry* >>> util::ProbingHashTable<EntryT, HashT, EqualT, ModT>::Insert(const T&) [with >>> T = Moses2::Entry; EntryT = Moses2::Entry; HashT = boost::hash<long >>> unsigned int>; EqualT = std::equal_to<long unsigned int>; ModT = >>> util::DivMod; util::ProbingHashTable<EntryT, HashT, EqualT, >>> ModT>::MutableIterator = Moses2::Entry*; util::ProbingHashTable<EntryT, >>> HashT, EqualT, ModT>::Entry = Moses2::Entry] threw ProbingSizeException >>> because `++entries_ >= buckets_'. >>> Hash table with 1 buckets is full. >>> >>> >>> >>> >>> 2016-09-29 16:25 GMT+02:00 Hieu Hoang <[email protected]>: >>> >>>> use the script: >>>> scripts/generic/binarize4moses2.perl >>>> It takes as input the (text) phrase-table and the (text) lexro model. >>>> It will give you the probing pt which contains the info for both. >>>> >>>> To use this script, Moses MUST be compiled with the flag --with-cmph. >>>> Also, the program in contrib/sigtest-filter MUST have been successfully >>>> compiled. >>>> >>>> >>>> >>>> Hieu Hoang >>>> http://www.hoang.co.uk/hieu >>>> >>>> On 29 September 2016 at 15:19, Vito Mandorino < >>>> [email protected]> wrote: >>>> >>>>> Ok thank you, I'll check that. Do you know how to perform the >>>>> integration? Juxtaposing the 4 phrase-table scores and the 6 reordering >>>>> scores before calling CreateProbingPT2 would be enough? >>>>> >>>>> (I have used the CreateProbingPT2 binary and not CreateProbingPT so >>>>> far) >>>>> >>>>> Vito >>>>> >>>>> 2016-09-29 16:07 GMT+02:00 Hieu Hoang <[email protected]>: >>>>> >>>>>> you will get another big speedup fromm integrating the lexro into the >>>>>> pt >>>>>> >>>>>> Hieu Hoang >>>>>> http://www.hoang.co.uk/hieu >>>>>> >>>>>> On 29 September 2016 at 15:03, Vito Mandorino < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Yes the model includes a lexicalised reordering model but is not >>>>>>> integrated into the probingPT. The size of the LM is 1.8G. >>>>>>> >>>>>>> 2016-09-29 15:59 GMT+02:00 Hieu Hoang <[email protected]>: >>>>>>> >>>>>>>> ps. how big is your LM? >>>>>>>> >>>>>>>> Hieu Hoang >>>>>>>> http://www.hoang.co.uk/hieu >>>>>>>> >>>>>>>> On 29 September 2016 at 14:58, Hieu Hoang <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> great, thanks. Do you use the lexicalised reordering model, and is >>>>>>>>> it integrated into the phrase-table in Moses2? >>>>>>>>> >>>>>>>>> There is latency in communicating with the server. As Moses2 is >>>>>>>>> much faster now, the client can't feed it fast enough. You should see >>>>>>>>> that >>>>>>>>> moses2 command line will max out the CPU, whereas the server won't. >>>>>>>>> I'm >>>>>>>>> thinking of extending the server to processing multiple sentences at >>>>>>>>> a time >>>>>>>>> to speed it up >>>>>>>>> >>>>>>>>> Hieu Hoang >>>>>>>>> http://www.hoang.co.uk/hieu >>>>>>>>> >>>>>>>>> On 29 September 2016 at 14:49, Vito Mandorino < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Yes, here are some data: >>>>>>>>>> >>>>>>>>>> Average source sentence length: 29 tokens >>>>>>>>>> Phrase-table size, probingPT: 11G >>>>>>>>>> Phrase-table size, compact phrase-table: 2.1G >>>>>>>>>> >>>>>>>>>> Translation time Moses2 with 32 threads: 1m36.511s >>>>>>>>>> Translation time Moses with 32 threads: 6m14.248s >>>>>>>>>> Translation time Moses2 with 32 threads in server mode: 16m30.137s >>>>>>>>>> Translation time Moses with 32 threads in server mode: 62m33.208s >>>>>>>>>> >>>>>>>>>> Ram consumption during decoding: 4G for Moses2, 5G for Moses >>>>>>>>>> >>>>>>>>>> So Moses2 is 4 times faster, and 3 times faster in server mode. >>>>>>>>>> >>>>>>>>>> Do you know why in server mode the speed is so much slower with >>>>>>>>>> respect to batch mode, for both Moses and Moses2? >>>>>>>>>> >>>>>>>>>> Best regards, >>>>>>>>>> Vito >>>>>>>>>> >>>>>>>>>> 2016-09-28 18:52 GMT+02:00 Hieu Hoang <[email protected]>: >>>>>>>>>> >>>>>>>>>>> cool. do you have any indications of speed, especially when >>>>>>>>>>> using multiple threads? model sizes and average input sentence >>>>>>>>>>> length are >>>>>>>>>>> also relevant. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *M**. Vito MANDORINO -- Chief Scientist* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [image: Description : Description : lingua_custodia_final full >>>>>>>>>> logo] >>>>>>>>>> >>>>>>>>>> *The Translation Trustee* >>>>>>>>>> >>>>>>>>>> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux* >>>>>>>>>> >>>>>>>>>> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89 >>>>>>>>>> <%2B33%206%2084%2065%2068%2089>* >>>>>>>>>> >>>>>>>>>> *Email :* *[email protected] >>>>>>>>>> <[email protected]>* >>>>>>>>>> >>>>>>>>>> *Website :* >>>>>>>>>> *www.linguacustodia.finance <http://www.linguacustodia.com/>* >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *M**. Vito MANDORINO -- Chief Scientist* >>>>>>> >>>>>>> >>>>>>> [image: Description : Description : lingua_custodia_final full logo] >>>>>>> >>>>>>> *The Translation Trustee* >>>>>>> >>>>>>> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux* >>>>>>> >>>>>>> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89 >>>>>>> <%2B33%206%2084%2065%2068%2089>* >>>>>>> >>>>>>> *Email :* *[email protected] >>>>>>> <[email protected]>* >>>>>>> >>>>>>> *Website :* >>>>>>> *www.linguacustodia.finance <http://www.linguacustodia.com/>* >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *M**. Vito MANDORINO -- Chief Scientist* >>>>> >>>>> >>>>> [image: Description : Description : lingua_custodia_final full logo] >>>>> >>>>> *The Translation Trustee* >>>>> >>>>> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux* >>>>> >>>>> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89 >>>>> <%2B33%206%2084%2065%2068%2089>* >>>>> >>>>> *Email :* *[email protected] >>>>> <[email protected]>* >>>>> >>>>> *Website :* >>>>> *www.linguacustodia.finance <http://www.linguacustodia.com/>* >>>>> >>>> >>>> >>> >>> >>> -- >>> *M**. Vito MANDORINO -- Chief Scientist* >>> >>> >>> [image: Description : Description : lingua_custodia_final full logo] >>> >>> *The Translation Trustee* >>> >>> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux* >>> >>> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89 >>> <%2B33%206%2084%2065%2068%2089>* >>> >>> *Email :* *[email protected] >>> <[email protected]>* >>> >>> *Website :* >>> *www.linguacustodia.finance <http://www.linguacustodia.com/>* >>> >> >> > > > -- > *M**. Vito MANDORINO -- Chief Scientist* > > > [image: Description : Description : lingua_custodia_final full logo] > > *The Translation Trustee* > > *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux* > > *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89 > <%2B33%206%2084%2065%2068%2089>* > > *Email :* *[email protected] > <[email protected]>* > > *Website :* > *www.linguacustodia.finance <http://www.linguacustodia.com/>* >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
