I managed to move to another machine, I have 8GB Ram, i5 intel processor (2 processor dual cores)
Ubuntu 14.04 I am decoding using moses_chart executable with 2 threads. It’s still slow. The rule table size is 7,800,000 entries. Model was trained on 1,000,000 sentences, String to tree model. English to Arabic. Arabic was parsed by Stanford parser & I made my own wrapper to change it to moses style trees. Using ghkm with minimal option made the rule table smaller but it was much faster in decoding still slower than other rule extraction algorithms though, but it gave better results. I am hoping I could somehow minimize the rule table to be able to measure bleu score and compare it with the other extraction algorithm. I am trying to translate 1000 sentence, but in 16 hours it translated 39 sentences & was starting the 40th Is there any parameter I can use during decoding that would affect table size? Or how it is loaded during decoding. Should the on disk table be faster or slower than in memory? From: Philip Williams [mailto:[email protected]] Sent: Thursday, April 07, 2016 10:46 PM To: Ayah ElMaghraby Cc: [email protected] Subject: Re: [Moses-support] GHKM translation is slow In Moses the -hierarchical option is used for all syntax-based models, not just hierarchical phrase-based models. It's fine to give both options. If you give the -ghkm option then the GHKM extractor will be used instead of the Hiero-style rule extractor. It's hard to tell why your system is slow without knowing more about it. How much data is your model trained on? 4GB is only enough RAM for a small model, so the most likely problem is that moses is consuming all of your RAM and then swapping to disk. Phil On 7 Apr 2016, at 15:12, Ayah ElMaghraby <[email protected]> wrote: Yes It didn’t work well. Does giving –hierarchichal & -ghkm together during training make the rule table huge !? But in the train-model.perl it looked fine. Will this extract too many grammar rules perhaps making my rule table huge? How do they work together ? Or I shouldn’t be using them together ? Or its normal? It’s the first time for me to use ghkm extraction in moses I am still trying it out. I have string-to-tree system translating from Arabic to English. From: Philip Williams [ <mailto:[email protected]> mailto:[email protected]] Sent: Thursday, April 07, 2016 11:03 AM To: Ayah ElMaghraby Cc: <mailto:[email protected]> [email protected] Subject: Re: [Moses-support] GHKM translation is slow Hi Ayah, the on-disk rule table works well for phrase-based and hierarchical phrase-based models, but not for GHKM models. Have you tried using the in-memory rule table? Phil On 6 Apr 2016, at 19:02, Ayah ElMaghraby < <mailto:[email protected]> [email protected]> wrote: Hello I am trying to created a SMT using ghkm extraction but it is very slow during translation it translated 53 sentences in 24 hrs using Ubuntu-64bit 14.04 & 4GB Ram I changed rule table to be on disk using executable createOnDiskPt. Is there any thing I can do to speed it up a little bit. I am currently using these options --target-syntax -glue-grammar -max-phrase-length 5 -ghkm --score-options="--GoodTuring" -external-bin-dir ~/mosesdecoder/tools/mgizapp -mgiza -mgiza-cpus 2 -snt2cooc snt2cooc.pl -parallel -sort-batch-size 253 -sort-compress gzip --giza-option m1=5,m2=2,mh=0,m3=3,m4=3 Regards, Ayah ElMaghraby _______________________________________________ Moses-support mailing list <mailto:[email protected]> [email protected] <http://mailman.mit.edu/mailman/listinfo/moses-support> http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
