hi I was trying with the language model created on monolingual Hindi corpus of ACL 2014. after binarizing it, its size was 1.9 GB. And that was creating problem in Tuning. When I used language model created on target language side of the parallel corpus. then its size was only 35 MB and it is working fine with tuning. So I think the problem was the huge size of the language model. Is there any solution to use such a huge language model with MOSES without having problem at the time of tuning.
Thanks in advance. On Sun, Aug 7, 2016 at 7:52 PM, <[email protected]> wrote: > Send Moses-support mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.mit.edu/mailman/listinfo/moses-support > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Moses-support digest..." > > > Today's Topics: > > 1. Re: Compiling Moses with openmp (g++-6) on Mac OS 10.11? > (Hieu Hoang) > 2. Problem in Tuning (Jyoti Srivastava) > 3. Re: Problem in Tuning (Hieu Hoang) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 7 Aug 2016 00:07:16 +0200 > From: Hieu Hoang <[email protected]> > Subject: Re: [Moses-support] Compiling Moses with openmp (g++-6) on > Mac OS 10.11? > To: James <[email protected]>, [email protected] > Message-ID: <[email protected]> > Content-Type: text/plain; charset="windows-1252" > > You're better of using Linux, in a virtual machine if you have to. > > I'm a Mac user myself but the reality is most open source MT software is > developed on Linux and not really tested on other platforms > > > On 03/08/2016 10:45, James wrote: > > Hi, > > > > Cause we cannot use clang equipped with openmp on Mac OS 10.11, so > > I've been working around how to compile Moses using G++ (g++-6) > > instead of clang++, but so far no luck. > > > > What I want to do is to use some neural network language models like > > nplm / OxLM with Moses, and these LMs require openmp which is not > > officially supported by clang/clang++ on Mac OS 10.11. > > > > Here is my command and it didn't successfully compile the Moses: > > > > "bjam --with-boost=/Users/james/Desktop/Workspace/moses/boost_1_61_0 > > --with-cmph=/Users/james/Desktop/Workspace/moses/cmph-2.0 > > --with-xmlrpc-/Users/james/Desktop/Workspace/moses/xmlrpc-c-1.33.17 > > -j12 --with-nplm=/Users/james/Desktop/Workspace/moses/nplm" > > > > And, my system specs are: > > > > Macbook Air 13inch, > > Processor 1.7GHz Intel Core i7, > > Memory 8GB DDR3, > > Storage: 80GB free of 500GB. > > > > BTW, if there is alternative neural language model I can use for Moses > > (without using openmp), please let me know. This would be really > > appreciated... Thanks. > > > > Or, is there any way to use clang with openmp on Mac OS 10.11? > > > > -James > > > > > > > > > > _______________________________________________ > > Moses-support mailing list > > [email protected] > > http://mailman.mit.edu/mailman/listinfo/moses-support > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mailman.mit.edu/mailman/private/moses-support/ > attachments/20160806/18d8ae57/attachment-0001.html > > ------------------------------ > > Message: 2 > Date: Sun, 7 Aug 2016 19:07:24 +0530 > From: Jyoti Srivastava <[email protected]> > Subject: [Moses-support] Problem in Tuning > To: [email protected] > Message-ID: > <CAFxbX9vJhji0oBAfEYj+Q+oDi5Pu6tt5J_TXD=gBR7-kWiTnuQ@ > mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi > > I installed Moses on Ubuntu and run the baseline system successfully with > the English-French corpus given on the website. but now when I am running > it on English-Hindi corpus of ACL 2014 hen At the time of tuning it is > giving following error > > nohup: ignoring input > Using SCRIPTS_ROOTDIR: /home/abc/mosesdecoder/scripts > filtering the phrase tables... Sun Aug 7 18:52:36 IST 2016 > exec: /home/abc/mosesdecoder/scripts/training/filter-model-given-input.pl > ./filtered /home/abc/ACL_working/train/model/moses.ini > /home/abc/ACL_corpus/newsdev2014.true.en > Executing: /home/abc/mosesdecoder/scripts/training/ > filter-model-given-input.pl ./filtered > /home/abc/ACL_working/train/model/moses.ini > /home/abc/ACL_corpus/newsdev2014.true.en > filterphrases.out 2> > filterphrases.err > Asking moses for feature names and values from filtered/moses.ini > Executing: /home/abc/mosesdecoder/bin/moses -config filtered/moses.ini > -show-weights > exec: /home/abc/mosesdecoder/bin/moses -config filtered/moses.ini > -show-weights > Executing: /home/abc/mosesdecoder/bin/moses -config filtered/moses.ini > -show-weights > ./features.list 2> /dev/null > Exit code: 1 > ERROR: Failed to run '/home/abc/mosesdecoder/bin/moses -config > filtered/moses.ini -show-weights'. at > /home/abc/mosesdecoder/scripts/training/mert-moses.pl line 1748. > > Even I ran the same command as before in English-French corpus. I ran it > successfully on other system too with same English-Hindi corpus. but this > time I tried it many times but every time it is giving same error. > Please help me. > > On Fri, Aug 5, 2016 at 9:31 PM, <[email protected]> wrote: > > > Send Moses-support mailing list submissions to > > [email protected] > > > > To subscribe or unsubscribe via the World Wide Web, visit > > http://mailman.mit.edu/mailman/listinfo/moses-support > > or, via email, send a message with subject or body 'help' to > > [email protected] > > > > You can reach the person managing the list at > > [email protected] > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of Moses-support digest..." > > > > > > Today's Topics: > > > > 1. Using Moses on Ubuntu (rcm) > > 2. Crushed tuning neural model (Tomasz Gawryl) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Fri, 5 Aug 2016 08:59:21 +0200 > > From: rcm <[email protected]> > > Subject: [Moses-support] Using Moses on Ubuntu > > To: [email protected] > > Message-ID: <[email protected]> > > Content-Type: text/plain; charset=utf-8; format=flowed > > > > > > Hello! > > > > Finally, I will compile Moses on Ubuntu successfully. > > > > Then, for the first time, I download the sample model, run the decoder > > and everything was ok, the German text was translated to English text: > > "das ist ein kleines haus" --> "this is a small house" > > > > I analysed the files in the sample model, and there was: > > - moses.ini: configuration file > > - nbest.txt: (setting in moses.ini) file with a lot of variations over > > the ONLY input phrase and values for: "d", "lm", "w", "tm" and another > > value > > - phrase-table: (setting in moses.ini) file with a translation table > > over the ONLY words in the input phrase to the English language and a > > statistical value > > - trans-detail: translation hypothesis details...(I dont understand what > > is that!) > > > > But, what must I do if I need translate another phrases?? > > > > By another side, in www.statmt.org/moses/ web site, in the Online Demos > > part, there are a link for (for example): > > English -> {German, Spanish, French} > > I can type any sentence, and its translation (to German, for example) is > > displayed. > > Is running in the background Moses as Service ? > > > > Finally, my goal is translate any word wrote in a Office document using > > Moses, in simular way that Microsoft Translator or WordLingo or another > > systems are using nowadays. > > Can I communicate in some way with a Moses as Service (like moses web > > site above) or must I run my owner Moses as Service to translate any > > word wrote in a Office document?? > > Is possible add Moses like Online Machine Translation in the Translation > > Options panel in Word Office, like Microsoft Translator and WordLingo in > > Office 2010 (for example) ?? > > > > Thanks in advance. > > > > > > ------------------------------ > > > > Message: 2 > > Date: Fri, 5 Aug 2016 10:35:06 +0200 > > From: "Tomasz Gawryl" <[email protected]> > > Subject: [Moses-support] Crushed tuning neural model > > To: <[email protected]> > > Message-ID: <[email protected]> > > Content-Type: text/plain; charset="iso-8859-2" > > > > Hi, > > > > I have strange problem while tuning neural model. Tuning crashes with > > following error: > > > > > > > > LOG: > > > > > > > > EXECUTE STEPS > > > > number of steps doable or running: 1 at pi?, 5 sie 2016, 09:50:58 CEST > > > > doable: TUNING:tune > > > > executing > > /home/moses/working/experiments/ENPL33M6NRN/steps/7/TUNING_tune.7 via sh > > (1 > > active) > > > > step TUNING:tune crashed > > > > > > > > > > > > TUNING_tune.7.STDERR: > > > > > > > > Not a label, not a score 'nan'. Failed to parse the scores string: > > 'NPLMNEURAL= nan LexicalReordering0= 0 0 0 0 0 0 Distortion0= -8 LM0= > > -67.8925 WordPenalty0= -5 PhrasePenalty0= 5 TranslationModel0= 0 0 0 0' > of > > nbestlist 'run1.best100.out' at > > /home/moses/src/mosesdecoder/scripts/training/mert-moses.pl line 1503. > > > > > > > > Run.best100.out: > > > > > > > > moses@smtserver:~/working/experiments/ENPL33M6NRN/tuning/tmp.7$ head > > run1.best100.out > > > > 0 ||| let . something 's try ||| NPLMNEURAL= nan > LexicalReordering0= > > 0 > > 0 0 0 0 0 Distortion0= -8 LM0= -67.8925 WordPenalty0= -5 PhrasePenalty0= > 5 > > TranslationModel0= 0 0 0 0 ||| nan > > > > 0 ||| try let . 's something ||| NPLMNEURAL= nan > LexicalReordering0= > > 0 > > 0 0 0 0 0 Distortion0= -13 LM0= -68.7339 WordPenalty0= -5 > PhrasePenalty0= 5 > > TranslationModel0= 0 0 0 0 ||| nan > > > > 0 ||| something 's . try let ||| NPLMNEURAL= nan > LexicalReordering0= > > 0 > > 0 0 0 0 0 Distortion0= -14 LM0= -70.253 WordPenalty0= -5 PhrasePenalty0= > 5 > > TranslationModel0= 0 0 0 0 ||| nan > > > > > > > > > > > > What is the reason of appearing these "nan" values? I've successfully > > trained smaller (1000 sentences) corpus using the same config file. > > > > > > > > Please help! :) > > > > > > > > Regards, > > > > Thomas > > > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: http://mailman.mit.edu/mailman/private/moses-support/ > > attachments/20160805/933d8db9/attachment-0001.html > > > > ------------------------------ > > > > _______________________________________________ > > Moses-support mailing list > > [email protected] > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > > End of Moses-support Digest, Vol 118, Issue 13 > > ********************************************** > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mailman.mit.edu/mailman/private/moses-support/ > attachments/20160807/0a1e14c4/attachment-0001.html > > ------------------------------ > > Message: 3 > Date: Sun, 7 Aug 2016 16:22:10 +0200 > From: Hieu Hoang <[email protected]> > Subject: Re: [Moses-support] Problem in Tuning > To: Jyoti Srivastava <[email protected]> > Cc: moses-support <[email protected]> > Message-ID: > <CAEKMkbjiMbt___2r1mK5H4MxgTk6L1G19333RCxbj2Yc= > [email protected]> > Content-Type: text/plain; charset="utf-8" > > make sure all your paths are absolute, not relative > > Hieu Hoang > http://www.hoang.co.uk/hieu > > On 7 August 2016 at 15:37, Jyoti Srivastava <[email protected]> wrote: > > > Hi > > > > I installed Moses on Ubuntu and run the baseline system successfully with > > the English-French corpus given on the website. but now when I am running > > it on English-Hindi corpus of ACL 2014 hen At the time of tuning it is > > giving following error > > > > nohup: ignoring input > > Using SCRIPTS_ROOTDIR: /home/abc/mosesdecoder/scripts > > filtering the phrase tables... Sun Aug 7 18:52:36 IST 2016 > > exec: /home/abc/mosesdecoder/scripts/training/filter-model- > given-input.pl > > ./filtered /home/abc/ACL_working/train/model/moses.ini > > /home/abc/ACL_corpus/newsdev2014.true.en > > Executing: /home/abc/mosesdecoder/scripts/training/filter-model- > > given-input.pl ./filtered /home/abc/ACL_working/train/model/moses.ini > > /home/abc/ACL_corpus/newsdev2014.true.en > filterphrases.out 2> > > filterphrases.err > > Asking moses for feature names and values from filtered/moses.ini > > Executing: /home/abc/mosesdecoder/bin/moses -config filtered/moses.ini > > -show-weights > > exec: /home/abc/mosesdecoder/bin/moses -config filtered/moses.ini > > -show-weights > > Executing: /home/abc/mosesdecoder/bin/moses -config filtered/moses.ini > > -show-weights > ./features.list 2> /dev/null > > Exit code: 1 > > ERROR: Failed to run '/home/abc/mosesdecoder/bin/moses -config > > filtered/moses.ini -show-weights'. at /home/abc/mosesdecoder/ > > scripts/training/mert-moses.pl line 1748. > > > > Even I ran the same command as before in English-French corpus. I ran it > > successfully on other system too with same English-Hindi corpus. but this > > time I tried it many times but every time it is giving same error. > > Please help me. > > > > On Fri, Aug 5, 2016 at 9:31 PM, <[email protected]> wrote: > > > >> Send Moses-support mailing list submissions to > >> [email protected] > >> > >> To subscribe or unsubscribe via the World Wide Web, visit > >> http://mailman.mit.edu/mailman/listinfo/moses-support > >> or, via email, send a message with subject or body 'help' to > >> [email protected] > >> > >> You can reach the person managing the list at > >> [email protected] > >> > >> When replying, please edit your Subject line so it is more specific > >> than "Re: Contents of Moses-support digest..." > >> > >> > >> Today's Topics: > >> > >> 1. Using Moses on Ubuntu (rcm) > >> 2. Crushed tuning neural model (Tomasz Gawryl) > >> > >> > >> ---------------------------------------------------------------------- > >> > >> Message: 1 > >> Date: Fri, 5 Aug 2016 08:59:21 +0200 > >> From: rcm <[email protected]> > >> Subject: [Moses-support] Using Moses on Ubuntu > >> To: [email protected] > >> Message-ID: <[email protected]> > >> Content-Type: text/plain; charset=utf-8; format=flowed > >> > >> > >> Hello! > >> > >> Finally, I will compile Moses on Ubuntu successfully. > >> > >> Then, for the first time, I download the sample model, run the decoder > >> and everything was ok, the German text was translated to English text: > >> "das ist ein kleines haus" --> "this is a small house" > >> > >> I analysed the files in the sample model, and there was: > >> - moses.ini: configuration file > >> - nbest.txt: (setting in moses.ini) file with a lot of variations over > >> the ONLY input phrase and values for: "d", "lm", "w", "tm" and another > >> value > >> - phrase-table: (setting in moses.ini) file with a translation table > >> over the ONLY words in the input phrase to the English language and a > >> statistical value > >> - trans-detail: translation hypothesis details...(I dont understand what > >> is that!) > >> > >> But, what must I do if I need translate another phrases?? > >> > >> By another side, in www.statmt.org/moses/ web site, in the Online Demos > >> part, there are a link for (for example): > >> English -> {German, Spanish, French} > >> I can type any sentence, and its translation (to German, for example) is > >> displayed. > >> Is running in the background Moses as Service ? > >> > >> Finally, my goal is translate any word wrote in a Office document using > >> Moses, in simular way that Microsoft Translator or WordLingo or another > >> systems are using nowadays. > >> Can I communicate in some way with a Moses as Service (like moses web > >> site above) or must I run my owner Moses as Service to translate any > >> word wrote in a Office document?? > >> Is possible add Moses like Online Machine Translation in the Translation > >> Options panel in Word Office, like Microsoft Translator and WordLingo in > >> Office 2010 (for example) ?? > >> > >> Thanks in advance. > >> > >> > >> ------------------------------ > >> > >> Message: 2 > >> Date: Fri, 5 Aug 2016 10:35:06 +0200 > >> From: "Tomasz Gawryl" <[email protected]> > >> Subject: [Moses-support] Crushed tuning neural model > >> To: <[email protected]> > >> Message-ID: <[email protected]> > >> Content-Type: text/plain; charset="iso-8859-2" > >> > >> Hi, > >> > >> I have strange problem while tuning neural model. Tuning crashes with > >> following error: > >> > >> > >> > >> LOG: > >> > >> > >> > >> EXECUTE STEPS > >> > >> number of steps doable or running: 1 at pi?, 5 sie 2016, 09:50:58 CEST > >> > >> doable: TUNING:tune > >> > >> executing > >> /home/moses/working/experiments/ENPL33M6NRN/steps/7/TUNING_tune.7 via > sh > >> (1 > >> active) > >> > >> step TUNING:tune crashed > >> > >> > >> > >> > >> > >> TUNING_tune.7.STDERR: > >> > >> > >> > >> Not a label, not a score 'nan'. Failed to parse the scores string: > >> 'NPLMNEURAL= nan LexicalReordering0= 0 0 0 0 0 0 Distortion0= -8 LM0= > >> -67.8925 WordPenalty0= -5 PhrasePenalty0= 5 TranslationModel0= 0 0 0 0' > of > >> nbestlist 'run1.best100.out' at > >> /home/moses/src/mosesdecoder/scripts/training/mert-moses.pl line 1503. > >> > >> > >> > >> Run.best100.out: > >> > >> > >> > >> moses@smtserver:~/working/experiments/ENPL33M6NRN/tuning/tmp.7$ head > >> run1.best100.out > >> > >> 0 ||| let . something 's try ||| NPLMNEURAL= nan > >> LexicalReordering0= 0 > >> 0 0 0 0 0 Distortion0= -8 LM0= -67.8925 WordPenalty0= -5 > PhrasePenalty0= 5 > >> TranslationModel0= 0 0 0 0 ||| nan > >> > >> 0 ||| try let . 's something ||| NPLMNEURAL= nan > >> LexicalReordering0= 0 > >> 0 0 0 0 0 Distortion0= -13 LM0= -68.7339 WordPenalty0= -5 > PhrasePenalty0= > >> 5 > >> TranslationModel0= 0 0 0 0 ||| nan > >> > >> 0 ||| something 's . try let ||| NPLMNEURAL= nan > >> LexicalReordering0= 0 > >> 0 0 0 0 0 Distortion0= -14 LM0= -70.253 WordPenalty0= -5 > PhrasePenalty0= 5 > >> TranslationModel0= 0 0 0 0 ||| nan > >> > >> > >> > >> > >> > >> What is the reason of appearing these "nan" values? I've successfully > >> trained smaller (1000 sentences) corpus using the same config file. > >> > >> > >> > >> Please help! :) > >> > >> > >> > >> Regards, > >> > >> Thomas > >> > >> -------------- next part -------------- > >> An HTML attachment was scrubbed... > >> URL: http://mailman.mit.edu/mailman/private/moses-support/attachm > >> ents/20160805/933d8db9/attachment-0001.html > >> > >> ------------------------------ > >> > >> _______________________________________________ > >> Moses-support mailing list > >> [email protected] > >> http://mailman.mit.edu/mailman/listinfo/moses-support > >> > >> > >> End of Moses-support Digest, Vol 118, Issue 13 > >> ********************************************** > >> > > > > > > _______________________________________________ > > Moses-support mailing list > > [email protected] > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mailman.mit.edu/mailman/private/moses-support/ > attachments/20160807/6695a864/attachment.html > > ------------------------------ > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > > End of Moses-support Digest, Vol 118, Issue 14 > ********************************************** >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
