Excellent. Thanks a lot Kenneth. Albert
-----Original Message----- From: Kenneth Heafield [mailto:[email protected]] Sent: lunes, 26 de mayo de 2014 20:05 To: Albert Llorens; [email protected] Subject: Re: [Moses-support] Get the probability of a given n-gram in a language model Hi, Here's a cheap server for fragment scoring. socat TCP4-LISTEN:2000,fork EXEC:"bin/fragment lm/test.arpa" Then in another terminal socat TCP4-CONNECT:localhost:2000 STDIO <text Now if you want to translate fragments instead then that's what Moses is for, though keep in mind that it will always prepend <s> and append </s> for translation. Kenneth On 05/26/14 02:04, Albert Llorens wrote: > Thanks, Kenneth. > > Yes, I want to score sentence fragments. I want to use Moses for fragment > translation, but only for frequent or probable fragments. I'll try what you > suggest. Any chance the query could be done remotely, using mosesserver or > anything else? > > Kind regards. > > Albert > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Kenneth Heafield > Sent: viernes, 23 de mayo de 2014 17:34 > To: [email protected] > Subject: Re: [Moses-support] Get the probability of a given n-gram in > a language model > > Hi, > > You can use bin/query on an ARPA or KenLM file. Then just type > sentences at it (or use a file as stdin). By default it will assume you are > scoring sentences. You can pass -n to not wrap in <s> and </s>. > > It appears that you are asking to score sentence fragments. The > leading words will be scored using unigrams, bigrams, etc. from, say, > a 5-gram model. If you are using Kneser-Ney, these lower-order > probabilities (unigrams through 4-grams) are conditioned on having > backed off to them. If you want accurate scores for sentence > fragments, build a model of order 1, order 2, order 3, etc. then > combine them using > > build_binary -r "1.arpa 2.arpa 3.arpa 4.arpa" 5.arpa 5.rest > > You can then use > > bin/fragment 5.rest <fragments > > to attain log10 frequencies. For more on this rant, read > > http://kheafield.com/professional/edinburgh/rest_paper.pdf > > Kenneth > > On 05/23/14 05:13, Albert Llorens wrote: >> Hi, >> >> >> >> Is there a straightforward way I can ask Moses for the probability >> (or the frequency) of a given n-gram in a given language model? If >> so, can I do the query through mosesserver? >> >> >> >> Thanks. >> >> >> >> Kind regards. >> >> >> >> Albert >> >> >> >> >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
