Re: [Moses-support] Run fast_align with multithreading

Hi Lane, I had to track this one down too. The multithreading uses OpenMP, so if you install OpenMP on your machine and build fast_align, multithreading should be baked in. By default, it should use all available CPUs. You can override this by setting the OMP_NUM_THREADS environment variable

Re: [Moses-support] Does PhraseDictionaryMultiModel require all models to contain all phrases?

Hi Lane, As Vito mentioned, PhraseDictionaryMultiModel is originally for linear interpolation. It has an option to output all scores from all models rather than interpolating but I ended up writing PhraseDictionaryGroup to have the specific functionality plus some additions. One of the

Re: [Moses-support] Age feature

Hi Marwa, The cache-based model, the bitext sampling model, and the cdec implementation are three different things. The cache-based model is described in this paper and there are directions for using it in

Re: [Moses-support] discriminative reranking

Hi Selva, You can check out scripts/nbest-rescore for training an N-best reranker. This is generally used if you have some extra features you want to compute on the N-best lists but sometimes just training a reranker

Re: [Moses-support] Core generated while tuning using METEOR

Hi Carlos, What happens when you run the extractor command manually? /veu4/usuaris24/xtrans/mosesdecoder/bin/extractor --sctype METEOR --scconfig jar:/veu4/usuaris24/cescola/meteor-1.5/meteor-1.5.jar,case:true --scfile run1.scores.dat --ffile run1.features.dat -r

Re: [Moses-support] Training backward LM?

gt; LM, >>>>> so I would advise looking at the unit tests. >>>>> >>>>> I have checked in a trained backward LM. I believe it's a backward >>>>> trigram model trained on the text of the GNU GPL license file. >>>>> >>>&

Re: [Moses-support] Training backward LM?

As a follow-up, how exactly does one activate the BackwardLanguageModel feature? It doesn't seem to be registered anywhere although it's been in Moses for awhile. Still work in progress? Best, Michael On Thu, Mar 10, 2016 at 4:33 PM, Michael Denkowski < michael.j.denkow...@gmail.com>

[Moses-support] Training backward LM?

Hi Moses LM enthusiasts, This may be a silly question. If I want to use the BackwardLanguageModel feature, what (if anything) should I do to let KenLM know that it's a backward model? If I just reverse the words in each sentence and run lmplz/build_binary, will Moses do the right thing with the

Re: [Moses-support] Scripts for n-best-list rescoring

n: > > Commit: c6314d927d8b7b638eca387f31ccfab7facb6624 > > https://github.com/moses-smt/mosesdecoder/commit/c6314d927d8b7b638eca387f31ccfab7facb6624 > Author: Michael Denkowski <mdenk...@amazon.com> > Date: 2016-02-23 (Tue, 23 Feb 2016) > > Changed paths: > A scripts/nbest

Re: [Moses-support] Multiple Translation Tables

Hi Dorra, It sounds like PhraseDictionaryGroup is probably the best fit for your task. If you have 2 phrase tables (A and B), each translation will get a full set of scores from A and a full set of scores from B. If a translation is in A but not B, B's scores will be filled in with zeros. Since

Re: [Moses-support] Doubts on Multiple Decoding Paths

, Anoop (അനൂപ്) <anoop.kunchukut...@gmail.com > wrote: > Hi Michael, > > Is there a publication/document describing the *union *option or its use > in an MT scenario? > > Regards, > Anoop. > > On Sat, Dec 19, 2015 at 3:38 AM, Michael Denkowski < > michael.

Re: [Moses-support] Doubts on Multiple Decoding Paths

s the size of the phrase feature set by the number of models, so I recommend running mer-moses.pl with --batch-mira. Best, Michael On Fri, Dec 18, 2015 at 1:08 PM, Philipp Koehn <p...@jhu.edu> wrote: > Hi, > > that sounds right. > > The "union" option is fairly ne

Re: [Moses-support] BLEU score difference about 0.13 for one dataset is normal?

Hi Davood, If you're comparing two versions of the system to see what effect your work has on translation quality, you can run Jon Clark's MultEval (an implementation of the hypothesis testing described in the paper). From the BLEU differences you reported,

Re: [Moses-support] BLEU score difference about 0.13 for one dataset is normal?

Hi Davood, Optimizers like MERT will give you a slightly different result every time you run them, leading to variance in BLEU score. It's generally a good idea to use multiple optimizer runs, especially when comparing two systems. There's a good paper on hypothesis testing for MT that goes

Re: [Moses-support] Faster decoding with multiple moses instances

memory could use "--threads 2:11:10" to run 2 instances with 11 threads each and 1 instance with 10 threads. The script can be used with mert-moses.pl via the --multi-moses flag and --decoder-flags='--threads P:T:E'. Best, Michael On Tue, Oct 6, 2015 at 4:39 PM, Michael Denkowski < mic

Re: [Moses-support] Faster decoding with multiple moses instances

ould be curious to see how things work out in other users' systems. Best, Michael On Thu, Oct 8, 2015 at 2:34 PM, Vincent Nguyen <vngu...@neuf.fr> wrote: > out of curiosity, what gain do you get with 400 for both stack and cube > pruning ? > > > Le 08/10/2015 20:26, Michael De

Re: [Moses-support] Faster decoding with multiple moses instances

Hi Vincent, I'm using cube pruning with the following options for all data points: [search-algorithm] 1 [cube-pruning-deterministic-search] true [cube-pruning-pop-limit] 2000 [stack] 2000 Best, Michael ___ Moses-support mailing list

Re: [Moses-support] Faster decoding with multiple moses instances

19s user5m42.070s user5m35.132s user6m46.001s > user7m38.151s user9m6.500s 10m32.739s > > sys 0m38.039s sys 0m45.753s sys 0m44.117s sys 0m52.285s > sys 0m56.655s sys 1m6.749s 1m16.935s > > On 05/10/2015 16:05, Michael Denkowski wrote: >

[Moses-support] Faster decoding with multiple moses instances

Hi all, Like some other Moses users, I noticed diminishing returns from running Moses with several threads. To work around this, I added a script to run multiple single-threaded instances of moses instead of one multi-threaded instance. In practice, this sped things up by about 2.5x for 16 cpus

Re: [Moses-support] Faster decoding with multiple moses instances

> > great - that will be very useful. > > Since you just ran the comparison - do you have any numbers on "still > allowed everything to fit into memory", i.e., how much more memory is used > by running parallel instances? > > -phi > > On Mon, Oct 5, 2015 at

Re: [Moses-support] Faster decoding with multiple moses instances

1 5 10 15 20 25 Unpruned compact pt 143 42 32 38 > 52 62 probing pt 245 58 33 25 24 21 Pruned compact pt 119 24 15 10 10 10 > probing pt 117 25 25 10 10 10 > > Hieu Hoang > http://www.hoang.co.uk/hieu > > On 5 October 2015 at 15:15, Michael Denkowski <

Re: [Moses-support] [Moses-developers] Generation models with Mmsapt

t; > On 4 September 2015 at 03:49, Michael Denkowski < > michael.j.denkow...@gmail.com> wrote: > >> Hi Hieu, >> >> Yes, I have everything working together with the caveat about order in >> the moses.ini file (https://github.com/moses-smt/mosesdecoder/pull/124). &g

Re: [Moses-support] [Moses-developers] Generation models with Mmsapt

an you do it via dropbox/google drive > rather than the Moses github) > > > On 03/09/2015 05:13, Michael Denkowski wrote: > > Sounds good. I added a small test model to my branch: > <https://github.com/moses-smt/mosesdecoder/raw/mjdenkowski/mmsapt-factor-test.tar.gz> > htt

Re: [Moses-support] [Moses-developers] Generation models with Mmsapt

gt; The most reasonable fix for this in my opinion is to move pruning where it > belongs --- post all the factor generation stuff. > > Hieu is probably still the person with the best understanding of how > factored phrase table entry generation works, so maybe he can chime in on > th

Re: [Moses-support] processLexicalTableMin with more than 6 scores

message here: moses/TranslationModel/CompactPT/LexicalReorderingTableCompact.cpp:83 std::cerr LexScores: scores.size() std::endl; W dniu 15.05.2015 o 21:11, Michael Denkowski pisze: Thanks! Yes, telling moses what to do with the scores is my next task. I was checking to make sure

[Moses-support] processLexicalTableMin with more than 6 scores

Hi all, Has anyone successfully used a compact reordering model with extra score components? I added some features to a reordering table and ran processLexicalTableMin, which appeared to encode everything (at least by output file size inspection), but moses still seemed to think it had only 6

Re: [Moses-support] processLexicalTableMin with more than 6 scores

would need to provide a reasonable interpretation for that additional scores in the feature function itself. It probable gets that scores, but does not use them. Retrieval should just work unless I am missing something. W dniu 15.05.2015 o 20:35, Michael Denkowski pisze: Hi all, Has anyone

Re: [Moses-support] METEOR: difference between ranking task and other tasks

Hi Marcin, Meteor scores can vary widely across tasks due to the training data and goal. The default ranking task tries to replicate WMT rankings, so the absolute scores are not as important as the relative scores between systems. The adequacy task tries to fit Meteor scores to numeric adequacy