Re: [Moses-support] Faster decoding with multiple moses instances

After many tests, as mentioned before I had made these changes in EMS score-settings = "--GoodTuring --MinScore 2:0.001" and pop limit cube pruning at 400 (instead of 5000 in EMS ) speed is much much higher (without impact on translation) Le 05/10/2015 17:20, Philipp Koehn a écrit : Hi,

[Moses-support] KenLM poison

no, suffix array PT, compact reordering, 5-gram KenLM) testing default > stack decoding vs cube pruning without and with the parallelization script > (+multi): > > --- > 1cpu sent/sec > stack 1.04 > cube 2.10 > --- > 16cpu sent/sec > stack 7.63 > +multi

Re: [Moses-support] Do debugging in the decoder?

Thanks a lot, Matthias and Hieu! I have the debug version in Eclipse already and can compiled it without errors. I could follow the debugging until to decoder(translation): pool.Submit(task); // in Exportinterface.cpp I didn't find a way to see what happen in the 'translation' task, e.g. how

Re: [Moses-support] (no subject)

Hello Sanjanasri, Basically, you can forget all results that you obtained without tuning. They are not a meaningful indicator of the quality of NPLM. If you add a new language model, the weight of the other language models, translation models etc. needs to be balanced accordingly, and that is

[Moses-support] Sixth Workshop on Patent and Scientific Literature Translation (PSLT 2015)

Dear Moses-support ML members, I introduce the Call for Participation of The 6th Workshop on Patent and Scientific Literature Translation (PSLT 2015) held at MT Summit XV. [Apologize for multiple copies] CALL FOR PARTICIPATION -- The 6th Workshop on Patent and Scientific Literature Translation

[Moses-support] KenLM poison

Dear all,I’m building the baseline system, and some error occurred during the last step of LM training process as the first attached file shows. I checked another case of “Last input should have been poison”, but that one has more detailed information “no space left on device”, while mine has

Re: [Moses-support] Do debugging in the decoder?

Hi Yuqi, I don't know. But maybe something like running a profiler on a small-scale setup and printing the call graph would be more convenient anyway? If you don't just want to try and read the source code right away. Maybe someone else has better suggestions. Cheers, Matthias On Mon,

[Moses-support] Faster decoding with multiple moses instances

Hi all, Like some other Moses users, I noticed diminishing returns from running Moses with several threads. To work around this, I added a script to run multiple single-threaded instances of moses instead of one multi-threaded instance. In practice, this sped things up by about 2.5x for 16 cpus

Re: [Moses-support] Faster decoding with multiple moses instances

Hi, great - that will be very useful. Since you just ran the comparison - do you have any numbers on "still allowed everything to fit into memory", i.e., how much more memory is used by running parallel instances? -phi On Mon, Oct 5, 2015 at 10:15 AM, Michael Denkowski <

Re: [Moses-support] (no subject)

Hi Sanjanasri, 1) your corpus is very small, and you may have to use more iterations of NPLM training and smaller vocabulary sizes. Just to double-check, are you tuning your systems? MERT (or PRO or MIRA) should normally ensure that adding a model doesn't make BLEU go down. 2) I'm not sure

Re: [Moses-support] KenLM poison

Hi, I'm still betting it's out of disk space writing the ARPA. Multithreaded exception handling is annoying. This is there to prevent deadlock. Kenneth On 10/05/2015 01:52 PM, 徐同学 wrote: > Dear all, > > I’m building the baseline system, and some error occurred during the > last step

Re: [Moses-support] Do debugging in the decoder?

You can use gdb to put breakpoints on the code and step through it. I personally use Eclipse+CDT to do my debugging, it's just a front end to gdb. You can see this video by Dominik to see how to set up Eclipse with moses https://vimeo.com/129306919 Hieu Hoang http://www.hoang.co.uk/hieu On 5

Re: [Moses-support] Faster decoding with multiple moses instances

what pt implementation did you use, and had it been pre-pruned so that there's a limit on how many target phrase for a particular source phrase? ie. don't have 10,000 entries for 'the' . I've been digging around multithreading in the last few weeks. I've noticed that the compact pt is VERY bad at

Re: [Moses-support] Faster decoding with multiple moses instances

Hi Philipp, Unfortunately I don't have a precise measurement. If anyone knows of a good way to benchmark a process tree with lots of memory mapping the same files, I would be glad to run it. --Michael On Mon, Oct 5, 2015 at 10:26 AM, Philipp Koehn wrote: > Hi, > > great - that

Re: [Moses-support] Faster decoding with multiple moses instances

https://github.com/kpu/usage This injects code into shared executables that makes them print usage statistics on termination to stderr. grep stderr, collate. Kenneth On 10/05/2015 04:05 PM, Michael Denkowski wrote: > Hi Philipp, > > Unfortunately I don't have a precise measurement. If anyone

Re: [Moses-support] Faster decoding with multiple moses instances

Hi Hieu, I'm using the memory mapped suffix array phrase table (PhraseDictionaryBitextSampling). I can run a test with compact PT as well. --Michael On Mon, Oct 5, 2015 at 10:48 AM, Hieu Hoang wrote: > what pt implementation did you use, and had it been pre-pruned so

Re: [Moses-support] Faster decoding with multiple moses instances

Hi Hieu That's exactly why I took to pre-pruning the phrase table, as I mentioned on Friday. I had something like 750,000 translations of the most common word, and it took half-an-hour to get the first sentence translated. cheers - Barry On 05/10/15 15:48, Hieu Hoang wrote: what pt

Re: [Moses-support] Faster decoding with multiple moses instances

Hi, with regard to pruning --- the example EMS config files have [TRAINING] score-settings = "--GoodTuring --MinScore 2:0.0001" which carries out threshold pruning during phrase table construction, going a good way towards avoiding too many translation options per phrase. -phi On Mon, Oct 5,

Re: [Moses-support] Faster decoding with multiple moses instances

Very bad unpruned and with mulithreading! :) Is this with the nonblockpt branch? I am slowly running out of ideas what might be the cause of this. Frequent vector realloaction? On 05.10.2015 16:48, Hieu Hoang wrote: > what pt implementation did you use, and had it been pre-pruned so that >