run this 1st
cat
/mnt/storage/Common/models/language_models/langmod.5gram.blm/mnt/storage/Common/models/translation_models/moses3_model/train/model/reordering-table.wbe-msd-bidirectional-fe.gz.*
> /dev/null
before running mosesserver. Is there a speedup? If there's speedup, then
it would suggest that IO is a problem
what happens if you use
moses
rather than
mosesserver
? if there's a speedup, then the mosesserver is not able to cope with
the number of connections
On 23/07/2015 18:19, Barry Haddow wrote:
> Hi Oren
>
> You can fit a lot of model in 50G RAM. It's worth looking at the
> compact phrase and reordering models, pruning options, and kenlm
> quantisation, and you might fit the models on one machine.
>
> As to the slowdown, a delay of 20s or so suggests that it's waiting on
> I/O, or perhaps xmlrpc-c is queueing up requests or connections.
>
> cheers - Barry
>
> On 23/07/15 15:06, Oren wrote:
>> More details...
>>
>> Our NFS is mounted at /mnt/storage .
>>
>> The command to run moses server is:
>>
>> /mnt/storage/Common/mosesdecoder3/bin/mosesserver -f
>> /mnt/storage/Common/models/translation_models/moses3_model/mert-work/moses.ini
>>
>> mark-unknown -xml-input -threads 18 exclusive
>>
>> Here is the complete moses.ini (excluding comments):
>>
>>> [input-factors]
>>> 0
>>>
>>> [mapping]
>>> 0 T 0
>>>
>>> [distortion-limit]
>>> 6
>>>
>>> [feature]
>>> UnknownWordPenalty
>>> WordPenalty
>>> PhrasePenalty
>>> PhraseDictionaryMemory name=TranslationModel0 num-features=4
>>> path=/mnt/storage/Common/models/translation_models/moses3_model/train/model/phrase-table.gz
>>>
>>> input-factor=0 output-factor=0
>>> LexicalReordering name=LexicalReordering0 num-features=6
>>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>>> path=/mnt/storage/Common/models/translation_models/moses3_model/train/model/reordering-table.wbe-msd-bidirectional-fe.gz
>>> Distortion
>>> KENLM lazyken=0 name=LM0
>>> path=/mnt/storage/Common/models/language_models/langmod.5gram.blmorder=5
>>>
>>> [threads]
>>> 6
>>>
>>> [weight]
>>>
>>> LexicalReordering0= 0.0268311 -0.0146878 0.0261305 0.0380759
>>> 0.0118265 0.07479
>>> Distortion0= 0.074665
>>> LM0= 0.0972206
>>> WordPenalty0= -0.18469
>>> PhrasePenalty0= -0.212528
>>> TranslationModel0= 0.0184105 0.091358 0.112618 0.0161682
>>> UnknownWordPenalty0= 1
>>
>> On Wednesday, July 22, 2015, Oren <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Yes, we use a lot of RAM in our setup. But the improved response
>> time justifies it.
>>
>> Our language is on a nfs, bit we've been working this way with
>> moses 1 for some time with no problems (ten different machines
>> using the same language model file over nfs). Same for the
>> reordering model. Neither of them is in memory. Raising these
>> models into memory will require raising our already excessive RAM
>> requirements...
>>
>> Thanks again for the help.
>>
>> On Wednesday, July 22, 2015, Barry Haddow
>> <[email protected]
>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote:
>>
>> Hi Oren
>>
>> I'm not aware of any threading problems with
>> PhraseDictionaryMemory, but not many people use it since it
>> takes up too much memory. Moses command line runs
>> multi-threaded using a thread pool.
>>
>> Is your language model on a local file system? Running it
>> over nfs can be bad. What about your reordering model? Are
>> you using the compact or the memory version?
>>
>> cheers - Barry
>>
>> On 22/07/15 15:09, Oren wrote:
>>> We have no swapping issues. I was asking if the use of
>>> an in-memory translation model might cause multithreading
>>> problems.
>>>
>>>
>>> I'm not sure how to replicate the problem on cmd moses,
>>> since it's purely a multithreading issue. Ca you run cmd
>>> moses multithreaded?
>>>
>>> I can't attach the complete moses.ini because it's on a
>>> separate network... But I copied below the stuff that looked
>>> relevant. I also tried to change the [threads] setting to
>>> 18, with no apparent effect.
>>>
>>> [input-factors]
>>> 0
>>>
>>> [mapping]
>>> 0 T 0
>>>
>>> [distortion-limit]
>>> 6
>>>
>>> [feature]
>>> UnknownWordPenalty
>>> WordPenalty
>>> PhrasePenalty
>>> PhraseDictionaryMemory name=TranslationModel0 num-features=4
>>> path=<path>/phrase-table.gz input-factor=0 output-factor=0
>>> LexicalReordering <parameters>
>>> Distortion
>>> KENLM lazyken=0 name=LM0 path=<path> order=5
>>>
>>> [threads]
>>> 6
>>>
>>> [weight]
>>>
>>> <weight parameters>
>>>
>>> On Tuesday, July 21, 2015, Hieu Hoang <[email protected]>
>>> wrote:
>>>
>>> is it possible you can make your moses.ini file
>>> available for us to see?
>>>
>>> do you know if the same problem occurs if you use the
>>> command line moses, rather than mosesserver?
>>>
>>>
>>> Hieu Hoang
>>> Researcher
>>> New York University, Abu Dhabi
>>> http://www.hoang.co.uk/hieu
>>>
>>> On 21 July 2015 at 18:07, Barry Haddow
>>> <[email protected]> wrote:
>>>
>>>
>>> On 21/07/15 14:51, Oren wrote:
>>>> I am using the in-memory mode, using about 50GB of
>>>> RAM. (No swap issues as far as I can tell.) Could
>>>> that cause issues?
>>>
>>> Yes, swapping would definitely cause issues - was
>>> that your question?
>>>
>>>
>>>>
>>>> I looked at the commit you linked to, but it
>>>> doesn't seem to be something configurable beyond
>>>> the -threads switch. Am I missing something?
>>>
>>> The commit enables you to set the maximum number of
>>> connections to be the same as the maximum number of
>>> threads.
>>>
>>>>
>>>> On Tuesday, July 21, 2015, Barry Haddow
>>>> <[email protected]> wrote:
>>>>
>>>> Hi Oren
>>>>
>>>> Does your host have 18 threads available? It
>>>> could also be that xmlrpc-c is limiting the
>>>> number of connections - this can now be configured:
>>>>
>>>> https://github.com/moses-smt/mosesdecoder/commit/b3baade7f022edbcea2969679a40616683f63523
>>>>
>>>> Slowdowns in Moses are often caused by disk
>>>> access bottlenecks. You can use --minphr-memory
>>>> and --minlexr-memory to make sure that the
>>>> phrase and reordering tables are loaded in to
>>>> memory, rather than being access on-demand.
>>>> Make sure your host has enough RAM and is not
>>>> swapping. As I mentioned before there are
>>>> various ways to make your models smaller
>>>> (http://www.statmt.org/moses/?n=Advanced.RuleTables),
>>>> which can make a big difference to speed
>>>> depending on your setup.
>>>>
>>>> cheers - Barry
>>>>
>>>> On 21/07/15 09:30, Oren wrote:
>>>>> Hi Barry,
>>>>>
>>>>> Thanks for the quick response.
>>>>>
>>>>> I added the switch "-threads 18" to the
>>>>> command to raise moses server. The slowness
>>>>> issue persists but in a different form. Most
>>>>> requests return right away, even under heavy
>>>>> load, but some requests (about 5%) take far
>>>>> longer - about 15-20seconds.
>>>>>
>>>>> Perhaps there are other relevant switches?
>>>>>
>>>>> Thanks again.
>>>>>
>>>>> On Monday, July 20, 2015, Barry Haddow
>>>>> <[email protected]> wrote:
>>>>>
>>>>> Hi Oren
>>>>>
>>>>> The threading model is different. In v1,
>>>>> the server created a new thread for every
>>>>> request, v3 uses a thread pool. Try
>>>>> increasing the number of threads.
>>>>>
>>>>> Also, make sure you use the compact phrase
>>>>> table and KenLM as they are normally
>>>>> faster, and pre-pruning your phrase table
>>>>> can help,
>>>>>
>>>>> cheers - Barry
>>>>>
>>>>> On 20/07/15 12:01, Oren wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> We are in the process of migrating from
>>>>>> Moses 1 to Moses 3. We have noticed a
>>>>>> significant slowdown when sending many
>>>>>> requests at once to Moses Server. The
>>>>>> first request will actually finish about
>>>>>> 25% faster that a single request using
>>>>>> Moses 1, but as more requests accumulate
>>>>>> there is a marked slowdown, until
>>>>>> requests take 5 times longer or more.
>>>>>>
>>>>>> Is this a known issue? Is it specific to
>>>>>> Moses Server? What can we do about it?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Oren.
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Moses-support mailing list
>>>>>> [email protected]
>>>>>>
>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Moses-support mailing list
>>>>> [email protected]
>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>
>>>
>>> The University of Edinburgh is a charitable body,
>>> registered in
>>> Scotland, with registration number SC005336.
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> [email protected]
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
>
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
--
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
