You'll also get thrashing with memory-mapped files if you don't have
enough memory.
Advantage of the file API:
1. can access 2+GB files even running a 32 bit OS
2. OS portable
3. The sees the process as a small-memory process and won't be
tempted to kill it when it's running out of memory
Disadvantage:
1. Slower (by how much?)
On 25/05/2012 16:16, Kenneth Heafield wrote:
> I have heard people have new phrase table formats.
>
> The OnDiskPt format is a file accessed with file APIs, not memory
> mapping. Functionally, it uses the disk cache as shared memory (and the
> kernel shares the disk cache across processes). There is also some
> funny accounting going on because a process that depends on the disk
> cache is not charged for usage of that memory while a mmapped process
> would be. That means you can run Moses, it looks like it's fitting in
> virtual memory, and still thrash the disk because you also need enough
> disk cache to fit the entire phrase table. In this case, it is very
> slow despite the name OnDiskPt.
>
> Kenneth
>
> On 05/25/2012 10:57 AM, Lane Schwartz wrote:
>> Is there no current option to allow memory mapped phrase tables? I
>> thought that's what the binary phrase table was.
>>
>> Lane
>>
>>
>> On Fri, May 25, 2012 at 10:50 AM, Kenneth Heafield<[email protected]>
>> wrote:
>>> Use memory mapping (KenLM 8 or 9 on Linux, 9 on non-Linux, or IRSTLM
>>> with .mm) and the kernel takes care of shared memory for you.
>>>
>>> But there is merit to your argument e.g. different weights with the same
>>> phrase tables. Perhaps the answer is to make the phrase tables memory
>>> mapped. . .
>>>
>>> Kenneth
>>>
>>> On 05/25/2012 09:13 AM, Lane Schwartz wrote:
>>>> I could imagine if you were translating N languages, all into a common
>>>> target language, that it might be a memory footprint savings to be able
>>>> to do this all within a common process. The savings would be from being
>>>> able to have a single language model instance.
>>>>
>>>> Lane
>>>>
>>>> On Fri, May 25, 2012 at 2:00 AM, Philipp Koehn<[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> my understanding is that this is not currently possible.
>>>>
>>>> But why would you want to do this? If you translate with different
>>>> systems, why not just run different processes?
>>>>
>>>> The motivation to do this in the server process is that it avoids
>>>> keeping multiple server processes at the same time, which is not
>>>> a concern with batch Moses.
>>>>
>>>> -phi
>>>>
>>>> On Thu, May 24, 2012 at 12:55 AM, Fong Po Po
>>>> <[email protected]<mailto:[email protected]>> wrote:
>>>>
>>>> Dear all:
>>>> I have read page in
>>>> http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc22
>>>> This page say that Moses Server can run in multi
>>>> translation systems.
>>>> Can Traditional Moses (not Moses Server) also run in
>>>> multi translation systems?
>>>> Can you help me? Thanks!
>>>> Best Regards,
>>>> Fong Pui Chi
>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> [email protected]<mailto:[email protected]>
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> [email protected]<mailto:[email protected]>
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> When a place gets crowded enough to require ID's, social collapse is not
>>>> far away. It is time to go elsewhere. The best thing about space travel
>>>> is that it made it possible to go elsewhere.
>>>> -- R.A. Heinlein, "Time Enough For Love"
>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> [email protected]
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>> _______________________________________________
>>> Moses-support mailing list
>>> [email protected]
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
