John,
You seem pretty knowledgable regarding MMAP. I was wondering if you could help
me with this MMAP scenario:
I'm curious as to how the OS and multple processes interact regarding file
i/o and mmap.
Process A --- Writes to a file sequentially using either pwrite or kaio.
I would like to write a process B. That performs a read against what was
written by A.
I'm able to coordinate where to stop the read in other words I don't want to
read more than what has been written by A. Currently I'm just using os calls
to "read" but I thought that maybe MMAP might give better performance
especially if the OS would just provide the written buffers performed by
Process A to Process B's address space that is MMAPed.
Thanks for any guidance.
Ken
John Stanton <[EMAIL PROTECTED]> wrote: MMAP just lets you avoid one or two
layers of buffering and APIs. If
you were to use fopen/fread you go to function calls then open/read plus
buffering and function calls then to to the VM to actually access the
data. Going direct to the VM and getting a pointer to the VM pages is
more efficient. I got about 30% better speed out of one of my compilers
just by removing the reads and local buffering and using mmap. A b-tree
index almost doubled in speed by removing local reads and buffering and
using mmap.
Mitchell Vincent wrote:
> Hi John! Thanks for the reply!
>
> I think that makes a good point that the vm page fault is probably
> faster than the overhead of copying the data to a local buffer. So, page
> fault or not, I think that's the way I'm going to do it.
>
> Again, thanks very much for your input!
>
> On 6/12/07, John Stanton wrote:
>
>> Mitchell Vincent wrote:
>> > Working with some data conversion here (that will eventually go into
>> > an SQLite database). I'm hoping you IO wizards can offer some help on
>> > a question that I've been trying to get answered.
>> >
>> > I'm using Solaris 10 for this.
>> >
>> > If I mmap a large file and use madvise with MADV_SEQUENTIAL and
>> > MADV_WILLNEED, then start processing the file, when will the system
>> > discard pages that have been referenced? I guess what I'm wondering is
>> > if there is any retention of "back" pages?
>> >
>> > Say for example I start reading the file, and after consuming 24,576
>> > bytes, will the first or second pages still be in memory (assuming
>> > 8192 byte pages)?
>> >
>> > Thanks!
>> >
>> In general it means that the file is mapped into virtual memory. How
>> much of it remains in actual memory depends upon the memory demands on
>> the OS at the time. If the sequential and random advice is used by the
>> OS it is most likely to implement a look ahead for requential access.
>> Not all OS's pay attention to those advisory settings.
>>
>> What you are doing is to access the file as if it were an executing
>> program image. Similar rules apply.
>>
>> The answer is that you cannot assume that pages you have read are in
>> actual memory and you cannot assume that they are not. When you access
>> a page not currently in memory the OS will read it in and find space for
>> it somehow, maybe by discarding some other page.
>>
>> This is an excellent way to read files because you avoid one level of
>> buffer shadowing and get cacheing adjusted to currently available memory.
>>
>> -----------------------------------------------------------------------------
>>
>>
>> To unsubscribe, send email to [EMAIL PROTECTED]
>> -----------------------------------------------------------------------------
>>
>>
>>
>>
>
>
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
