Michael Ruck wrote:
My experience differs on this one. When processing large amounts of data
with a processing function, which takes longer than the I/O using a mapped
file is faster than sequential I/O, as the Windows memory manager and
caching system will pull in the I/O pages asynchronously. At least on
Windows I'd say using mapped files for really large data transfers is
reasonable and most likely faster, than using sequential I/O using any other
method.
Mike
-----Ursprüngliche Nachricht-----
Von: Teg [mailto:[EMAIL PROTECTED]
Gesendet: Sonntag, 16. April 2006 16:55
An: John Stanton
Cc: sqlite-users@sqlite.org
Betreff: Re[2]: [sqlite] MMAP
In my experience under windows, using memory mapped files is no faster than
opening and reading the file into memory then processing it. The limiting
factor is the disk drives. I use both methods and the choice of method is
always a tossup. If the API I'm using will accept an entire buffer for the
file, say a JPG decode, I'll use mmap (windows equivalent) otherwise I may
just read it a chunk at a time.
My impression is, he's asking a general MMAP versus fopen question and not
planning on integrating MMAP into SQLite.
C
Sunday, April 16, 2006, 10:00:57 AM, you wrote:
JS> John Stanton wrote:
>> I wonder if members can help me with some advice. I have a program >>
which is a multi-threaded application server with Sqlite embedded which >>
runs on Unix and Windows. For an i/o buffer per thread I have the idea >>
of using a mmap'd file so that it can be transferred using >>
sendfile/TransmitFile with minimum overhead and no buffer shadowing.
>> The program design is such that there is a pool of inactive threads >>
waiting for a connection, and that pool could be quite large. Each one >>
would have a mmap'd file with an open fd involved.
>>
>> Does anyone know the performance impact of having a considerable number
of mmap'd files attached to a single process in Unix/Linux and Windows?
>> Has anyone tried such a strategy? My guess is that the impact is not
>> great, but I have no evidence in support.
>> JS
>>
JS> You have a lot of work ahead of you if you want to modify SQLite to
JS> use MMAP. SQLite was not designed for this. What are you trying to
JS> accomplish?
JS> I have no idea what the performance implications of using MMAP are.
JS> The only way to know is to try it and see.
JS> ===
JS> Thankyou for the response. I am not modifying Sqlite, just
JS> embedding it in an application server, and looking at ways to optimally
construct the
JS> server. It seems to me ideal to use zero buffering capabilities
JS> of the host OS if possible and to use a mmap'd file as a buffer and
JS> hold its fd so that it can be sent to the network without further
JS> buffer copying. Since it is difficult to set up realistic tests to
JS> simulate high volumes and loads I was curious if others had tried
JS> this approach and experienced any unexpected side effects in practice.
JS> BTW, Sqlite fits very neatly into such an environment. This
JS> particular application has hundreds of databases, each with a
JS> handful of users, a situation where Sqlite's single file simplicity
JS> is a great asset and no impediment to high performance. By having a
JS> purpose-designed server and application specific language plus the
JS> unbloated Sqlite I am looking to achieve higher performance and
JS> efficiency than by cobbling it together from some general purpose
language processor and generic web server.
JS> The recent discussion about locality of reference made me cautious.
JS> My concern is about having local memory references in the program
JS> held up by thrashing in virtual memory and losing more than would be
JS> gained by the more efficient sendfile type transfer.
JS> The system I am replacing was put together on the basis that there
JS> were plenty of resources to compensate for each added inefficiency,
JS> until one day it did matter - the camel's back eventually broke.
JS> JS
--
Best regards,
Teg mailto:[EMAIL PROTECTED]
I use mmap'd files extensively to avoid buffer shadowing and see some
big performance gains. Instead of having to read into a buffer or
allocate stack or heap and write it, you get a pointer to access that
the virtual memory area and avoid reads and writes. I also use
anonymous mapping as an alternative to heap allocation because it
bypasses the malloc-free snakepit.
Thankyou everyone who answered.
JS
