Re: [sqlite] for what reason :memory: is much slower than /dev/shm/dummy.db
On 12/1/06, Isaac Raway <[EMAIL PROTECTED]> wrote: > Because our project needs to be ported to windows - the /dev/shm is not an > option - because win2000 does not support any temporary memory based file > system. Not so. "FILE: Ramdisk.sys sample driver for Windows 2000" http://support.microsoft.com/kb/257405 Even includes C code, along with a binary. A Google search reveals several other ones, some commercial, for 2K and XP. Try http://www.winsoft.sk/ramdisk.htm Just installed it and it works perfectly. Looks like it's $35 through SWREG. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] for what reason :memory: is much slower than /dev/shm/dummy.db
Because our project needs to be ported to windows - the /dev/shm is not an option - because win2000 does not support any temporary memory based file system. Not so. "FILE: Ramdisk.sys sample driver for Windows 2000" http://support.microsoft.com/kb/257405 Even includes C code, along with a binary. A Google search reveals several other ones, some commercial, for 2K and XP. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] for what reason :memory: is much slower than /dev/shm/dummy.db
Eduardo Morras wrote: At 09:34 01/12/2006, you wrote: Hi there, we are on an challanging project with very high requirements on performance. When doing some debugging we discover, that the sqlite method for creating an memory-based database is much slower than using e.g /dev/shm on linux or /tempfs on solaris. (We have measured an 20min performance advantage for the /dev/shm style on a batch run which takes 70min with :memory: and just 49min using /dev/shm. Because our project needs to be ported to windows - the /dev/shm is not an option - because win2000 does not support any temporary memory based file system. But beside that, we guess, that there will be a possiblity to tune :memory: or we belief, that we to something wrong when using :memory: (for example pragma page_size ...). Is there any body who can give us some advises to tune up our :memory: database to become as fast as the /dev/shm alternativ? Thanks roland On our project we desisted to use :memory: databases, only a ram disk file system. From time to time make a snapshot to hard disk or other persistent medium. In windows i suppouse you can make a ram disk using malloc and copy there your database file, set the pragma for temporary files to memory and disable journaling. Make a new io routines access based on windows, open/close, write/read etc... for access your memory malloc ram disk. Again, from time to time stop reads/writes to database and save it to disk. HTH You might find you can get the same performance in a simpler way by just disabling synchronous writes. Read Dr Hipp's explanation of a memory database for the reason. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] for what reason :memory: is much slower than /dev/shm/dummy.db
At 09:34 01/12/2006, you wrote: Hi there, we are on an challanging project with very high requirements on performance. When doing some debugging we discover, that the sqlite method for creating an memory-based database is much slower than using e.g /dev/shm on linux or /tempfs on solaris. (We have measured an 20min performance advantage for the /dev/shm style on a batch run which takes 70min with :memory: and just 49min using /dev/shm. Because our project needs to be ported to windows - the /dev/shm is not an option - because win2000 does not support any temporary memory based file system. But beside that, we guess, that there will be a possiblity to tune :memory: or we belief, that we to something wrong when using :memory: (for example pragma page_size ...). Is there any body who can give us some advises to tune up our :memory: database to become as fast as the /dev/shm alternativ? Thanks roland On our project we desisted to use :memory: databases, only a ram disk file system. From time to time make a snapshot to hard disk or other persistent medium. In windows i suppouse you can make a ram disk using malloc and copy there your database file, set the pragma for temporary files to memory and disable journaling. Make a new io routines access based on windows, open/close, write/read etc... for access your memory malloc ram disk. Again, from time to time stop reads/writes to database and save it to disk. HTH --- Scientists have shown that the moon is moving away at a tiny yet measurable distance from the earth every year. If you do the math, you can calculate that 85 million years ago the moon was orbiting the earth at a distance of about 35 feet from the earth's surface. This would explain the death of the dinosaurs. The tallest ones, anyway. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] for what reason :memory: is much slower than /dev/shm/dummy.db
John Stanton <[EMAIL PROTECTED]> wrote: > > I understand that Sqlite uses a less efficient algorithm for :memory > files than for the cacheing on regular files and that means lower > performance on those memory DBs. > It uses exactly the same algorithm. It is just that the hash tables are fixed sized and are designed for a cache, not for an entire database. -- D. Richard Hipp <[EMAIL PROTECTED]> - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] for what reason :memory: is much slower than /dev/shm/dummy.db
You might discover that a memory based database has few advantages over a disk based one, since Sqlite uses cacheing and the OS uses virtual memory file cacheing. The main difference might be initial accesses being slower while the cache fills up. I understand that Sqlite uses a less efficient algorithm for :memory files than for the cacheing on regular files and that means lower performance on those memory DBs. [EMAIL PROTECTED] wrote: Hi there, we are on an challanging project with very high requirements on performance. When doing some debugging we discover, that the sqlite method for creating an memory-based database is much slower than using e.g /dev/shm on linux or /tempfs on solaris. (We have measured an 20min performance advantage for the /dev/shm style on a batch run which takes 70min with :memory: and just 49min using /dev/shm. Because our project needs to be ported to windows - the /dev/shm is not an option - because win2000 does not support any temporary memory based file system. But beside that, we guess, that there will be a possiblity to tune :memory: or we belief, that we to something wrong when using :memory: (for example pragma page_size ...). Is there any body who can give us some advises to tune up our :memory: database to become as fast as the /dev/shm alternativ? Thanks roland - To unsubscribe, send email to [EMAIL PROTECTED] - - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] for what reason :memory: is much slower than /dev/shm/dummy.db
[EMAIL PROTECTED] wrote: Hi there, we are on an challanging project with very high requirements on performance. When doing some debugging we discover, that the sqlite method for creating an memory-based database is much slower than using e.g /dev/shm on linux or /tempfs on solaris. (We have measured an 20min performance advantage for the /dev/shm style on a batch run which takes 70min with :memory: and just 49min using /dev/shm. Because our project needs to be ported to windows - the /dev/shm is not an option - because win2000 does not support any temporary memory based file system. But beside that, we guess, that there will be a possiblity to tune :memory: or we belief, that we to something wrong when using :memory: (for example pragma page_size ...). Is there any body who can give us some advises to tune up our :memory: database to become as fast as the /dev/shm alternativ? Roland, There was a previous thread about the speed of :memory: databases that you may want to look for. Basically, SQLite3 is actually slightly slower using a :memory: database than an actual file on both Windows and Linux. This changed when the SQLite stared using the same pager for both memory and file based databases. If you want better memory performance, and don't need any of the newest features, you might want to look at using SQLite version 2 instead. It was significantly faster when using a memory based store since it had a separate, purpose designed, memory pager. HTH Dennis Cote - To unsubscribe, send email to [EMAIL PROTECTED] -
