There is an important element to bear in mind when using mmaping to take
advanrage of physical memory. The POSIX advisory file locks are not
necessarily effective in that mode.
Joe Wilson wrote:
Sure, pre-caching is a hack - but a very effective one.
It has also been suggested to use MAP_POPULATE for mmap() and
posix_fadvise() in Linux.
http://www.mail-archive.com/monotone-devel@nongnu.org/msg03222.html
The general reaction against pre-caching (especially for tiny 100M
databases) is amusing to me. Some programmers would rather wait 3 times
longer for a database operation while the majority of the RAM in their
machine goes unused. But if the OS did the same pre-cache thing behind
their back without their knowledge - they would be good with that.
--- Scott Hess <[EMAIL PROTECTED]> wrote:
An interesting approach would be to use some sort of async I/O
facility to implement read-ahead.
Short of that, I have found that in some cases, on some operating
systems, implementing explicit read-ahead buffering for fts2 segment
merges improves performance when the disk caches are cold. Linux
kernel 2.6 seems to get no benefit, 2.4 gets more. This is somewhat
of a special case, though, as fts2 segment merges are merging streams
from different locations together, like an external sort.
-scott
On 3/26/07, Joe Wilson <[EMAIL PROTECTED]> wrote:
--- Joe Wilson <[EMAIL PROTECTED]> wrote:
improved dramatically. So I attempted the creation of the index off hours on
the production system, and after 4 hours no index. I can't detect any
activity at all. The journal file and the .db file just sit at the same size
for 4 hours. Why is this failing? It seems like it is just sitting there
doing nothing. When I created the test index, I noticed the journal file
changing and the .db file changing during the 2.5 hours to create. On the
production .db file, nothing is happening. I have all associated processes
killed that ineract with the db file, so I know it is not locked.
I assume that the copied "test" database was indexed immediately after its
creation. If this was the case then the entire file may have been in the OS
cache resulting in very quick indexing. Try running "wc prod.db" or
"cat prod.db >/dev/null" and then creating the indexes on prod.db to see
what happens.
The original poster confirmed that cat'ting the file to /dev/null reduced index
creation time to 2.5 hours on the original database file.
Could some optional heuristic be incorporated into SQLite's pager to do
something
similar for such large transactions and/or queries?
____________________________________________________________________________________
Don't get soaked. Take a quick peek at the forecast
with the Yahoo! Search weather shortcut.
http://tools.search.yahoo.com/shortcuts/#loc_weather
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
