Pavel,
Thank you for this information. It is exactly what I was looking for.
That said if you are absolutely sure that no other process will write
to the database while you are reading you can create your own VFS and
make xAccess and xCheckReservedLock methods a no-op. You can read more
about
a lock going into it.
Thank you all once again for your help,
Drew
Message: 17
Date: Fri, 12 Aug 2011 19:28:33 +0100
From: Simon Slavin slav...@bigfraud.org
Subject: Re: [sqlite] Read only scaling optimization
To: General Discussion of SQLite Database sqlite-users@sqlite.org
Message-ID: b4050f5b
...@bigfraud.org
Subject: Re: [sqlite] Read only scaling optimization
To: General Discussion of SQLite Database sqlite-users@sqlite.org
Message-ID: b4050f5b-3edb-4b79-a58c-3788d971c...@bigfraud.org
Content-Type: text/plain; charset=us-ascii
On 12 Aug 2011, at 7:01pm, Drew Kozicki wrote:
I have
2011/8/12 Pavel Ivanov paiva...@gmail.com:
It's a little surprising to me that with all the same conditions 2
files residing on the same drive have better performance than the same
files residing on different drives. Theoretically that shouldn't
happen.
Yes, it's not right behaviour. Is
Alright so to sum up all the testing here it goes.
I have a Driver doing this pulling in 32 queries aimed at randomness and
different tables, much like that would be experienced in typical usage. Best
performance comes from having 2 separate programs running on 2 separate
files.
Single file
On 12 Aug 2011, at 7:01pm, Drew Kozicki wrote:
I have a Driver doing this pulling in 32 queries aimed at randomness and
different tables, much like that would be experienced in typical usage. Best
performance comes from having 2 separate programs running on 2 separate
files.
I'm no expert,
I have a Driver doing this pulling in 32 queries aimed at randomness and
different tables, much like that would be experienced in typical usage. Best
performance comes from having 2 separate programs running on 2 separate
files.
I'm no expert, but that suggests to me that your bottleneck is
I'll try this.
Unfortunately reading from disk appears not to be the problem. Even at 32
threads the IO appears to be very minimal. Our inability to scale appears to
be caused by a mutex in the caching. My CPU usage is at 30% and my disk is
near silent watching the performance monitor. This is on
On 11 Aug 2011, at 9:51pm, Drew Kozicki wrote:
Unfortunately reading from disk appears not to be the problem. Even at 32
threads the IO appears to be very minimal. Our inability to scale appears to
be caused by a mutex in the caching. My CPU usage is at 30% and my disk is
near silent
Look at disc transfer. With 16 queries in one thread on single disc,
disc may be accesed linearly (depeding on query). With 16 threads
accesing disc at the same time linear disc access is impossible
(however os may do some prefetching) and queries will by slower. You
may try increasing page size -
2011/8/10 Wiktor Adamski bardzotajneko...@interia.pl:
You
may try increasing page size - bigger block means less near-random
reads from the disc.
It's good way. With page size 8k instead of default 1k selects performance
may increasing ~3x. Note: PostgreSQL use 8k disk pages.
--
Best
To Answer several questions at once.
Simon,
Just checking: by 'queries' you mean 'SELECT', right ? You're not making
changes, just searching
Yes to optimize we average about 5-6 indexes per table.
D. Richard Hipp,
Open a separate database connection for each thread. Don't try to use the
same
We have an application that does not scale well – especially 8 and 16
threads we are getting about 10% at best to -50% at worst (2 and 4 are okay
aprox. 40-50%).
We have narrowed the problem down to the SQLite code.
Looking for best practices from anyone that overcame scaling issues.
First
On 9 Aug 2011, at 9:46pm, Drew Kozicki wrote:
The application is a C++ app. that accesses all the databases in a read
only style. Size of database range from 1MB to 10GB.
Each thread does the following in a loop
1. Grab a record from an external system (not SQLite)
2. Runs several
On Tue, Aug 9, 2011 at 4:46 PM, Drew Kozicki drewkozi...@gmail.com wrote:
We have an application that does not scale well – especially 8 and 16
threads we are getting about 10% at best to -50% at worst (2 and 4 are okay
aprox. 40-50%).
Open a separate database connection for each thread.
Hello Drew,
Why multiple threads? What kind of performance do you get if you only
use a single thread?
Is it one thread per database perhaps?
C
Tuesday, August 9, 2011, 4:46:13 PM, you wrote:
DK We have an application that does not scale well especially 8 and 16
DK threads we are getting
16 matches
Mail list logo
