Re: [sqlite] SQLite really SLOW on Solaris?

[Simon Slavin]

On 21 Nov 2010, at 11:46am, Kees Nuyt wrote:

>  wrote:
> 
>> Hi Simon,
>> 
>> Yup, on Solaris I ran this and it was 1000 milliseconds for the following:
>> 
>> int theReturn = sqlite3_sleep(1);
>> 
>> Any way you know I can make it sleep on Solaris for 1 millisecond? Any other
>> C tricks you know of?
> 
> Standard C Library Functions   usleep(3C)
> 
> NAME
> usleep - suspend execution for interval in microseconds

Good note.  But the original question remains: why does the SQLite VM under 
Solaris not implement sqlite3_sleep() with a resolution of at least 
milliseconds ?  It's not like Solaris is hampered by requirements for a tiny 
library !

I found some code in 'os_unix.c' that says

static int unixSleep(sqlite3_vfs *NotUsed, int microseconds){
#if OS_VXWORKS
[...]
#elif defined(HAVE_USLEEP) && HAVE_USLEEP

which suggests that OS_VXWORKS and HAVE_USLEEP are consulted to work out what 
resolution is available.  But then we descend into compile-time parameters and 
I don't know anything about them.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite really SLOW on Solaris?

[Lynton Grice]

Thanks Kees, much appreciated ;-)

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Kees Nuyt
Sent: 21 November 2010 01:47 PM
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] SQLite really SLOW on Solaris?

On Sun, 21 Nov 2010 13:08:31 +0200, "Lynton Grice"
<lynton.gr...@logosworld.com> wrote:

>Hi Simon,
>
>Yup, on Solaris I ran this and it was 1000 milliseconds for the following:
>
>int theReturn = sqlite3_sleep(1);
>
>Any way you know I can make it sleep on Solaris for 1 millisecond? Any
other
>C tricks you know of?
>

Standard C Library Functions   usleep(3C)

NAME
 usleep - suspend execution for interval in microseconds

SYNOPSIS
 #include 

 int usleep(useconds_t useconds);

DESCRIPTION
 The usleep() function suspends the caller from execution for
 the  number  of microseconds specified by the useconds argu-
 ment.  The  actual  suspension  time  might  be  less   than
 requested  because any caught signal will terminate usleep()
 following execution of that signal's catching  routine.  The
 suspension  time  might be longer than requested by an arbi-
 trary amount because of the scheduling of other activity  in
 the system.

 If the value of useconds is 0, then the call has no effect.

 The use of the usleep() function has no effect on the action
 or  blockage of any signal. In a multithreaded process, only
 the invoking thread is suspended from execution.

RETURN VALUES
 On completion,  usleep()  returns  0.  There  are  no  error
 returns.

ERRORS
 No errors are returned.

USAGE
 The usleep() function is included for its historical  usage.
 The nanosleep(3C) function is preferred over this function.

-- 
  (  Kees Nuyt
  )
c[_]
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite really SLOW on Solaris?

[Kees Nuyt]

On Sun, 21 Nov 2010 13:08:31 +0200, "Lynton Grice"
 wrote:

>Hi Simon,
>
>Yup, on Solaris I ran this and it was 1000 milliseconds for the following:
>
>int theReturn = sqlite3_sleep(1);
>
>Any way you know I can make it sleep on Solaris for 1 millisecond? Any other
>C tricks you know of?
>

Standard C Library Functions   usleep(3C)

NAME
 usleep - suspend execution for interval in microseconds

SYNOPSIS
 #include 

 int usleep(useconds_t useconds);

DESCRIPTION
 The usleep() function suspends the caller from execution for
 the  number  of microseconds specified by the useconds argu-
 ment.  The  actual  suspension  time  might  be  less   than
 requested  because any caught signal will terminate usleep()
 following execution of that signal's catching  routine.  The
 suspension  time  might be longer than requested by an arbi-
 trary amount because of the scheduling of other activity  in
 the system.

 If the value of useconds is 0, then the call has no effect.

 The use of the usleep() function has no effect on the action
 or  blockage of any signal. In a multithreaded process, only
 the invoking thread is suspended from execution.

RETURN VALUES
 On completion,  usleep()  returns  0.  There  are  no  error
 returns.

ERRORS
 No errors are returned.

USAGE
 The usleep() function is included for its historical  usage.
 The nanosleep(3C) function is preferred over this function.

-- 
  (  Kees Nuyt
  )
c[_]
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite really SLOW on Solaris?

[Lynton Grice]

Hi Simon,

Yup, on Solaris I ran this and it was 1000 milliseconds for the following:

int theReturn = sqlite3_sleep(1);

Any way you know I can make it sleep on Solaris for 1 millisecond? Any other
C tricks you know of?

Thanks ;-)

Lynton

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Simon Slavin
Sent: 21 November 2010 12:23 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] SQLite really SLOW on Solaris?


On 21 Nov 2010, at 10:10am, Lynton Grice wrote:

> in my "queue implementation" I add in a:
> 
> sqlite3_sleep(1);
> 
> Essentially I wanted to sleep for "1 millisecond" between sends to the
queue
> (in the send function itself), so that if the sender is sending too fast
it
> allows receivers to get messages 
> 
> On Debian it sleeps for 1 millisecond, BUT on the Solaris VM it sleeps for
1
> second!!! Not one millisecond..why???

See

<http://www.sqlite.org/c3ref/sleep.html>

Apparently the Solaris implementation of xSleep() is weird.  Just out of
interest, do

int theReturn = sqlite3_sleep(1);

(or some equivalent) on both platforms and look at the values you get
returned.  Do you get 1 on one platform and 1000 on the other ?

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite really SLOW on Solaris?

[Simon Slavin]

On 21 Nov 2010, at 10:10am, Lynton Grice wrote:

> in my "queue implementation" I add in a:
> 
> sqlite3_sleep(1);
> 
> Essentially I wanted to sleep for "1 millisecond" between sends to the queue
> (in the send function itself), so that if the sender is sending too fast it
> allows receivers to get messages 
> 
> On Debian it sleeps for 1 millisecond, BUT on the Solaris VM it sleeps for 1
> second!!! Not one millisecond..why???

See



Apparently the Solaris implementation of xSleep() is weird.  Just out of 
interest, do

int theReturn = sqlite3_sleep(1);

(or some equivalent) on both platforms and look at the values you get returned. 
 Do you get 1 on one platform and 1000 on the other ?

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite really SLOW on Solaris?

[Lynton Grice]

Hi Eric, Hi Roger,

Interesting, I found the issue with the "slow solaris issue".

You see in my "queue implementation" I add in a:

sqlite3_sleep(1);

Essentially I wanted to sleep for "1 millisecond" between sends to the queue
(in the send function itself), so that if the sender is sending too fast it
allows receivers to get messages 

On Debian it sleeps for 1 millisecond, BUT on the Solaris VM it sleeps for 1
second!!! Not one millisecond..why???

Why would that be? Could it be that the OS clocks are out? The moment I took
this out of the code on Solaris it was super fast just like it was on
Debian.

Any ideas?

Lynton

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Eric Smith
Sent: 20 November 2010 09:13 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] SQLite really SLOW on Solaris?

Lynton Grice wrote:

> Many thanks for your feedback, much appreciated ;-)
> 
> But why would that happen on Solaris and not Debian?

Did you try it on real OSs & not VMs?  Are you sure the host OS is
giving the same compute & disk slices to the VMs?  Any other differences
in those mappings?

Are the VMs running on the same host box?  If not, are they running on
the same hardware and host OS?

Along the lines of Roger's comment, some VM implementations have 
OS clocks that simply don't work right -- they run too fast or too slow 
(I have personally observed differences of up to 3x), because they 
assume they are on real hardware.  Have you tested that?

Did you try compiling it with profiling data turned on to see where the
bottlenecks are (using gcc's -pg flag plus gprof(1))?

Or, as a simpler test, you could run your app from strace(1) (assuming
solaris has such a thing) to see if you are spending loads of time in
system calls as Roger hypothesized.

> Also, how would I pass the -D flags? Do I do that when I compile SQLite?

Yes -- those are arguments to GCC that are similar to writing a #define 
compiler directive into your code.  So saying

gcc -DFOO=bar ... 

would be like writing

#define FOO bar

into the code.

The sqlite.org docs describe what they do pretty well iirc.

Eric

--
Eric A. Smith

You can observe a lot by watching.
-- Yogi Berra
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite really SLOW on Solaris?

[Lynton Grice]

Hi Eric,

I will be trying it on real machines next week but for now I have one Debian
and one Solaris VM on my laptop, each allocated the same memory etc

I will try profiling what is really going on and will let you know, but I
hope it is just a VM thing

Thanks ;-)

Lynton

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Eric Smith
Sent: 20 November 2010 09:13 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] SQLite really SLOW on Solaris?

Lynton Grice wrote:

> Many thanks for your feedback, much appreciated ;-)
> 
> But why would that happen on Solaris and not Debian?

Did you try it on real OSs & not VMs?  Are you sure the host OS is
giving the same compute & disk slices to the VMs?  Any other differences
in those mappings?

Are the VMs running on the same host box?  If not, are they running on
the same hardware and host OS?

Along the lines of Roger's comment, some VM implementations have 
OS clocks that simply don't work right -- they run too fast or too slow 
(I have personally observed differences of up to 3x), because they 
assume they are on real hardware.  Have you tested that?

Did you try compiling it with profiling data turned on to see where the
bottlenecks are (using gcc's -pg flag plus gprof(1))?

Or, as a simpler test, you could run your app from strace(1) (assuming
solaris has such a thing) to see if you are spending loads of time in
system calls as Roger hypothesized.

> Also, how would I pass the -D flags? Do I do that when I compile SQLite?

Yes -- those are arguments to GCC that are similar to writing a #define 
compiler directive into your code.  So saying

gcc -DFOO=bar ... 

would be like writing

#define FOO bar

into the code.

The sqlite.org docs describe what they do pretty well iirc.

Eric

--
Eric A. Smith

You can observe a lot by watching.
-- Yogi Berra
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite really SLOW on Solaris?

[Eric Smith]

Lynton Grice wrote:

> Many thanks for your feedback, much appreciated ;-)
> 
> But why would that happen on Solaris and not Debian?

Did you try it on real OSs & not VMs?  Are you sure the host OS is
giving the same compute & disk slices to the VMs?  Any other differences
in those mappings?

Are the VMs running on the same host box?  If not, are they running on
the same hardware and host OS?

Along the lines of Roger's comment, some VM implementations have 
OS clocks that simply don't work right -- they run too fast or too slow 
(I have personally observed differences of up to 3x), because they 
assume they are on real hardware.  Have you tested that?

Did you try compiling it with profiling data turned on to see where the
bottlenecks are (using gcc's -pg flag plus gprof(1))?

Or, as a simpler test, you could run your app from strace(1) (assuming
solaris has such a thing) to see if you are spending loads of time in
system calls as Roger hypothesized.

> Also, how would I pass the -D flags? Do I do that when I compile SQLite?

Yes -- those are arguments to GCC that are similar to writing a #define 
compiler directive into your code.  So saying

gcc -DFOO=bar ... 

would be like writing

#define FOO bar

into the code.

The sqlite.org docs describe what they do pretty well iirc.

Eric

--
Eric A. Smith

You can observe a lot by watching.
-- Yogi Berra
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite really SLOW on Solaris?

[Lynton Grice]

Hi Roger,

Many thanks for your feedback, much appreciated ;-)

But why would that happen on Solaris and not Debian?

Also, how would I pass the -D flags? Do I do that when I compile SQLite?

Thanks again

Lynton

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Roger Binns
Sent: 20 November 2010 07:31 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] SQLite really SLOW on Solaris?

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 11/20/2010 07:12 AM, Lynton Grice wrote:
> On Debian it processes a couple hundred messages per second, on Solaris
> using the SAME code it does like ONE message per second.

One possible cause is the busy handler but this only applies if you have
concurrent access to the file and have set a busy timeout.

The default SQLite busy handler keeps trying again and if it fails sleeps,
repeating the process.  It will sleep for longer and longer periods until
the overall busy timeout period has expired.

If you do not provide any -D flags then the call used to do the sleeping has
a granularity of one second - ie even if SQLite wants to sleep for 50ms it
will sleep for one second.  If you have the usleep() function and supply
- -DHAVE_USLEEP then microsecond resolution sleeps are available.

These are the sleep intervals in milliseconds that sqliteDefaultBusyCallback
tries when USLEEP is available:

  { 1, 2, 5, 10, 15, 20, 25, 25,  25,  50,  50, 100 }

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkzoBeIACgkQmOOfHg372QTXHQCfQ3HIoXF5FcqP9AttZ7gNdPEZ
IKYAoMVuIqoSa4e08nsbVQk5ABWZdq1G
=+TOO
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite really SLOW on Solaris?

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 11/20/2010 07:12 AM, Lynton Grice wrote:
> On Debian it processes a couple hundred messages per second, on Solaris
> using the SAME code it does like ONE message per second.

One possible cause is the busy handler but this only applies if you have
concurrent access to the file and have set a busy timeout.

The default SQLite busy handler keeps trying again and if it fails sleeps,
repeating the process.  It will sleep for longer and longer periods until
the overall busy timeout period has expired.

If you do not provide any -D flags then the call used to do the sleeping has
a granularity of one second - ie even if SQLite wants to sleep for 50ms it
will sleep for one second.  If you have the usleep() function and supply
- -DHAVE_USLEEP then microsecond resolution sleeps are available.

These are the sleep intervals in milliseconds that sqliteDefaultBusyCallback
tries when USLEEP is available:

  { 1, 2, 5, 10, 15, 20, 25, 25,  25,  50,  50, 100 }

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkzoBeIACgkQmOOfHg372QTXHQCfQ3HIoXF5FcqP9AttZ7gNdPEZ
IKYAoMVuIqoSa4e08nsbVQk5ABWZdq1G
=+TOO
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] SQLite really SLOW on Solaris?

[Lynton Grice]

Hi there,

 

I compiled SQLite on Debain (VM) first using the following:

 

/usr/local/bin/gcc -c -fPIC sqlite3.c

/usr/local/bin/gcc -shared -o libsqlite3.so sqlite3.o

 

And then I created my own shared lib using SQLite and when I use the lib on
Debian it is SUPER FAST.

 

Then I installed SQLIte on Solaris (VM) using the same compile commands
above. I copied my shared lib code from Debian to Solaris and ran the same
"test code".and it is SO SLOWlike ridiculously slow.

 

On Debian it processes a couple hundred messages per second, on Solaris
using the SAME code it does like ONE message per second.

 

What could be wrong here? Must I change the compile options on Solaris? 

 

Any ideas?

 

Chat later 

 

Lynton 

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users