Re: [sqlite] Linux top command and sqlite

[Hick Gunter]

Definitely 'D' as in "Disk I/O", if you type "1" while runing top, the results 
for each CPU are displayed separately und you will find at least 1 line (not 
necessarily always the same line) with a significant percentage of "wa" (I/O 
wait state).

Apparently your transactions involve very much more data changes than 
computations. A common reason ist hat you are running. each. change. in. a. 
separate. transaction. As suggested, try grouping several (the optimal size 
typically ranges from 100s to 1000s) small updates into a single transaction so 
that all the changes hit the disk together instead of separately.

-Ursprüngliche Nachricht-
Von: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im 
Auftrag von Kevin O'Gorman
Gesendet: Dienstag, 21. Februar 2017 19:22
An: sqlite-users <sqlite-users@mailinglists.sqlite.org>
Betreff: [sqlite] Linux top command and sqlite

I'm not at all sure this is the right place to ask, but as it only comes up 
when I'm running one of my sqlite jobs, I thought I'd start here.  I'm running 
Python 3.5 scripts in Linux 16.04.1 using the sqlite3 package.  Machine is Core 
i5, 32GB RAM.

Some of my stuff takes a while to run, and I like to keep tabs on it.
Right now, I'm running one of those, and the Linux top command shows extremely 
small CPU usage, and a status ("S" column) of "D" which the man page defines as 
"uninterruptable sleep".  Huh?

My code does not use an intentional sleep at all.  It's traversing parts of a 
graph, and not interacting with anything else.  As far as I know, anyway, and I 
wrote the whole thing.

Now it's true that it's updating the consequences of changes to graph nodes 
that took about 3 hours just to install, so taking a while is not a surprise by 
itself.  I just wonder what that status means and how it could arise.

--
word of the year: *kakistocracy*
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___
 Gunter Hick
Software Engineer
Scientific Games International GmbH
FN 157284 a, HG Wien
Klitschgasse 2-4, A-1130 Vienna, Austria
Tel: +43 1 80100 0
E-Mail: h...@scigames.at

This communication (including any attachments) is intended for the use of the 
intended recipient(s) only and may contain information that is confidential, 
privileged or legally protected. Any unauthorized use or dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please immediately notify the sender by return e-mail message and 
delete all copies of the original communication. Thank you for your cooperation.


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Linux top command and sqlite

[Graham Holden]

 Original message From: Roger Binns <rog...@rogerbinns.com> 
Date: 21/02/2017  20:48  (GMT+00:00) To: SQLite mailing list 
<sqlite-users@mailinglists.sqlite.org> Subject: Re: [sqlite] Linux top command 
and sqlite 
On 21/02/17 10:22, Kevin O'Gorman wrote:
> Some of my stuff takes a while to run, and I like to keep tabs on it.
> Right now, I'm running one of those, and the Linux top command shows
> extremely small CPU usage, and a status ("S" column) of "D" which the man
> page defines as "uninterruptable sleep".  Huh?

In this case you can read D as waiting on disk.


According to this answer on StackOverflow:  
http://stackoverflow.com/a/6685764/2096401 spotting a "D" is relatively rare, 
although can be more common if NFS or similar is involved. No idea whether 
that's true or not, nor whether it applies to the OP.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Linux top command and sqlite

[Roger Binns]

On 21/02/17 10:22, Kevin O'Gorman wrote:
> Some of my stuff takes a while to run, and I like to keep tabs on it.
> Right now, I'm running one of those, and the Linux top command shows
> extremely small CPU usage, and a status ("S" column) of "D" which the man
> page defines as "uninterruptable sleep".  Huh?

In this case you can read D as waiting on disk.

Unless you are using multithreading, each query that needs to get data
not already in cache will result in disk activity.  And the code will
not continue until it gets the data from the disk.  The waiting is why
CPU usage is low.

You can use multithreading to allow more concurrent disk I/O although
Python's GIL complicates matters.  You can also tune the caches used by
SQLite.  If your database is smaller than your RAM then you can
prepopulate the OS cache with the database file contents.

Roger



signature.asc
Description: OpenPGP digital signature
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Linux top command and sqlite

[Kevin O'Gorman]

I'll try the synchronous=off the next time I run it.  As it happens, this
one
finished in 57 minutes, which is not bad, considering.

But, I wonder, could i not tell if it's doing lots of commits by looking at
the size of the ?.db-journal file?  It gets pretty large.


On Tue, Feb 21, 2017 at 10:38 AM, Richard Hipp  wrote:

> On 2/21/17, Kevin O'Gorman  wrote:
> > I'm not at all sure this is the right place to ask, but as it only comes
> up
> > when I'm running one of
> > my sqlite jobs, I thought I'd start here.  I'm running Python 3.5 scripts
> > in Linux 16.04.1 using the sqlite3 package.  Machine is Core i5, 32GB
> RAM.
> >
> > Some of my stuff takes a while to run, and I like to keep tabs on it.
> > Right now, I'm running one of those, and the Linux top command shows
> > extremely small CPU usage, and a status ("S" column) of "D" which the man
> > page defines as "uninterruptable sleep".  Huh?
>
> My guess:  It is busying doing an fsync() after a transaction commit.
> To find out, temporarily set "PRAGMA synchronous=off" in your script
> and see if that makes the pauses go away.
>
> A better long-term solution would be:
>
> (1) Group multiple changes into a single transaction using BEGIN...COMMIT.
> (2) Set PRAGMA journal_mode=WAL with PRAGMA synchronous=NORMAL.
>
>
> >
> > My code does not use an intentional sleep at all.  It's traversing parts
> of
> > a graph, and not interacting with anything else.  As far as I know,
> anyway,
> > and I wrote the whole thing.
> >
> > Now it's true that it's updating the consequences of changes to graph
> nodes
> > that took about 3 hours just to install, so taking a while is not a
> > surprise by itself.  I just wonder what that status means and how it
> could
> > arise.
> >
> > --
> > word of the year: *kakistocracy*
> > ___
> > sqlite-users mailing list
> > sqlite-users@mailinglists.sqlite.org
> > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> >
>
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
word of the year: *kakistocracy*
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Linux top command and sqlite

[Richard Hipp]

On 2/21/17, Kevin O'Gorman  wrote:
> I'm not at all sure this is the right place to ask, but as it only comes up
> when I'm running one of
> my sqlite jobs, I thought I'd start here.  I'm running Python 3.5 scripts
> in Linux 16.04.1 using the sqlite3 package.  Machine is Core i5, 32GB RAM.
>
> Some of my stuff takes a while to run, and I like to keep tabs on it.
> Right now, I'm running one of those, and the Linux top command shows
> extremely small CPU usage, and a status ("S" column) of "D" which the man
> page defines as "uninterruptable sleep".  Huh?

My guess:  It is busying doing an fsync() after a transaction commit.
To find out, temporarily set "PRAGMA synchronous=off" in your script
and see if that makes the pauses go away.

A better long-term solution would be:

(1) Group multiple changes into a single transaction using BEGIN...COMMIT.
(2) Set PRAGMA journal_mode=WAL with PRAGMA synchronous=NORMAL.


>
> My code does not use an intentional sleep at all.  It's traversing parts of
> a graph, and not interacting with anything else.  As far as I know, anyway,
> and I wrote the whole thing.
>
> Now it's true that it's updating the consequences of changes to graph nodes
> that took about 3 hours just to install, so taking a while is not a
> surprise by itself.  I just wonder what that status means and how it could
> arise.
>
> --
> word of the year: *kakistocracy*
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Linux top command and sqlite

[Kevin O'Gorman]

I'm not at all sure this is the right place to ask, but as it only comes up
when I'm running one of
my sqlite jobs, I thought I'd start here.  I'm running Python 3.5 scripts
in Linux 16.04.1 using the sqlite3 package.  Machine is Core i5, 32GB RAM.

Some of my stuff takes a while to run, and I like to keep tabs on it.
Right now, I'm running one of those, and the Linux top command shows
extremely small CPU usage, and a status ("S" column) of "D" which the man
page defines as "uninterruptable sleep".  Huh?

My code does not use an intentional sleep at all.  It's traversing parts of
a graph, and not interacting with anything else.  As far as I know, anyway,
and I wrote the whole thing.

Now it's true that it's updating the consequences of changes to graph nodes
that took about 3 hours just to install, so taking a while is not a
surprise by itself.  I just wonder what that status means and how it could
arise.

-- 
word of the year: *kakistocracy*
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users