Re: [sqlite] Optimizations to recover early from D state of the process

[Simon Slavin]

On 19 Oct 2016, at 2:22pm, Rutuja Shah  wrote:

> I observed that the process goes into D state and takes
> around 5 to 15 minutes to recover.

There is nothing in SQLite that could put a process in TASK_UNINTERRUPTIBLE 
mode.  SQLite doesn't know much about your storage system and certainly doesn't 
try to talk to such a low level of your operating system.  It seems possible 
that you have a hardware fault or a bug in your storage system.

Are you storing your data on a disk inside the computer running SQLite ?  Or is 
it using some kind of storage across a network ?

Are you calling the SQLite C API directly or are you using a SQLite library ?

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Optimizations to recover early from D state of the process

[Clemens Ladisch]

Rutuja Shah wrote:
> the process goes into D state and takes around 5 to 15 minutes to recover.

There appears to be something wrong with your storage system.
Are you using a local disk, without RAID, not in a VM?


Regards,
Clemens
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Optimizations to recover early from D state of the process

[Rutuja Shah]

Hi,
Requirement 1:
Data dumped to the DB every 15 minutes. One table per day, data is
stored for the last 7 days. 2 such tables needed.
Total number of tables - 7 days * 2 tables = 14
Average number of rows inserted per transaction (per 15 minutes per
table) - 1,20,000

Requirement 2:
Data dumped to the DB every 15minutes. One table to store data for 1 hour.
Total number of tables to store data for last 7 days - 7 * 24 = 168
Average number of rows inserted per transaction (per 15 minutes) - 30,00,000

Version sqlite-3.8.8.1, WAL mode and SYNCHRONOUS=NORMAL

I wrote a C program which performs the above operations every 15
minutes. I observed that the process goes into D state and takes
around 5 to 15 minutes to recover. On seeing the strace of the
process, I observed multiple file operations assuming that data is
being dumped into the database file. Also, I have created index at the
time of table creation.

Are there any optimizations to reduce the time the process is in D
state? Should I consider sqlite or move to some other DB for such
heavy write operations?

Thanks
Rutuja
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users