Hi,
Really hope someone out there can help me with this.
I've written a C++ cache class which extracts ad-hoc data from a Sybase
database and inserts it into an in-memory SQLite database. This is working
fine but is very slow during the insert data phase. To give some
quantitative idea of what I mean by slow:
a) If I run the extract with the Sybase isql command line utility and dump
that to a file it takes 2/3 minutes (~800,000 rows, 100MB of data).
b) If I run my cache class with the insert into SQLite commented out it
takes 2/3 minutes.
c) Uncommenting the insert statement, the extract/insert takes ~45 mins!
Something is misconfigured as it takes ~20 times as long to insert into an
in-memory db, as it does to select from Sybase and store to a file. I had a
look through the docs and had a google, and came up with some PRAGMA
options, so I tried:
PRAGMA synchronous = OFF
PRAGMA temp_store = MEMORY
Sadly, these made no difference. Does anyone have any idea what is causing
this? Code is running on Solaris 8 on a Sun workstation 1GB of memory 2x
450MHz CPU, SQLite version 3.2.7. I'm opening the database and calling
sqlite3_exec in a pretty standard way:
// Open the database:
sqlite3 *cache;
rc = sqlite3_open(":memory:", &cache)
if ( rc )
{
fprintf(stderr, "Can't open database: %s\n", sqlite3_errmsg(cache));
sqlite3_close(cache);
exit(1);
} else {
// set the database params
string sql = "PRAGMA synchronous = OFF\n";
sql += "PRAGMA temp_store = MEMORY\n";
rc = sqlite3_exec(cache, sql.c_str(), NULL, 0, &zErrMsg);
if ( rc != SQLITE_OK )
{
fprintf(stderr, "SQL error: %s\nSQL Text: %s", zErrMsg,
sql.c_str());
}
}
// Create some tables = called once per table:
rc = sqlite3_exec(cache, create_sql.c_str(), NULL, 0, &zErrMsg);
if ( rc != SQLITE_OK )
{
fprintf(stderr, "SQL error: %s\nSQL text: %s\n", zErrMsg,
create_sql.c_str());
}
// Then the insert - called 800,000 times for the extract that takes 45
minutes :
rc = sqlite3_exec(cache, populate_sql.c_str(), NULL, 0, &zErrMsg);
if ( rc != SQLITE_OK )
{
fprintf(stderr, "SQL error: %s\nSQL text: %s\n\n", zErrMsg,
populate_sql.c_str());
}
Any ideas, however slight, will be much appreciated!
Many thanks,
David Carter-Hitchin.
--LongSig
***********************************************************************************
The Royal Bank of Scotland plc. Registered in Scotland No 90312.
Registered Office: 36 St Andrew Square, Edinburgh EH2 2YB.
Authorised and regulated by the Financial Services Authority
This e-mail message is confidential and for use by the
addressee only. If the message is received by anyone other
than the addressee, please return the message to the sender
by replying to it and then delete the message from your
computer. Internet e-mails are not necessarily secure. The
Royal Bank of Scotland plc does not accept responsibility for
changes made to this message after it was sent.
Whilst all reasonable care has been taken to avoid the
transmission of viruses, it is the responsibility of the recipient to
ensure that the onward transmission, opening or use of this
message and any attachments will not adversely affect its
systems or data. No responsibility is accepted by The Royal
Bank of Scotland plc in this regard and the recipient should carry
out such virus and other checks as it considers appropriate.
Visit our websites at:
http://www.rbs.co.uk/CBFM
http://www.rbsmarkets.com
********************************************************************************
