Re: [sqlite] Is version 3 seriously broken?

[Jakub Adamek]

Nuno, hurrah, it works on first tests. I will run more later. Clever 
idea! I am now looking much forward to the release ... Thanks for your work.

Jakub
Nuno Lucas wrote:
Jakub Adamek, dando pulos de alegria, escreveu :
 > Nuno, I am much surprised that version 3.0.5 helped you. It didn't help
 > me. Neither my nor your port. BUT your remark helped me! You are right
 > that it is because of stack space, and the default setting in Windows CE
 > projects is 0x1, i.e. 65 kB. After changing to 0x10, i.e. 1 MB
 > my test program which first added 1000 rows of size 0..2000, than
 > deleted all of them and created 1000 tables works fine. Did you perhaps
 > also change this setting?
Now that that you talked about it, I remembered an irritating bug in
VC++ that forces us to rebuild all after changing linking options
(clean, build doesn't work). You are right, it's the same...
But the good news is that I found the cure (well, thinking about it, I
should have remembered about it earlier). There is a
SQLITE_MAX_PAGE_SIZE constant exactly for the sake of embedded systems.
I committed a new version, where I defined that to be 1024 (instead of
the default of 8192) and it works now. I tried 2048 but it isn't low
enough, too.
If someone wants more, it will have to increase the stack size in the
linking options, and it should be done by someone that understands what
it wants to do.
 > Are you going to merge 3.0.6?
Already done and committed, stay tuned for the release...
Regards,
~Nuno Lucas

Re: [sqlite] Is version 3 seriously broken?

[Nuno Lucas]

Jakub Adamek, dando pulos de alegria, escreveu :
> Nuno, I am much surprised that version 3.0.5 helped you. It didn't help
> me. Neither my nor your port. BUT your remark helped me! You are right
> that it is because of stack space, and the default setting in Windows CE
> projects is 0x1, i.e. 65 kB. After changing to 0x10, i.e. 1 MB
> my test program which first added 1000 rows of size 0..2000, than
> deleted all of them and created 1000 tables works fine. Did you perhaps
> also change this setting?
Now that that you talked about it, I remembered an irritating bug in
VC++ that forces us to rebuild all after changing linking options
(clean, build doesn't work). You are right, it's the same...
But the good news is that I found the cure (well, thinking about it, I
should have remembered about it earlier). There is a
SQLITE_MAX_PAGE_SIZE constant exactly for the sake of embedded systems.
I committed a new version, where I defined that to be 1024 (instead of
the default of 8192) and it works now. I tried 2048 but it isn't low
enough, too.
If someone wants more, it will have to increase the stack size in the
linking options, and it should be done by someone that understands what
it wants to do.
> Are you going to merge 3.0.6?
Already done and committed, stay tuned for the release...
Regards,
~Nuno Lucas

Re: [sqlite] Is version 3 seriously broken?

[Jakub Adamek]

Nuno, I am much surprised that version 3.0.5 helped you. It didn't help 
me. Neither my nor your port. BUT your remark helped me! You are right 
that it is because of stack space, and the default setting in Windows CE 
projects is 0x1, i.e. 65 kB. After changing to 0x10, i.e. 1 MB 
my test program which first added 1000 rows of size 0..2000, than 
deleted all of them and created 1000 tables works fine. Did you perhaps 
also change this setting?

Are you going to merge 3.0.6?
Best regards,
Jakub
Nuno Lucas wrote:
Jakub, I must ask you to forgive me as only now noticed you were not the 
sender of the tracker message for the bug. Sorry for the confusion (I 
must stop replying messages late at night).

In the code I tested it was always the 113th row because I only created 
a single table. I would assume it would happen sooner to you because you 
have several tables created.

It's a little difficult to test your code as it is dependent on other 
libraries. Could you post just the SQL statements used so we could check 
it with the new SQLite-wince v3.0.5?

As I don't know the classes you are using I have no way of checking the 
stack use of them (could they be creating rowsets in the stack, for 
example?).

Anyway, with the 3.0.5 port, I could now insert 5 rows without 
problems. Maybe you could check if my merge is different from yours and 
if it works now.

Regards,
~Nuno Lucas
Jakub Adamek, dando pulos de alegria, escreveu :
Nuno, I have changed back to SQLite 2 and have my code ready to switch 
to version 3 again if the bug is repaired. The thing happens when 
first calling balance_nonroot, which is in your case always on the 
113th row, in my case on the 33th row and in the mail message
http://www.mail-archive.com/[EMAIL PROTECTED]/msg03112.html
always at the sixth CREATE TABLE. I did try SQLite 3.0.5 and copied 
the changes as in SQLite-WinCE with no improvement.

I would agree with too much stack space, which may be perhaps solved 
by using malloc() and free() in sqlite_nonroot.

Jakub

Re: [sqlite] Is version 3 seriously broken?

[Nuno Lucas]

Jakub, I must ask you to forgive me as only now noticed you were not the 
sender of the tracker message for the bug. Sorry for the confusion (I 
must stop replying messages late at night).

In the code I tested it was always the 113th row because I only created 
a single table. I would assume it would happen sooner to you because you 
have several tables created.

It's a little difficult to test your code as it is dependent on other 
libraries. Could you post just the SQL statements used so we could check 
it with the new SQLite-wince v3.0.5?

As I don't know the classes you are using I have no way of checking the 
stack use of them (could they be creating rowsets in the stack, for 
example?).

Anyway, with the 3.0.5 port, I could now insert 5 rows without 
problems. Maybe you could check if my merge is different from yours and 
if it works now.

Regards,
~Nuno Lucas
Jakub Adamek, dando pulos de alegria, escreveu :
Nuno, I have changed back to SQLite 2 and have my code ready to switch 
to version 3 again if the bug is repaired. The thing happens when first 
calling balance_nonroot, which is in your case always on the 113th row, 
in my case on the 33th row and in the mail message
http://www.mail-archive.com/[EMAIL PROTECTED]/msg03112.html
always at the sixth CREATE TABLE. I did try SQLite 3.0.5 and copied the 
changes as in SQLite-WinCE with no improvement.

I would agree with too much stack space, which may be perhaps solved by 
using malloc() and free() in sqlite_nonroot.

Jakub

Re: [sqlite] Is version 3 seriously broken?

[Jakub Adamek]

Nuno, I have changed back to SQLite 2 and have my code ready to switch 
to version 3 again if the bug is repaired. The thing happens when first 
calling balance_nonroot, which is in your case always on the 113th row, 
in my case on the 33th row and in the mail message
http://www.mail-archive.com/[EMAIL PROTECTED]/msg03112.html
always at the sixth CREATE TABLE. I did try SQLite 3.0.5 and copied the 
changes as in SQLite-WinCE with no improvement.

I would agree with too much stack space, which may be perhaps solved by 
using malloc() and free() in sqlite_nonroot.

Jakub
Nuno Lucas wrote:
Jakub,
the "official" sqlite version isn't broken, it's the WinCE port that is.
I decided to announce that in the SQLite-WinCE page so people wouldn't
expect it to just work. There are other issues with the WinCE port (like
the Unicode handling, and that is also in the official version), but
this is serious enough because it crashes (with a stack overflow
exception) after inserting just 113 rows (and always 113 rows).
My guess is that the official version allocated too much stack space,
resulting in that exception.
---
I now have tested the same program with SQLite v3.0.5 (inserted 5
rows) and the problem disappeared. So something was corrected in the 
source code that makes it use less stack.

I would advise you to wait until I
commit those changes to CVS and make a new release so you can try by
yourself.
I annexed your test program (with your bugs corrected), so you can check 
it works now.

Regards,
~Nuno Lucas

/* BUG: the right header to include is "sqlite3.h".
   "sqliteint.h" is an internal header and can't be used safelly */
/*/
#include 
#include 
#include 
#include 
bool TestExecuter( sqlite3 *db_hnd, char *query, ... )
{
//  char **_result_set;
//  int _n_row=0;
//  int _n_col=0;
//  char *_error_msg;
va_list tmp_args;
va_start( tmp_args, query );
char *tmp_query = sqlite3_vmprintf( query, tmp_args );
sqlite3_stmt * stmt = 0;
const char* tail;
//  int tmp_ec = sqlite3_get_table( db_hnd, tmp_query, &_result_set, &_n_row, &_n_col, 
&_error_msg );
if ( sqlite3_prepare( db_hnd, tmp_query, strlen(tmp_query), ,  ) )
{
OutputDebugString( _T("sqlite3_prepare: error\n") );
return false;
}
int rc = sqlite3_step( stmt );
switch ( rc )
{
case SQLITE_ROW :   // First row ready
break;
case SQLITE_DONE:   // Ok, simply no results for this command
break;
default :   // Some error
OutputDebugString( 
(LPCWSTR)sqlite3_errmsg16(db_hnd) );
}
rc = sqlite3_finalize( stmt );
if ( rc != SQLITE_OK )
OutputDebugString( (LPCWSTR)sqlite3_errmsg16(db_hnd) );
// Libero la query.
sqlite3_free(tmp_query);
/** BUG: no release of result set */
//  sqlite3_free_table( _result_set );
/**/
//  if ( (_n_row == -1) || (_error_msg != NULL) )
//  {
//  TCHAR buf[512];
//  _stprintf( buf, _T("Query Failed! - errmsg: %hs\n"), _error_msg );
//  OutputDebugString( buf );
//  /** BUG: no release of error message */
//  sqlite3_free( _error_msg );
//  /**/
//  }
// speedup things while testing
//  else
//  OutputDebugString( "Query Ok!\n" );
/** BUG: no va_end(tmp_args) */
va_end( tmp_args );
/**/
//  return (_n_row == -1) || (_error_msg != NULL);
return rc == SQLITE_OK;
}
int Test_Insert2(sqlite3 *db_hnd)
{
// Create some tables with data that we can select against
TestExecuter( db_hnd, "CREATE TABLE d1(n int, log int)" );
TestExecuter( db_hnd, "BEGIN" );
char tmp_str[256];
for (int i=1;i<=5;i++)
{
//  for ( int j=0; (1 << j) < i; j++ )
{
sprintf(tmp_str,"INSERT INTO d1 VALUES(%d,%d)",i,i);
TestExecuter( db_hnd, tmp_str );
}
}
TestExecuter( db_hnd, "END" );
TestExecuter( db_hnd, "SELECT * FROM d1 ORDER BY n" );
// finish_test
return 0;
}
#define DB_TEST "\\Test.db"
int WINAPI WinMain( HINSTANCE,HINSTANCE,LPTSTR,int )
{
sqlite3 * db;
int err = sqlite3_open( DB_TEST,  );
if ( err )
return -1;
Test_Insert2( db );
/ BUG: no sqlite3_close can eventually corrupt the database */
sqlite3_close( db );
//
return 0;
}

Re: [sqlite] Is version 3 seriously broken?

[Nuno Lucas]

Jakub,
the "official" sqlite version isn't broken, it's the WinCE port that is.
I decided to announce that in the SQLite-WinCE page so people wouldn't
expect it to just work. There are other issues with the WinCE port (like
the Unicode handling, and that is also in the official version), but
this is serious enough because it crashes (with a stack overflow
exception) after inserting just 113 rows (and always 113 rows).
My guess is that the official version allocated too much stack space,
resulting in that exception.
---
I now have tested the same program with SQLite v3.0.5 (inserted 5
rows) and the problem disappeared. So something was corrected in the 
source code that makes it use less stack.

I would advise you to wait until I
commit those changes to CVS and make a new release so you can try by
yourself.
I annexed your test program (with your bugs corrected), so you can check 
it works now.

Regards,
~Nuno Lucas
/* BUG: the right header to include is "sqlite3.h".
   "sqliteint.h" is an internal header and can't be used safelly */
/*/
#include 
#include 
#include 
#include 


bool TestExecuter( sqlite3 *db_hnd, char *query, ... )
{
//  char **_result_set;
//  int _n_row=0;
//  int _n_col=0;
//  char *_error_msg;

va_list tmp_args;
va_start( tmp_args, query );
char *tmp_query = sqlite3_vmprintf( query, tmp_args );

sqlite3_stmt * stmt = 0;
const char* tail;

//  int tmp_ec = sqlite3_get_table( db_hnd, tmp_query, &_result_set, &_n_row, 
&_n_col, &_error_msg );

if ( sqlite3_prepare( db_hnd, tmp_query, strlen(tmp_query), ,  ) )
{
OutputDebugString( _T("sqlite3_prepare: error\n") );
return false;
}

int rc = sqlite3_step( stmt );
switch ( rc )
{
case SQLITE_ROW :   // First row ready
break;
case SQLITE_DONE:   // Ok, simply no results for this command
break;
default :   // Some error
OutputDebugString( 
(LPCWSTR)sqlite3_errmsg16(db_hnd) );
}

rc = sqlite3_finalize( stmt );
if ( rc != SQLITE_OK )
OutputDebugString( (LPCWSTR)sqlite3_errmsg16(db_hnd) );

// Libero la query.
sqlite3_free(tmp_query);

/** BUG: no release of result set */
//  sqlite3_free_table( _result_set );
/**/

//  if ( (_n_row == -1) || (_error_msg != NULL) )
//  {
//  TCHAR buf[512];
//  _stprintf( buf, _T("Query Failed! - errmsg: %hs\n"), _error_msg );
//  OutputDebugString( buf );
//  /** BUG: no release of error message */
//  sqlite3_free( _error_msg );
//  /**/
//  }
// speedup things while testing
//  else
//  OutputDebugString( "Query Ok!\n" );

/** BUG: no va_end(tmp_args) */
va_end( tmp_args );
/**/

//  return (_n_row == -1) || (_error_msg != NULL);
return rc == SQLITE_OK;
}

int Test_Insert2(sqlite3 *db_hnd)
{
// Create some tables with data that we can select against
TestExecuter( db_hnd, "CREATE TABLE d1(n int, log int)" );
TestExecuter( db_hnd, "BEGIN" );
char tmp_str[256];
for (int i=1;i<=5;i++)
{
//  for ( int j=0; (1 << j) < i; j++ )
{
sprintf(tmp_str,"INSERT INTO d1 VALUES(%d,%d)",i,i);
TestExecuter( db_hnd, tmp_str );
}
}
TestExecuter( db_hnd, "END" );
TestExecuter( db_hnd, "SELECT * FROM d1 ORDER BY n" );

// finish_test
return 0;
}


#define DB_TEST "\\Test.db"


int WINAPI WinMain( HINSTANCE,HINSTANCE,LPTSTR,int )
{
sqlite3 * db;
int err = sqlite3_open( DB_TEST,  );
if ( err )
return -1;
Test_Insert2( db );

/ BUG: no sqlite3_close can eventually corrupt the database */
sqlite3_close( db );
//
return 0;
}

Re: [sqlite] Is version 3 seriously broken?

[Jakub Adamek]

I didn't mention that I applied the diffs from SQLite-WinCE to version 
3.0.5 with the same results.

Jakub
Jakub Adamek wrote:
Hello,
the version 3 is most probably broken. I traced down the crash problem 
in Windows CE, see
http://sourceforge.net/mailarchive/forum.php?thread_id=5438459_id=35230 

It happens in balance_nonroot but in the declare variable section! If I 
change the variables in balance_nonroot to "static", the test goes OK. 
But such an error must come from a wrong memory write before this 
function. I am not able to find this. I attach my test program, it only 
creates a table and starts inserting rows.

As the simiral problem appears on other OSes, see 
http://www.mail-archive.com/[EMAIL PROTECTED]/msg03112.html,
I wonder if some of the great developers could solve it?

Best regards
Jakub
Darren Duncan wrote:
At 2:20 PM -0400 8/29/04, D. Richard Hipp wrote:
SQLite version 3.0.5 (beta) is now available on the website.

Richard, thank you very much for that release.
I have downloaded and compiled it under both Mac OS X 10.2.8 (GCC 3.1, 
3.3), and Mac OS X 10.3.5 (GCC3.3), with each development environment 
having the default set of libraries and headers.  It compiles without 
errors (but with warnings about long integer types) on all 
configurations, and ./sqlite3 starts up and quits normally.

Under both Mac OS X 10.2.8 configurations, 'make test' continues to 
fail due to my TCL library being out of date, as I had reported 
previously.

Under the Mac OS X 10.3.5 configurations, 'make test' succeeds, with a 
result of '0 errors out of 22364 tests'.  10.3.5 does have a new 
enough TCL bundled.

So at least with the newer setup, I can confirm that everything works.
Matt, I will eagerly test a newer DBD::SQLite as soon as you release it.
-- Darren Duncan


if (! db.execute (
"CREATE TABLE test ("
"i INTEGER NOT NULL,"
"v VARCHAR (50),"
"PRIMARY KEY (i));")) ERROR_RETURN_FALSE;
vector values;
values.push_back (12);
values.push_back ("ahoj brouku");
values.push_back (13);
values.push_back ("nazdar brouku");
if (! db.execute (
"INSERT INTO test VALUES (12, 'ahoj brouku')")) ERROR_RETURN_FALSE;
if (db.execute (
"INSERT INTO test VALUES (12, 'nazdar brouku')")) ERROR_RETURN_FALSE;
if (! db.execute (
"INSERT INTO test VALUES (13, 'nazdar brouku')")) ERROR_RETURN_FALSE;
if (db.isExecutePrecompiledSupported()) {
vector params;
params.push_back (14);
params.push_back ("cau brundibare");
if (! db.executePrecompiled (
"INSERT INTO test VALUES (?, ?)", params)) ERROR_RETURN_FALSE;
if (db.executePrecompiled (
"INSERT INTO test VALUES (?, ?)", params)) ERROR_RETURN_FALSE;
values.push_back (14);
values.push_back ("cau brundibare");
}
COneWayRecordset rs (db);
if (! rs.openTable ("test")) ERROR_RETURN_FALSE;
if (! rs.add()) ERROR_RETURN_FALSE;
rs ["i"] = 15;
rs ["v"] = "Hello world";
if (! rs.update()) ERROR_RETURN_FALSE;
values.push_back (15);
values.push_back ("Hello world");
if (! rs.open ("SELECT i, v FROM test")) ERROR_RETURN_FALSE;
int irecord = 0;
bool ok;
while (rs.next()) {
int i = rs ["i"].forceInt ();
string s = rs ["v"].asString();
if (i != values [irecord*2].forceInt ()) ERROR_RETURN_FALSE;
if (s != values [irecord*2+1].asString()) ERROR_RETURN_FALSE;
irecord ++;
}
	// fill some rows with string and binary data
	if (! db.begin()) ERROR_RETURN_FALSE;
if (! rs.openTable ("test")) ERROR_RETURN_FALSE;
	vector vec;
	string s;
	for (int row=1; row < 1000; row ++) {
		vec.push_back ((uint8_t) (row & 0xFF));
		s += long2string (row);
		CVariantValueBinary binary (& *vec.begin(), vec.size());
		if (! rs.add()) ERROR_RETURN_FALSE;
		rs ["i"] = row * row;
		rs ["v"] = s; 
		if (! rs.update()) ERROR_RETURN_FALSE;	
		if (! rs.add()) ERROR_RETURN_FALSE;
		rs ["i"] = row * row + 1;
		rs ["v"] = binary; 
		if (! rs.update()) ERROR_RETURN_FALSE;	
	}
	if (! db.close()) ERROR_RETURN_FALSE;
return true;

[sqlite] Is version 3 seriously broken?

[Jakub Adamek]

Hello,
the version 3 is most probably broken. I traced down the crash problem 
in Windows CE, see
http://sourceforge.net/mailarchive/forum.php?thread_id=5438459_id=35230

It happens in balance_nonroot but in the declare variable section! If I 
change the variables in balance_nonroot to "static", the test goes OK. 
But such an error must come from a wrong memory write before this 
function. I am not able to find this. I attach my test program, it only 
creates a table and starts inserting rows.

As the simiral problem appears on other OSes, see 
http://www.mail-archive.com/[EMAIL PROTECTED]/msg03112.html,
I wonder if some of the great developers could solve it?

Best regards
Jakub
Darren Duncan wrote:
At 2:20 PM -0400 8/29/04, D. Richard Hipp wrote:
SQLite version 3.0.5 (beta) is now available on the website.

Richard, thank you very much for that release.
I have downloaded and compiled it under both Mac OS X 10.2.8 (GCC 3.1, 
3.3), and Mac OS X 10.3.5 (GCC3.3), with each development environment 
having the default set of libraries and headers.  It compiles without 
errors (but with warnings about long integer types) on all 
configurations, and ./sqlite3 starts up and quits normally.

Under both Mac OS X 10.2.8 configurations, 'make test' continues to fail 
due to my TCL library being out of date, as I had reported previously.

Under the Mac OS X 10.3.5 configurations, 'make test' succeeds, with a 
result of '0 errors out of 22364 tests'.  10.3.5 does have a new enough 
TCL bundled.

So at least with the newer setup, I can confirm that everything works.
Matt, I will eagerly test a newer DBD::SQLite as soon as you release it.
-- Darren Duncan
if (! db.execute (
"CREATE TABLE test ("
"i INTEGER NOT NULL,"
"v VARCHAR (50),"
"PRIMARY KEY (i));")) ERROR_RETURN_FALSE;

vector values;
values.push_back (12);
values.push_back ("ahoj brouku");
values.push_back (13);
values.push_back ("nazdar brouku");

if (! db.execute (
"INSERT INTO test VALUES (12, 'ahoj brouku')")) ERROR_RETURN_FALSE;
if (db.execute (
"INSERT INTO test VALUES (12, 'nazdar brouku')")) ERROR_RETURN_FALSE;
if (! db.execute (
"INSERT INTO test VALUES (13, 'nazdar brouku')")) ERROR_RETURN_FALSE;

if (db.isExecutePrecompiledSupported()) {
vector params;
params.push_back (14);
params.push_back ("cau brundibare");
if (! db.executePrecompiled (
"INSERT INTO test VALUES (?, ?)", params)) ERROR_RETURN_FALSE;
if (db.executePrecompiled (
"INSERT INTO test VALUES (?, ?)", params)) ERROR_RETURN_FALSE;
values.push_back (14);
values.push_back ("cau brundibare");
}

COneWayRecordset rs (db);
if (! rs.openTable ("test")) ERROR_RETURN_FALSE;
if (! rs.add()) ERROR_RETURN_FALSE;
rs ["i"] = 15;
rs ["v"] = "Hello world";
if (! rs.update()) ERROR_RETURN_FALSE;
values.push_back (15);
values.push_back ("Hello world");

if (! rs.open ("SELECT i, v FROM test")) ERROR_RETURN_FALSE;
int irecord = 0;
bool ok;
while (rs.next()) {
int i = rs ["i"].forceInt ();
string s = rs ["v"].asString();
if (i != values [irecord*2].forceInt ()) ERROR_RETURN_FALSE;
if (s != values [irecord*2+1].asString()) ERROR_RETURN_FALSE;
irecord ++;
}

// fill some rows with string and binary data
if (! db.begin()) ERROR_RETURN_FALSE;
if (! rs.openTable ("test")) ERROR_RETURN_FALSE;
vector vec;
string s;
for (int row=1; row < 1000; row ++) {
vec.push_back ((uint8_t) (row & 0xFF));
s += long2string (row);
CVariantValueBinary binary (& *vec.begin(), vec.size());
if (! rs.add()) ERROR_RETURN_FALSE;
rs ["i"] = row * row;
rs ["v"] = s; 
if (! rs.update()) ERROR_RETURN_FALSE;  
if (! rs.add()) ERROR_RETURN_FALSE;
rs ["i"] = row * row + 1;
rs ["v"] = binary; 
if (! rs.update()) ERROR_RETURN_FALSE;  
}
if (! db.close()) ERROR_RETURN_FALSE;
return true;