Re: [sqlite] Create table returns "database disk image is malformed" when disk is full and "SQLITE_IOCAP_ATOMIC" and "SQLITE_DEFAULT_AUTOVACUUM" is enabled

[Yongil Jang]

Dear all,

I changed btree.c like as follows.
But, it may not be a best solution.
Whatever TCL test for "quick.test" is passed, following code can have some
mistakes...
I couldn't change it more with my poor knowledge.

*static int btreeCreateTable(Btree *p, int *piTable, int createTabFlags)*{
  BtShared *pBt = p->pBt;
  MemPage *pRoot;
  Pgno pgnoRoot;
  int rc;
  int ptfFlags;  /* Page-type flage for the root page of new table
*/
*  //yongiljang*
*  int nOrgPage;*
*  //yongiljang_end*
  ..
while( pgnoRoot==PTRMAP_PAGENO(pBt, pgnoRoot) ||
pgnoRoot==PENDING_BYTE_PAGE(pBt) ){
  pgnoRoot++;
}
assert( pgnoRoot>=3 );

*//yongiljang*
*nOrgPage = pBt->nPage;*
*//yongiljang_end*

/* Allocate a page. The page that currently resides at pgnoRoot will
** be moved to the allocated page (unless the allocated page happens
** to reside at pgnoRoot).
*/
rc = allocateBtreePage(pBt, , , pgnoRoot, 1);
..
/* Update the pointer-map and meta-data with the new root-page number.
*/
ptrmapPut(pBt, pgnoRoot, PTRMAP_ROOTPAGE, 0, );
if( rc ){
  *//yongiljang*
*  pBt->nPage = nOrgPage;*
*  put4byte(28 + (u8*)pBt->pPage1->aData, pBt->nPage);*
*  //yongiljang_end*
  releasePage(pRoot);
  return rc;
}
..
}

B.R.
yongil jang.



2012/11/30 Yongil Jang 

> In btreeCreateTable, a new page is allocated by allocateBtreePage function
> without any failures.
> In normal case, this should be failed because of there is no space to
> write journal file.
> jrnlWrite function called from allocateBtree is passed by writing memory
> block(no real file) in atomic write mode.
>
> After finished of allocateBtreePage, ptrmapPut function called to update
> the pointer-map and meta-data with the new root-page number.
> Finally, this function is failed at sqlite3PagerWrite because of real file
> write is called at this time.
> But, the newly allocated page number in btree structure(or pager?) didn't
> changed to original value.
>
> In my opinion, issue point is updated page number is not returned to
> previous value.
> the allocateBtreePage function doesn't return disk full error and this
> wrong value is remained until next query execution.
> ptrmapPut function doesn't clear this page number.
>
> If I turn off "atomic write" option, it works very well.
> (allocateBtreePage function will return disk full error and page number
> will be replaced to original value)
>
> My question is how to replace page number in btree structure to original
> value when failure is occurred after call of allocateBtreePage funciton.
> Or... is there any better code for this?
>
> Thank you.
> Yongil Jang.
>
>
>
> 2012/11/26 Yongil Jang 
>
>> Hi, there.
>>
>> I just found some illegal processing of SQLite.
>> As I mentioned in title, "CREATE TABLE" query returns "database disk
>> image is malformed" when disk is full and "SQLITE_IOCAP_ATOMIC" and
>> "SQLITE_DEFAULT_AUTOVACUUM" is enabled.
>> Here is my test scripts.
>>
>> sudo mkdir /mnt/db
>> sudo chmod 777 /mnt/db
>> sudo mount -t tmpfs -o size=16K tmpfs /mnt/db
>> /home/yi.jang/git/sqlite_source/sqlite3 /mnt/db/test.db
>>
>> SQLite version 3.7.13 2012-06-11 02:05:22
>> Enter ".help" for instructions
>> Enter SQL statements terminated with a ";"
>> sqlite> pragma journal_mode=persist;
>> persist
>> sqlite> pragma page_size=4096;
>> sqlite> create table a (id);
>> sqlite> create table b (id);
>> *Error: database or disk is full*
>> sqlite> create table b (id);
>> *Error: database disk image is malformed*
>> sqlite>
>>
>> What I attached options to Makefile are as following.
>>
>> -DSQLITE_ENABLE_ATOMIC_WRITE
>> -DSQLITE_DEFAULT_AUTOVACUUM=1
>> -DSQLITE_DEFAULT_PAGE_SIZE=4096
>>
>> sqlite3.c is also changed.
>>
>> static int unixDeviceCharacteristics(sqlite3_file *id){
>>unixFile *p = (unixFile*)id;
>>   if( p->ctrlFlags & UNIXFILE_PSOW ){
>> return SQLITE_IOCAP_POWERSAFE_OVERWRITE | SQLITE_IOCAP_ATOMIC;
>>   }else{
>> return SQLITE_IOCAP_ATOMIC;
>>   }
>> }
>>
>> B.R.
>> Yongil Jang.
>>
>>
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Create table returns "database disk image is malformed" when disk is full and "SQLITE_IOCAP_ATOMIC" and "SQLITE_DEFAULT_AUTOVACUUM" is enabled

[Yongil Jang]

In btreeCreateTable, a new page is allocated by allocateBtreePage function
without any failures.
In normal case, this should be failed because of there is no space to write
journal file.
jrnlWrite function called from allocateBtree is passed by writing memory
block(no real file) in atomic write mode.

After finished of allocateBtreePage, ptrmapPut function called to update
the pointer-map and meta-data with the new root-page number.
Finally, this function is failed at sqlite3PagerWrite because of real file
write is called at this time.
But, the newly allocated page number in btree structure(or pager?) didn't
changed to original value.

In my opinion, issue point is updated page number is not returned to
previous value.
the allocateBtreePage function doesn't return disk full error and this
wrong value is remained until next query execution.
ptrmapPut function doesn't clear this page number.

If I turn off "atomic write" option, it works very well. (allocateBtreePage
function will return disk full error and page number will be replaced to
original value)

My question is how to replace page number in btree structure to original
value when failure is occurred after call of allocateBtreePage funciton.
Or... is there any better code for this?

Thank you.
Yongil Jang.



2012/11/26 Yongil Jang 

> Hi, there.
>
> I just found some illegal processing of SQLite.
> As I mentioned in title, "CREATE TABLE" query returns "database disk image
> is malformed" when disk is full and "SQLITE_IOCAP_ATOMIC" and
> "SQLITE_DEFAULT_AUTOVACUUM" is enabled.
> Here is my test scripts.
>
> sudo mkdir /mnt/db
> sudo chmod 777 /mnt/db
> sudo mount -t tmpfs -o size=16K tmpfs /mnt/db
> /home/yi.jang/git/sqlite_source/sqlite3 /mnt/db/test.db
>
> SQLite version 3.7.13 2012-06-11 02:05:22
> Enter ".help" for instructions
> Enter SQL statements terminated with a ";"
> sqlite> pragma journal_mode=persist;
> persist
> sqlite> pragma page_size=4096;
> sqlite> create table a (id);
> sqlite> create table b (id);
> *Error: database or disk is full*
> sqlite> create table b (id);
> *Error: database disk image is malformed*
> sqlite>
>
> What I attached options to Makefile are as following.
>
> -DSQLITE_ENABLE_ATOMIC_WRITE
> -DSQLITE_DEFAULT_AUTOVACUUM=1
> -DSQLITE_DEFAULT_PAGE_SIZE=4096
>
> sqlite3.c is also changed.
>
> static int unixDeviceCharacteristics(sqlite3_file *id){
>   unixFile *p = (unixFile*)id;
>   if( p->ctrlFlags & UNIXFILE_PSOW ){
> return SQLITE_IOCAP_POWERSAFE_OVERWRITE | SQLITE_IOCAP_ATOMIC;
>   }else{
> return SQLITE_IOCAP_ATOMIC;
>   }
> }
>
> B.R.
> Yongil Jang.
>
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Create table returns "database disk image is malformed" when disk is full and "SQLITE_IOCAP_ATOMIC" and "SQLITE_DEFAULT_AUTOVACUUM" is enabled

[Yongil Jang]

I tried to solve this problem by myself...
But, It's very difficult work to me. :)

SQLITE_CORRUPT_BKPT is returned in lockBtree, because of, nPage(=4) is
bigger than nPageFile(=3).

First call of creating table "b" returns "database or disk is full" and it
is normal case.
But, second call of creating table "b" returns database corruption.
If I close database connection and reconnect to target database, it shows
same patterns with above.

In my opinion, a new page information of a table is not cleaned when meets
first "disk is full" in "ATOMIC WRITE" and "AUTOVACUUM" mode. It makes
database corruption status when nPage and nPageFile is compared.
For example, If auto_vacuum == 0, then nPage and nPageFile have same values
of #2. In btreeCreateTable function, it is changed with "if
(pBt->autoVacuum)" condition.

Could anyone give me some hints to solve it?

B. R.
Yongil Jang.



2012/11/26 Yongil Jang 

> Hi, there.
>
> I just found some illegal processing of SQLite.
> As I mentioned in title, "CREATE TABLE" query returns "database disk image
> is malformed" when disk is full and "SQLITE_IOCAP_ATOMIC" and
> "SQLITE_DEFAULT_AUTOVACUUM" is enabled.
> Here is my test scripts.
>
> sudo mkdir /mnt/db
> sudo chmod 777 /mnt/db
> sudo mount -t tmpfs -o size=16K tmpfs /mnt/db
> /home/yi.jang/git/sqlite_source/sqlite3 /mnt/db/test.db
>
> SQLite version 3.7.13 2012-06-11 02:05:22
> Enter ".help" for instructions
> Enter SQL statements terminated with a ";"
> sqlite> pragma journal_mode=persist;
> persist
> sqlite> pragma page_size=4096;
> sqlite> create table a (id);
> sqlite> create table b (id);
> *Error: database or disk is full*
> sqlite> create table b (id);
> *Error: database disk image is malformed*
> sqlite>
>
> What I attached options to Makefile are as following.
>
> -DSQLITE_ENABLE_ATOMIC_WRITE
> -DSQLITE_DEFAULT_AUTOVACUUM=1
> -DSQLITE_DEFAULT_PAGE_SIZE=4096
>
> sqlite3.c is also changed.
>
> static int unixDeviceCharacteristics(sqlite3_file *id){
>   unixFile *p = (unixFile*)id;
>   if( p->ctrlFlags & UNIXFILE_PSOW ){
> return SQLITE_IOCAP_POWERSAFE_OVERWRITE | SQLITE_IOCAP_ATOMIC;
>   }else{
> return SQLITE_IOCAP_ATOMIC;
>   }
> }
>
> B.R.
> Yongil Jang.
>
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users