Re: [sqlite] SIGBUS error in case of disk full with WAL mode

[Kees Nuyt]

On Sat, 10 Mar 2012 10:25:10 +0900, Yongil Jang 
wrote:

>Following sentences are found on sqlite source.
>And, there is a comment about database corruption.
>Could I get more detailed explanation about this?
>
>The use of the SQLITE_SHM_DIRECTORY compile-time
>** option results in an incompatible build of SQLite;  *builds of SQLite*
>*** that with differing SQLITE_SHM_DIRECTORY settings attempt to use the*
>*** same database file at the same time, database corruption will likely*
>*** result*. 

It means that when two (or more) concurrent processes which access the
same database some using SQLITE_SHM_DIRECTORY, some without that option
will have different ideas of what memory to share.

If one process decides to use a shm file in the directory where the
database is located and an other process decides to use /dev/shm, the
processes don't "see" eachothers shared memory, and are not coordinated.

This will almost certainly cause database corruption.

> The SQLITE_SHM_DIRECTORY compile-time option is considered
>** "unsupported" and may go away in a future SQLite release.
>

That is the usual disclaimer for features that are not chosen to be part
of the final architecture. They may not be maintained, and when the rest
of the code evolves in an incompatible way, they may not work properly
any more.


-- 
Regards,

Kees Nuyt

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SIGBUS error in case of disk full with WAL mode

[Yongil Jang]

Dear all,

Thank you for your help.

I tried -DSQLITE_SHM_DIRECTORY="/dev/shm" option for make file.
It works very well.
(One line should be added to make "/dev/shm" directory before calling file
open)

But, I didn't tried to use locking mode because of multiple processes can
use same sqlite database at a same time.

I have one more question.

Following sentences are found on sqlite source.
And, there is a comment about database corruption.
Could I get more detailed explanation about this?


** can be enabled at compile-time using -DSQLITE_SHM_DIRECTORY="/dev/shm"
** or the equivalent.  The use of the SQLITE_SHM_DIRECTORY compile-time
** option results in an incompatible build of SQLite;  *builds of SQLite*
*** that with differing SQLITE_SHM_DIRECTORY settings attempt to use the*
*** same database file at the same time, database corruption will likely*
*** result*. The SQLITE_SHM_DIRECTORY compile-time option is considered
** "unsupported" and may go away in a future SQLite release.


Best wishes,
Jang.

2012/3/9 Scott Hess 

> On Thu, Mar 8, 2012 at 9:28 AM, Pavel Ivanov  wrote:
> >> Question:  Does anybody know of a better way to get memory shared among
> >> processes other than to create a fake file and mmap() it?  Are there
> some
> >> magic options to mmap() (perhaps Linux-only options) that prevent it
> from
> >> actually writing to disk?
> >
> > Why don't you use shm_open() instead of a real file? I'm not sure
> > though how it behaves with chroot jail.
>
> I do not recall the full semantics of shm_open(), but I _think_ that
> it shares the sysv shared-memory problem where the memory sticks
> around until explicitly deleted.  Using a fake file with mmap() will
> create a segment which only exists until the last process using it
> goes away.  This can become a resource issue on some systems.  Also,
> the name would have to be carefully constructed to prevent conflicts,
> as the namespace may not be the same as the file path namespace (even
> path-like names may have different length restrictions).  Like maybe
> base64(hash(canonical_path(dbpath))).
>
> BTW, the SQLite docs indicate that as of 3.7.4, you can arrange to use
> an exclusive mode which allows WAL to work without shm.  That may be a
> reasonable approach for some subset of users with this kind of
> problem.
>
> -scott
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SIGBUS error in case of disk full with WAL mode

[Scott Hess]

On Thu, Mar 8, 2012 at 9:28 AM, Pavel Ivanov  wrote:
>> Question:  Does anybody know of a better way to get memory shared among
>> processes other than to create a fake file and mmap() it?  Are there some
>> magic options to mmap() (perhaps Linux-only options) that prevent it from
>> actually writing to disk?
>
> Why don't you use shm_open() instead of a real file? I'm not sure
> though how it behaves with chroot jail.

I do not recall the full semantics of shm_open(), but I _think_ that
it shares the sysv shared-memory problem where the memory sticks
around until explicitly deleted.  Using a fake file with mmap() will
create a segment which only exists until the last process using it
goes away.  This can become a resource issue on some systems.  Also,
the name would have to be carefully constructed to prevent conflicts,
as the namespace may not be the same as the file path namespace (even
path-like names may have different length restrictions).  Like maybe
base64(hash(canonical_path(dbpath))).

BTW, the SQLite docs indicate that as of 3.7.4, you can arrange to use
an exclusive mode which allows WAL to work without shm.  That may be a
reasonable approach for some subset of users with this kind of
problem.

-scott
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SIGBUS error in case of disk full with WAL mode

[Pavel Ivanov]

> Question:  Does anybody know of a better way to get memory shared among
> processes other than to create a fake file and mmap() it?  Are there some
> magic options to mmap() (perhaps Linux-only options) that prevent it from
> actually writing to disk?

Why don't you use shm_open() instead of a real file? I'm not sure
though how it behaves with chroot jail.


Pavel


On Thu, Mar 8, 2012 at 11:35 AM, Richard Hipp  wrote:
> On Thu, Mar 8, 2012 at 10:58 AM, Yongiljang  wrote:
>
>> Dear all,
>> I'm an android developer in charge of sqlite database.
>>
>> Some days ago, I'd got a SIGBUS error from sqlite when there is no space
>> in current partition and WAL journal mode is used.
>> This error was occurred from memset function in libc that was called by
>> libsqlite and debugging information shows shm file was related to this
>> issue.
>>
>> Following sequence shows how to generate this error.
>>
>> 1) Make disk full
>> 2) Reboot device - wal and shm files are remained in some applications
>> folder
>> 3) Do 1) ~ 2) until free space remained under 32KB
>> 4) SIGBUS error is occurred randomly
>>
>> I'd tried to solve this problem and guessed following scenario.
>>
>> 1) shm file is truncated to zero size by calling robust_truncate function
>> when a new connection is opened in unixOpenSharedMemory
>> 2) shm file is extended to 32KB by calling robust_truncate function in
>> unixShmMap
>> 3) mmap function is called with 32KB length
>>
>> In my guess, problem is occurred from robust_truncate or mmap functions,
>> because of they didn't returned error code whether shm file is extended to
>> 32KB or not on disk full status.
>> robust_truncate and mmap may caused illegal memset operation because of
>> shm file actually doesn't have 32KB size, it may less than 32KB.
>> Interesting point is when I tested it with above scenario, there was a shm
>> file that has 32KB size.
>> It is impossible because of there was no space to make 32KB sized file in
>> current partiton.
>>
>
> We do not want to really make a file.  The purpose of the -shm file is
> merely to give a name to a block of memory that various processes accessing
> the database can share between themselves using mmap().  Ideally, the
> content of the -shm file remains the OS page cache and is never written to
> disk.
>
> Can you please try this experiment for us:  Beginning with a standard
> SQLite build (without your patches) recompile using
> -DSQLITE_SHM_DIRECTORY="/dev/shm".  That option will cause the shared
> memory file to be created in the /dev/shm directory rather than in the same
> directory as the database.  Since /dev/shm is not backed by disk (or flash)
> the problem should be solved.
>
> FWIW:  We looked at always putting the -shm files in a special directory
> like this when we were first designing WAL.  But we noticed that design
> fails if two programs in different chroot jails try to access the database
> at the same time, and so we switched to the current design of using the
> -shm file in the same directory as the database.
>
> Question:  Does anybody know of a better way to get memory shared among
> processes other than to create a fake file and mmap() it?  Are there some
> magic options to mmap() (perhaps Linux-only options) that prevent it from
> actually writing to disk?
>
>
>>
>> Whatever, I'd changed unixOpenSharedMemory to solve it.
>>
>> 1) make a shm file and write null data until 32KB if this file doesn't
>> exists
>>    - return SQLITE_FULL error when write operation is failed
>> 2) write null data until 32KB if this file exists and got write lock
>> instead of calling truncate function to shrink shm file to zero size
>>   - return same error code when it failed
>>
>> By changed source, I could solve this problem.
>> Sqlite returns SQLITE_FULL errors only without SIGBUS core dump.
>>
>> However, this instant code changing may not be a good solution.
>> I wish to get better comment or source patches from here.
>>
>> Thank you for reading this including my poor english. :)
>>
>> Best wishes,
>> Jang.
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>
>
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SIGBUS error in case of disk full with WAL mode

[Black, Michael (IS)]

I guess you could go the IPC methods described in the same reference?  Been a 
long time since I've used those.



Michael D. Black

Senior Scientist

Advanced Analytics Directorate

Advanced GEOINT Solutions Operating Unit

Northrop Grumman Information Systems


From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on 
behalf of Richard Hipp [d...@sqlite.org]
Sent: Thursday, March 08, 2012 11:03 AM
To: General Discussion of SQLite Database
Subject: EXT :Re: [sqlite] SIGBUS error in case of disk full with WAL mode

On Thu, Mar 8, 2012 at 11:45 AM, Black, Michael (IS)  wrote:

> Looks like this should work...
>

No, it won't work.  The memory has to be shared in common among all
connections to a particular database.  If two separate processes connection
to the same database, they must get the same block of shared memory.  If
they connect to different databases, they must get different blocks of
shared memory.



>
>
>
> From http://www.cs.cf.ac.uk/Dave/C/node27.html
>
>
>
> The following code fragment demonstrates a use of this to create a block
> of scratch storage in a program, at an address that the system chooses.:
>
> int fd;
> caddr_t result;
> if ((fd = open("/dev/zero", O_RDWR)) == -1)
>   return ((caddr_t)-1);
>
> result = mmap(0, len, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
> (void) close(fd);
>
>
>
>
> Michael D. Black
>
> Senior Scientist
>
> Advanced Analytics Directorate
>
> Advanced GEOINT Solutions Operating Unit
>
> Northrop Grumman Information Systems
>
> 
> From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org]
> on behalf of Richard Hipp [d...@sqlite.org]
> Sent: Thursday, March 08, 2012 10:35 AM
> To: General Discussion of SQLite Database
> Subject: EXT :Re: [sqlite] SIGBUS error in case of disk full with WAL mode
>
> On Thu, Mar 8, 2012 at 10:58 AM, Yongiljang  wrote:
>
> > Dear all,
> > I'm an android developer in charge of sqlite database.
> >
> > Some days ago, I'd got a SIGBUS error from sqlite when there is no space
> > in current partition and WAL journal mode is used.
> > This error was occurred from memset function in libc that was called by
> > libsqlite and debugging information shows shm file was related to this
> > issue.
> >
> > Following sequence shows how to generate this error.
> >
> > 1) Make disk full
> > 2) Reboot device - wal and shm files are remained in some applications
> > folder
> > 3) Do 1) ~ 2) until free space remained under 32KB
> > 4) SIGBUS error is occurred randomly
> >
> > I'd tried to solve this problem and guessed following scenario.
> >
> > 1) shm file is truncated to zero size by calling robust_truncate function
> > when a new connection is opened in unixOpenSharedMemory
> > 2) shm file is extended to 32KB by calling robust_truncate function in
> > unixShmMap
> > 3) mmap function is called with 32KB length
> >
> > In my guess, problem is occurred from robust_truncate or mmap functions,
> > because of they didn't returned error code whether shm file is extended
> to
> > 32KB or not on disk full status.
> > robust_truncate and mmap may caused illegal memset operation because of
> > shm file actually doesn't have 32KB size, it may less than 32KB.
> > Interesting point is when I tested it with above scenario, there was a
> shm
> > file that has 32KB size.
> > It is impossible because of there was no space to make 32KB sized file in
> > current partiton.
> >
>
> We do not want to really make a file.  The purpose of the -shm file is
> merely to give a name to a block of memory that various processes accessing
> the database can share between themselves using mmap().  Ideally, the
> content of the -shm file remains the OS page cache and is never written to
> disk.
>
> Can you please try this experiment for us:  Beginning with a standard
> SQLite build (without your patches) recompile using
> -DSQLITE_SHM_DIRECTORY="/dev/shm".  That option will cause the shared
> memory file to be created in the /dev/shm directory rather than in the same
> directory as the database.  Since /dev/shm is not backed by disk (or flash)
> the problem should be solved.
>
> FWIW:  We looked at always putting the -shm files in a special directory
> like this when we were first designing WAL.  But we noticed that design
> fails if two programs in different chroot jails try to access the database
> at the same time, and so we switched to the current design of using the
> -shm file in the same directory as the database.
>
> Question:  Does anybody know of a better way to get memory shared among
> processes other than to create a fake file and mmap() it?  Are there some
> magic options to mmap() (perhaps Linux-only options) that prevent it from
> actually writing to disk?
>
>
> >
> > Whatever, I'd changed unixOpenSharedMemory to solve it.
> >
> > 1) make a shm file and write null data until 32KB if this file doesn't
> > exists
> >- 

Re: [sqlite] SIGBUS error in case of disk full with WAL mode

[Richard Hipp]

On Thu, Mar 8, 2012 at 11:45 AM, Black, Michael (IS)  wrote:

> Looks like this should work...
>

No, it won't work.  The memory has to be shared in common among all
connections to a particular database.  If two separate processes connection
to the same database, they must get the same block of shared memory.  If
they connect to different databases, they must get different blocks of
shared memory.



>
>
>
> From http://www.cs.cf.ac.uk/Dave/C/node27.html
>
>
>
> The following code fragment demonstrates a use of this to create a block
> of scratch storage in a program, at an address that the system chooses.:
>
> int fd;
> caddr_t result;
> if ((fd = open("/dev/zero", O_RDWR)) == -1)
>   return ((caddr_t)-1);
>
> result = mmap(0, len, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
> (void) close(fd);
>
>
>
>
> Michael D. Black
>
> Senior Scientist
>
> Advanced Analytics Directorate
>
> Advanced GEOINT Solutions Operating Unit
>
> Northrop Grumman Information Systems
>
> 
> From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org]
> on behalf of Richard Hipp [d...@sqlite.org]
> Sent: Thursday, March 08, 2012 10:35 AM
> To: General Discussion of SQLite Database
> Subject: EXT :Re: [sqlite] SIGBUS error in case of disk full with WAL mode
>
> On Thu, Mar 8, 2012 at 10:58 AM, Yongiljang  wrote:
>
> > Dear all,
> > I'm an android developer in charge of sqlite database.
> >
> > Some days ago, I'd got a SIGBUS error from sqlite when there is no space
> > in current partition and WAL journal mode is used.
> > This error was occurred from memset function in libc that was called by
> > libsqlite and debugging information shows shm file was related to this
> > issue.
> >
> > Following sequence shows how to generate this error.
> >
> > 1) Make disk full
> > 2) Reboot device - wal and shm files are remained in some applications
> > folder
> > 3) Do 1) ~ 2) until free space remained under 32KB
> > 4) SIGBUS error is occurred randomly
> >
> > I'd tried to solve this problem and guessed following scenario.
> >
> > 1) shm file is truncated to zero size by calling robust_truncate function
> > when a new connection is opened in unixOpenSharedMemory
> > 2) shm file is extended to 32KB by calling robust_truncate function in
> > unixShmMap
> > 3) mmap function is called with 32KB length
> >
> > In my guess, problem is occurred from robust_truncate or mmap functions,
> > because of they didn't returned error code whether shm file is extended
> to
> > 32KB or not on disk full status.
> > robust_truncate and mmap may caused illegal memset operation because of
> > shm file actually doesn't have 32KB size, it may less than 32KB.
> > Interesting point is when I tested it with above scenario, there was a
> shm
> > file that has 32KB size.
> > It is impossible because of there was no space to make 32KB sized file in
> > current partiton.
> >
>
> We do not want to really make a file.  The purpose of the -shm file is
> merely to give a name to a block of memory that various processes accessing
> the database can share between themselves using mmap().  Ideally, the
> content of the -shm file remains the OS page cache and is never written to
> disk.
>
> Can you please try this experiment for us:  Beginning with a standard
> SQLite build (without your patches) recompile using
> -DSQLITE_SHM_DIRECTORY="/dev/shm".  That option will cause the shared
> memory file to be created in the /dev/shm directory rather than in the same
> directory as the database.  Since /dev/shm is not backed by disk (or flash)
> the problem should be solved.
>
> FWIW:  We looked at always putting the -shm files in a special directory
> like this when we were first designing WAL.  But we noticed that design
> fails if two programs in different chroot jails try to access the database
> at the same time, and so we switched to the current design of using the
> -shm file in the same directory as the database.
>
> Question:  Does anybody know of a better way to get memory shared among
> processes other than to create a fake file and mmap() it?  Are there some
> magic options to mmap() (perhaps Linux-only options) that prevent it from
> actually writing to disk?
>
>
> >
> > Whatever, I'd changed unixOpenSharedMemory to solve it.
> >
> > 1) make a shm file and write null data until 32KB if this file doesn't
> > exists
> >- return SQLITE_FULL error when write operation is failed
> > 2) write null data until 32KB if this file exists and got write lock
> > instead of calling truncate function to shrink shm file to zero size
> >   - return same error code when it failed
> >
> > By changed source, I could solve this problem.
> > Sqlite returns SQLITE_FULL errors only without SIGBUS core dump.
> >
> > However, this instant code changing may not be a good solution.
> > I wish to get better comment or source patches from here.
> >
> > Thank you for reading this including my poor english. :)
> >

Re: [sqlite] SIGBUS error in case of disk full with WAL mode

[Black, Michael (IS)]

Looks like this should work...



>From http://www.cs.cf.ac.uk/Dave/C/node27.html



The following code fragment demonstrates a use of this to create a block of 
scratch storage in a program, at an address that the system chooses.:

int fd;
caddr_t result;
if ((fd = open("/dev/zero", O_RDWR)) == -1)
   return ((caddr_t)-1);

result = mmap(0, len, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
(void) close(fd);




Michael D. Black

Senior Scientist

Advanced Analytics Directorate

Advanced GEOINT Solutions Operating Unit

Northrop Grumman Information Systems


From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on 
behalf of Richard Hipp [d...@sqlite.org]
Sent: Thursday, March 08, 2012 10:35 AM
To: General Discussion of SQLite Database
Subject: EXT :Re: [sqlite] SIGBUS error in case of disk full with WAL mode

On Thu, Mar 8, 2012 at 10:58 AM, Yongiljang  wrote:

> Dear all,
> I'm an android developer in charge of sqlite database.
>
> Some days ago, I'd got a SIGBUS error from sqlite when there is no space
> in current partition and WAL journal mode is used.
> This error was occurred from memset function in libc that was called by
> libsqlite and debugging information shows shm file was related to this
> issue.
>
> Following sequence shows how to generate this error.
>
> 1) Make disk full
> 2) Reboot device - wal and shm files are remained in some applications
> folder
> 3) Do 1) ~ 2) until free space remained under 32KB
> 4) SIGBUS error is occurred randomly
>
> I'd tried to solve this problem and guessed following scenario.
>
> 1) shm file is truncated to zero size by calling robust_truncate function
> when a new connection is opened in unixOpenSharedMemory
> 2) shm file is extended to 32KB by calling robust_truncate function in
> unixShmMap
> 3) mmap function is called with 32KB length
>
> In my guess, problem is occurred from robust_truncate or mmap functions,
> because of they didn't returned error code whether shm file is extended to
> 32KB or not on disk full status.
> robust_truncate and mmap may caused illegal memset operation because of
> shm file actually doesn't have 32KB size, it may less than 32KB.
> Interesting point is when I tested it with above scenario, there was a shm
> file that has 32KB size.
> It is impossible because of there was no space to make 32KB sized file in
> current partiton.
>

We do not want to really make a file.  The purpose of the -shm file is
merely to give a name to a block of memory that various processes accessing
the database can share between themselves using mmap().  Ideally, the
content of the -shm file remains the OS page cache and is never written to
disk.

Can you please try this experiment for us:  Beginning with a standard
SQLite build (without your patches) recompile using
-DSQLITE_SHM_DIRECTORY="/dev/shm".  That option will cause the shared
memory file to be created in the /dev/shm directory rather than in the same
directory as the database.  Since /dev/shm is not backed by disk (or flash)
the problem should be solved.

FWIW:  We looked at always putting the -shm files in a special directory
like this when we were first designing WAL.  But we noticed that design
fails if two programs in different chroot jails try to access the database
at the same time, and so we switched to the current design of using the
-shm file in the same directory as the database.

Question:  Does anybody know of a better way to get memory shared among
processes other than to create a fake file and mmap() it?  Are there some
magic options to mmap() (perhaps Linux-only options) that prevent it from
actually writing to disk?


>
> Whatever, I'd changed unixOpenSharedMemory to solve it.
>
> 1) make a shm file and write null data until 32KB if this file doesn't
> exists
>- return SQLITE_FULL error when write operation is failed
> 2) write null data until 32KB if this file exists and got write lock
> instead of calling truncate function to shrink shm file to zero size
>   - return same error code when it failed
>
> By changed source, I could solve this problem.
> Sqlite returns SQLITE_FULL errors only without SIGBUS core dump.
>
> However, this instant code changing may not be a good solution.
> I wish to get better comment or source patches from here.
>
> Thank you for reading this including my poor english. :)
>
> Best wishes,
> Jang.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



--
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SIGBUS error in case of disk full with WAL mode

[Richard Hipp]

On Thu, Mar 8, 2012 at 10:58 AM, Yongiljang  wrote:

> Dear all,
> I'm an android developer in charge of sqlite database.
>
> Some days ago, I'd got a SIGBUS error from sqlite when there is no space
> in current partition and WAL journal mode is used.
> This error was occurred from memset function in libc that was called by
> libsqlite and debugging information shows shm file was related to this
> issue.
>
> Following sequence shows how to generate this error.
>
> 1) Make disk full
> 2) Reboot device - wal and shm files are remained in some applications
> folder
> 3) Do 1) ~ 2) until free space remained under 32KB
> 4) SIGBUS error is occurred randomly
>
> I'd tried to solve this problem and guessed following scenario.
>
> 1) shm file is truncated to zero size by calling robust_truncate function
> when a new connection is opened in unixOpenSharedMemory
> 2) shm file is extended to 32KB by calling robust_truncate function in
> unixShmMap
> 3) mmap function is called with 32KB length
>
> In my guess, problem is occurred from robust_truncate or mmap functions,
> because of they didn't returned error code whether shm file is extended to
> 32KB or not on disk full status.
> robust_truncate and mmap may caused illegal memset operation because of
> shm file actually doesn't have 32KB size, it may less than 32KB.
> Interesting point is when I tested it with above scenario, there was a shm
> file that has 32KB size.
> It is impossible because of there was no space to make 32KB sized file in
> current partiton.
>

We do not want to really make a file.  The purpose of the -shm file is
merely to give a name to a block of memory that various processes accessing
the database can share between themselves using mmap().  Ideally, the
content of the -shm file remains the OS page cache and is never written to
disk.

Can you please try this experiment for us:  Beginning with a standard
SQLite build (without your patches) recompile using
-DSQLITE_SHM_DIRECTORY="/dev/shm".  That option will cause the shared
memory file to be created in the /dev/shm directory rather than in the same
directory as the database.  Since /dev/shm is not backed by disk (or flash)
the problem should be solved.

FWIW:  We looked at always putting the -shm files in a special directory
like this when we were first designing WAL.  But we noticed that design
fails if two programs in different chroot jails try to access the database
at the same time, and so we switched to the current design of using the
-shm file in the same directory as the database.

Question:  Does anybody know of a better way to get memory shared among
processes other than to create a fake file and mmap() it?  Are there some
magic options to mmap() (perhaps Linux-only options) that prevent it from
actually writing to disk?


>
> Whatever, I'd changed unixOpenSharedMemory to solve it.
>
> 1) make a shm file and write null data until 32KB if this file doesn't
> exists
>- return SQLITE_FULL error when write operation is failed
> 2) write null data until 32KB if this file exists and got write lock
> instead of calling truncate function to shrink shm file to zero size
>   - return same error code when it failed
>
> By changed source, I could solve this problem.
> Sqlite returns SQLITE_FULL errors only without SIGBUS core dump.
>
> However, this instant code changing may not be a good solution.
> I wish to get better comment or source patches from here.
>
> Thank you for reading this including my poor english. :)
>
> Best wishes,
> Jang.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] SIGBUS error in case of disk full with WAL mode

[Yongiljang]

Dear all,
I'm an android developer in charge of sqlite database.

Some days ago, I'd got a SIGBUS error from sqlite when there is no space in 
current partition and WAL journal mode is used.
This error was occurred from memset function in libc that was called by 
libsqlite and debugging information shows shm file was related to this issue.

Following sequence shows how to generate this error.

1) Make disk full
2) Reboot device - wal and shm files are remained in some applications folder
3) Do 1) ~ 2) until free space remained under 32KB
4) SIGBUS error is occurred randomly

I'd tried to solve this problem and guessed following scenario.

1) shm file is truncated to zero size by calling robust_truncate function when 
a new connection is opened in unixOpenSharedMemory
2) shm file is extended to 32KB by calling robust_truncate function in 
unixShmMap
3) mmap function is called with 32KB length

In my guess, problem is occurred from robust_truncate or mmap functions, 
because of they didn't returned error code whether shm file is extended to 32KB 
or not on disk full status.
robust_truncate and mmap may caused illegal memset operation because of shm 
file actually doesn't have 32KB size, it may less than 32KB.
Interesting point is when I tested it with above scenario, there was a shm file 
that has 32KB size.
It is impossible because of there was no space to make 32KB sized file in 
current partiton.

Whatever, I'd changed unixOpenSharedMemory to solve it.

1) make a shm file and write null data until 32KB if this file doesn't exists
- return SQLITE_FULL error when write operation is failed
2) write null data until 32KB if this file exists and got write lock instead of 
calling truncate function to shrink shm file to zero size
   - return same error code when it failed

By changed source, I could solve this problem.
Sqlite returns SQLITE_FULL errors only without SIGBUS core dump.

However, this instant code changing may not be a good solution.
I wish to get better comment or source patches from here.

Thank you for reading this including my poor english. :)

Best wishes,
Jang.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users