[sqlite] The value of nFetchOut and zPath after wal-file calls sqlite3OsOpen()

[advancenOO]

I want to analyze the executive process of wal-file with the source code
3.16.2 .
And I added:
unixFile *pFd = (unixFile *)pRet->pWalFd;
if(pFd->nFetchOut != 0){
printf("hi\n");
}
to sqlite3WalOpen() after sqlite3OsOpen() is called:
rc = sqlite3OsOpen(pVfs, zWalName, pRet->pWalFd, vfsFlags, );

I think sqlite3OsOpen() should open the pWalFd file and set the most of its
member elements in struct unixFile to 0. But I saw so many 'hi' when I ran
the TCL test of sqlite, especially in e_walckpt.test, wal.test and
wal2.test.

I find nFetchOut has been correctly initialized after fillInUnixFile() is
called before unixOpen() returned. But nFetchOut, and sometimes zPath,
mmapSize and other data elements seem to have an incorrect value when I
check it after sqliteOsOpen().

Is it right? And why?

Thanks 



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Does sqlite have official development testing tool?

[advancenOO]

Thanks for all your suggestions. 
Is it correct to add my compile-options(such as
-DSQLITE_DEFAULT_MMAP_SIZE=0x7fff) just after CFLAGS in Makefile? As I
found some weird bugs if I do so. 
For example, sometimes wal2.test can pass but sometimes it reports "cannot
open savepoint - SQL statements in progress".
 
Sorry for all the questions...

Thanks.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Notification mechanism when truncate a mapped file

[advancenOO]

I am trying to make some change to the source code of sqlite as I want to map
a file using unixMapfile(). But I guess there will be a SIGBUS crash when
one process truncate the file and do not notify other processes who want to
access the file later.
So I am wondering if there is any appropriate mechanism in sqlite to notify
other processes?
Or, in another way, is it OK to update the mmapSize of file when I catch the
signal SIGBUS? As I guess I can use sigsetjmp() and siglongjmp() to restore
the environment.

Thanks.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Does sqlite have official development testing tool?

[advancenOO]

Hello Richard,

I hope to run some tests by myself and I think TCL tests in your link are
what I want.
There are so many .tcl and .test in Sqlite source tree.
Could someone share what commands I need to run to start all TCL tests? 

Thanks.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] the timing to truncate WAL file to journal_size_limit

[advancenOO]

I noticed that, 
“The journal_size_limit pragma may be used to limit the size of WAL files
left in the file-system after transactions or checkpoints. Each time a WAL
file resets, SQLite compares the size of the WAL file left in the
file-system to the size limit.”

But I think only when the first transaction commits AFTER a checkpoint will
WAL file truncate to the limit. As the src code is, 
if( isCommit && pWal->truncateOnCommit && pWal->mxWalSize>=0 ). 

Which means WAL file may still consume a large amount of space after
checkpoints, unless a new transaction commits. Is that right?



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Does sqlite have official development testing tool?

[advancenOO]

I am interested in the source code of sqlite and I want to make some change
to it.
I wondering if sqlite has official development testing to do a full
functional check or evaluate performance lose of my code?

I have found sqlspeedtest1,8 and sqlthreadtest3,4. Is there any other
testing I should make? 




--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[advancenOO]

OK, I know it is not safe when use mmap in Android 7 fuse. 
And do you mean the bug is in fuse kernel framework as you mentioned Android
FUSE 
filesystem driver? Which corresponding to Linux 3.10?
 
By the way, do you have any possible repro steps especially in Android
Sqlite?



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[advancenOO]

Yeah. Thanks for your suggestion about -DSQLITE_ENABLE_BATCH_ATOMIC_WRITE.
But the version of my sqlite is 3.16.2, so I guess I may use it at a later
time int the future.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[advancenOO]

Thanks for the notification about the bugs in MacOS.
So the most important reason for disable SQLITE_MMAP_READWRITE is the bugs
in OS, not the stray pointer in upper applications. Is it right?



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[advancenOO]

I am a developer of system, not application. So I have nothing to do with the
SQL or indices even though I know these may cut orders of magnitude from
execution times. 

And that is why I use speedtest1.c of Sqlite to evaluate the performance of
my system.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[advancenOO]

Um, stray pointer or array overrun in upper applications will be treated as
BUG in my system. And I believe all these bugs could be fixed before using
sqlite.
So can I safely enable SQLITE_MMAP_READWRITE directly if I can ignore the
possibility of the BUG?

But I notice that Sqlite may not call msync() in mmap, calls fsync instead. 
So I am wondering if it is safe to use SQLITE_MMAP_READWRITE to write to
mmapped memory as msync() is not called.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Enabling MMAP in Android

[advancenOO]

I find MMAP is able to be used for db files(getPageMMap()) but is not enabled
by default.
I wonder if it is recommended to use MMAP in Android? As I think MMAP is one
way to improve the speed performance(I use speedtest1.c to evaluate).

At the same time, Mmap() is only used for reading by default, not writing.
But why? And in what situation I can compile with the
-DSQLITE_MMAP_READWRITE option?

By the way, is there any other recommended compile-options? I find some
references in  http://www.sqlite.org/compile.html
  , but I am wondering if there is any
other method to enhance performance?

Thanks for your help.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] A walIndexAppend() crash after calling "PRAGMA journal_mode"

[advancenOO]

I tracked down a occasional system crash problem of calling "PRAGMA
journal_mode" and found that walIndexRecover()->walIndexAppend() is finally
called before the crash according to my callstack. 

In sqlite3.c, walIndexAppend() can be called after walIndexRecover() only
when nSize, the size of WAL-file, is bigger than WAL_HDRSIZE. But I do not
think this could happen when I call "PRAGMA journal_mode" as the DB
connection is first open at that time. 

I am not familiar with the operations of upper APPs and this crash makes me
confuse. 

I guess there may be two threads access the same db at the same time and
both of threads find the journal_mode is DELETE and then call PRAGMA. The
crash may happen when the second PRAGMA executes as the first thread may
have written something into the db. 
Is it possible? Or, in what other situation could this crash happen?

Thanks a lot.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Does wal-file support MMAP？

[advancenOO]

Purely for speed :)
I hope to improve the performance inside sqlite in order to optimize the
performance of the upper applications. As there are so many different scenes
in APPs.
 



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Does wal-file support MMAP？

[advancenOO]

In multi-thread scenario, I guess that threads may not use MMAP correctly
after one thread truncates wal-file. And that is the reason MMAP can not be
used together with WAL mode.

So, I am wondering if I can simply make the size of Wal-file a fixed size to
support MMAP for wal-file.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Does wal-file support MMAP？

[advancenOO]

I am trying to understand how SQLITE_DEFAULT_MMAP_SIZE works and I think db
files can use MMAP by using this compilation option, as Sqlite3PagerGet()
mapped to getPageMMap(). 
And I know "/the current VFS implementations use a mmapped file for the
wal-index/"

But is there a way that wal-files can use MMAP? Why not? As I guess it can
improve the IO performance in WAL mode.

Thanks for your insights.




--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Performance issue for version 3.9.2 compared with 3.16.2

[advancenOO]

In order to profile sqlite 3.9.2 and sqlite 3.16.2, I run speedtest1.c on my
mobile phone.
It is weird to find that 3.9.2 is better than 3.16.2, especially in the
following case:
“270, 1 DELETEs, numeric BETWEEN, indexed"
3.9.2 uses 0.7s while 3.16.2 uses 1.8s;

My relevant compilation options are shown below:
-DSQLITE_THREADSAFE=1
-DSQLITE_DEFAULT_AUTOVACUUM=1
-DSQLITE_DEFAULT_PAGE_SIZE=4096
-DSQLITE_DEFAULT_JOURNAL_SIZE_LIMIT=1048576

Further more, 3.16.2 performs better when I decrease the delete times from
1 to 1000.
So is it expected? If not, then what might be the possible reason? 

Thanks for your help.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users