Re: [sqlite] [PATCH] Bug fix - Makefile.in: add extension 'BEXE' to prerequisite 'lemon' (for platforms with non-void BEXE)
On 12/25/19, Jannick wrote: > I hope sending to this list an email with the patch in the email body is OK. Tnx. Implemented at https://sqlite.org/src/info/f482a4cdfa768941 -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] oserror-2.1.x fails when run on an BTRFS volume - directories may have any st_size.
Tnx for the report. Should be fixed as of
https://sqlite.org/src/info/c8c6dd0e6582ec91
Please do us the favor of trying this out on both Btrfs and XFS and
making sure it works correctly on both filesystems. Tnx.
On 12/25/19, Stefan Brüns wrote:
> The oserror-2.1.1 test fails, as a exisiting test.db-wal is silently
> ignored.
>
> Running this small example on e.g. XFS or tmpfs yields the following:
> $> mkdir test.db-wal
> $> strace -efile sqlite3 test.db
> ...
> sqlite> .databases
> openat(AT_FDCWD, "test.db", O_RDONLY) = -1 ENOENT (Datei oder Verzeichnis
>
> nicht gefunden)
> lstat("test.db", 0x7ffc38d3b8c0)= -1 ENOENT (Datei oder Verzeichnis
>
> nicht gefunden)
> getcwd("/home/stefan", 511) = 13
> openat(AT_FDCWD, "/home/stefan/test.db", O_RDWR|O_CREAT|O_CLOEXEC, 0644) =
> 3
> stat("/home/stefan/test.db", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
> stat("/home/stefan/test.db-journal", 0x7ffc38d3a730) = -1 ENOENT (Datei oder
>
> Verzeichnis nicht gefunden)
> stat("/home/stefan/test.db-wal", {st_mode=S_IFDIR|0755, st_size=6, ...}) =
> 0
> unlink("/home/stefan/test.db-wal") = -1 EISDIR (Ist ein Verzeichnis)
> Error: disk I/O error
>
> Doing the same on an btrfs volume:
> $> mkdir test.db-wal
> $> strace -efile sqlite3 test.db
> ...
> sqlite> .databases
> stat("/var/tmp/t/test.db-journal", 0x7fff01578140) = -1 ENOENT (Datei oder
> Verzeichnis nicht gefunden)
> stat("/var/tmp/t/test.db-wal", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
> stat("/var/tmp/t/test.db-journal", 0x7fff015790e0) = -1 ENOENT (Datei oder
> Verzeichnis nicht gefunden)
> stat("/var/tmp/t/test.db-wal", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
> main: /var/tmp/t/test.db
> sqlite>
>
> The culprit is the st_size check in os_unix.c/unixAccess:
> ...
> if( flags==SQLITE_ACCESS_EXISTS ){
> struct stat buf;
> *pResOut = (0==osStat(zPath, ) && buf.st_size>0);
> }else{
> ...
>
> Changing it to ".. && ((buf.st_size>0) || (buf.st_mode & S_IFMT ==
> S_IFDIR)));" fixes the problem.
>
>
> --
> Stefan Brüns / Bergstraße 21 / 52062 Aachen
> home: +49 241 53809034 mobile: +49 151 50412019
--
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] fts3corrupt4-33.0 depends on SQLITE_ENABLE_ICU
On 12/25/19, Stefan Brüns wrote:
> The test added in https://www.sqlite.org/src/info/e01fdbf9f700e1bd
>
> requires icu to run successfully. Otherwise it errors out with:
>
> ! fts3corrupt4-33.0 expected: [1 {database disk image is malformed}]
> ! fts3corrupt4-33.0 got: [1 {unknown tokenizer: icu}]
Tnx for the report. Should be fixed in
https://sqlite.org/src/info/19c6240bdbb022b2
--
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] oserror-2.1.x fails when run on an BTRFS volume - directories may have any st_size.
The oserror-2.1.1 test fails, as a exisiting test.db-wal is silently ignored.
Running this small example on e.g. XFS or tmpfs yields the following:
$> mkdir test.db-wal
$> strace -efile sqlite3 test.db
...
sqlite> .databases
openat(AT_FDCWD, "test.db", O_RDONLY) = -1 ENOENT (Datei oder Verzeichnis
nicht gefunden)
lstat("test.db", 0x7ffc38d3b8c0)= -1 ENOENT (Datei oder Verzeichnis
nicht gefunden)
getcwd("/home/stefan", 511) = 13
openat(AT_FDCWD, "/home/stefan/test.db", O_RDWR|O_CREAT|O_CLOEXEC, 0644) = 3
stat("/home/stefan/test.db", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
stat("/home/stefan/test.db-journal", 0x7ffc38d3a730) = -1 ENOENT (Datei oder
Verzeichnis nicht gefunden)
stat("/home/stefan/test.db-wal", {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0
unlink("/home/stefan/test.db-wal") = -1 EISDIR (Ist ein Verzeichnis)
Error: disk I/O error
Doing the same on an btrfs volume:
$> mkdir test.db-wal
$> strace -efile sqlite3 test.db
...
sqlite> .databases
stat("/var/tmp/t/test.db-journal", 0x7fff01578140) = -1 ENOENT (Datei oder
Verzeichnis nicht gefunden)
stat("/var/tmp/t/test.db-wal", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("/var/tmp/t/test.db-journal", 0x7fff015790e0) = -1 ENOENT (Datei oder
Verzeichnis nicht gefunden)
stat("/var/tmp/t/test.db-wal", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
main: /var/tmp/t/test.db
sqlite>
The culprit is the st_size check in os_unix.c/unixAccess:
...
if( flags==SQLITE_ACCESS_EXISTS ){
struct stat buf;
*pResOut = (0==osStat(zPath, ) && buf.st_size>0);
}else{
...
Changing it to ".. && ((buf.st_size>0) || (buf.st_mode & S_IFMT ==
S_IFDIR)));" fixes the problem.
--
Stefan Brüns / Bergstraße 21 / 52062 Aachen
home: +49 241 53809034 mobile: +49 151 50412019
signature.asc
Description: This is a digitally signed message part.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Misleading comment in various test/fts*.test files
In the various test files, there (in most cases) is a misleading comment.
In i.e. fts3expr.test, the comment is inverted, in fts3fault2 it is correct:
test/fts3expr.test:# If SQLITE_ENABLE_FTS3 is defined, omit this file.
test/fts3expr.test-ifcapable !fts3 {
--
test/fts3fault2.test:# If SQLITE_ENABLE_FTS3 is not defined, omit this file.
test/fts3fault2.test-ifcapable !fts3 { finish_test ; return }
Kind regards, Stefan
--
Stefan Brüns / Bergstraße 21 / 52062 Aachen
home: +49 241 53809034 mobile: +49 151 50412019
signature.asc
Description: This is a digitally signed message part.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] fts3corrupt4-33.0 depends on SQLITE_ENABLE_ICU
The test added in https://www.sqlite.org/src/info/e01fdbf9f700e1bd
requires icu to run successfully. Otherwise it errors out with:
! fts3corrupt4-33.0 expected: [1 {database disk image is malformed}]
! fts3corrupt4-33.0 got: [1 {unknown tokenizer: icu}]
Kind regards, Stefan
--
Stefan Brüns / Bergstraße 21 / 52062 Aachen
home: +49 241 53809034 mobile: +49 151 50412019
signature.asc
Description: This is a digitally signed message part.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
On 12/25/19, Doug wrote: > Richard, can you please explain each of these? > > 1. API break > I wrote an application in Qt which uses SQLite. Therefore, I invoke SQLite > functions with some wrapper. For a 9% performance improvement in SQLite > using the direct call versus indirect call (as discussed in the talk), > cannot the wrapper functions be changed so my application doesn't know the > difference? You can completely disable all of the mutexes by compiling with SQLITE_THREADSAFE=0. That works fine, as long as you don't use any SQLite API in more than one thread at a time. And it does, definitely, make SQLite run faster. If you feel like you have to use threads, then you can run SQLite in multi-thread mode and most of the mutex calls will be omitted. Multi-thread mode allows multiple threads to use SQLite at the same time, as long as every thread is using a different database connection. If you run in serialized threading mode, then there will be many mutex calls. There is no way around that. Threading modes described here: https://www.sqlite.org/threadsafe.html > > 2. Render SQLITE untestable > Does that mean that you are doing whitebox testing? Surely, all the > thousands of queries vs responses are blackbox, not whitebox. Why would > changing indirect calls to direct calls render SQLite untestable? By "untestable" I mean that we would be unable to obtain 100% MC/DC (essentually 100% branch test coverage) in an SQLite compiled as for delivery. We go by the philosophy that "If it isn't tested, then it doesn't work" and so if there are branches that are unreachable by our tests, then SQLite is "untestable". We also go by "fly what you test and test what you fly", so adding a compile-time option that allows mutex calls to be intercepted and redirected in testing builds but not in release builds won't work for us. > > 3. Unable to replicate performance gains > This says to me you actually made the change suggested. And then you ran a > test suite against the amalgamation. And you actually measured the result. > How can you have done that if such a change renders SQLite untestable? And > (sneaking a peak at the talk again re performance measurements), what did > you use to measure the results? > The video provided details on what they did. I could not find any performance improvement by making mutexes direct calls instead of indirect calls. Maybe I did something wrong. Maybe it depends on your compiler and or optimization options and I didn't use the right combination. Maybe they are measuring performance differently than me. (I use CPU cycle counts measured by valgrind.) Maybe the team that studied this made a mistake in their testing and using indirect calls to mutexes really doesn't matter that much after all. Maybe the problem is some combination of all of the above. I don't know. If you want to try to replicate the performance improvement yourself, and report your detailed findings on this mailing list, you are welcomed to do so. If you have a reproducible test case - perhaps we will go in and provide a compile-time option that makes SQLite run faster in intensely multi-threaded applications at the expense of also rendering it untestable -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
Richard, can you please explain each of these? 1. API break I wrote an application in Qt which uses SQLite. Therefore, I invoke SQLite functions with some wrapper. For a 9% performance improvement in SQLite using the direct call versus indirect call (as discussed in the talk), cannot the wrapper functions be changed so my application doesn't know the difference? 2. Render SQLITE untestable Does that mean that you are doing whitebox testing? Surely, all the thousands of queries vs responses are blackbox, not whitebox. Why would changing indirect calls to direct calls render SQLite untestable? 3. Unable to replicate performance gains This says to me you actually made the change suggested. And then you ran a test suite against the amalgamation. And you actually measured the result. How can you have done that if such a change renders SQLite untestable? And (sneaking a peak at the talk again re performance measurements), what did you use to measure the results? Best, Doug > -Original Message- > From: sqlite-users > On Behalf Of Richard Hipp > Sent: Wednesday, December 25, 2019 3:18 PM > To: SQLite mailing list > Subject: Re: [sqlite] Causal profiling > > On 12/25/19, sky5w...@gmail.com wrote: > > Thanks for sharing! > > Did his suggested optimization make it to a commit? > > No. That would be an API break, and would also render SQLite > untestable. Furthermore, we have been unable to replicate the > performance gains. > -- > D. Richard Hipp > d...@sqlite.org > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite- > users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
On 12/25/19, sky5w...@gmail.com wrote: > Thanks for sharing! > Did his suggested optimization make it to a commit? No. That would be an API break, and would also render SQLite untestable. Furthermore, we have been unable to replicate the performance gains. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
Thanks for sharing! Did his suggested optimization make it to a commit? On Wed, Dec 25, 2019 at 10:46 AM Craig H Maynard wrote: > All, > > Just watched an interesting lecture by UMass professor Emery Berger on > improving software performance: > > https://www.youtube.com/watch?v=r-TLSBdHe1A > > SQLite is discussed in the section on causal profiling, which begins at > 34:12. > > Craig > > -- > Craig H Maynard > Rhode Island, USA > 401.413.2376 > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Causal profiling
All, Just watched an interesting lecture by UMass professor Emery Berger on improving software performance: https://www.youtube.com/watch?v=r-TLSBdHe1A SQLite is discussed in the section on causal profiling, which begins at 34:12. Craig -- Craig H Maynard Rhode Island, USA 401.413.2376 ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] When will the next version abaliable? Or any schedule available?
On 12/22/19, Ming Ding wrote: > We are using SQLite 3.30.1(which release in 2019-10-10) in our project now. > > But there are 5 security vulnerabilities published recently, > CVE-2019-19317,CVE > -2019-19244,CVE-2019-19603,CVE-2019-19645,CVE-2019-19646. We do not have an anticipated release date for 3.31.0 at this time. You may safely ignore all of the CVEs above. One of those CVEs is simply wrong. The other four only come into play if you allow unauthenticated users to enter arbitrary SQL statements into SQLite, and even in that case, they only allow for a denial-of-service attack, as far as I know. You can also deploy with a prerelease snapshot, which fixes all of the CVEs. Download a prerelease snapshot from the https://sqlite.org/download.html page. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] When will the next version abaliable? Or any schedule available?
We are using SQLite 3.30.1(which release in 2019-10-10) in our project now. But there are 5 security vulnerabilities published recently, CVE-2019-19317,CVE -2019-19244,CVE-2019-19603,CVE-2019-19645,CVE-2019-19646. I found that both of them can be fixed by using the patch from github. I will appreciate it so much if there is a version (maybe 3.30.2A )including all the bugs above fixed recently! Best, Yxd ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] CVE's opened on 3.30.1 status
On 12/25/19, Dominique Pellé wrote: > > According to https://sqlite.org/draft/releaselog/3_31_0.html > SQLite-3.31 is scheduled for 2019-12-31. > I have idea how accurate this date is though. :-) I made that estimate on 2019-10-29. Now that we are closer to the date, I can confidently predict that the release will be delayed. February maybe? That is a guess, not a promise! -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] CVE's opened on 3.30.1 status
Raitses, Alex wrote: > Thanks a lot for the prompt response, > As far as I found in Fossil repository fixes for all CVE's , > excepting erroneously submitted CVE-2019-19646, were merged to Fossil. > Can you please estimate next official release of SQLite including these fixes? According to https://sqlite.org/draft/releaselog/3_31_0.html SQLite-3.31 is scheduled for 2019-12-31. I have idea how accurate this date is though. Regards Dominique ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Long long int constants in sources
On Wed, 25 Dec 2019 at 01:24, Clemens Ladisch wrote: > Max Vlasov wrote: > > bcc 5.5 compiler ... didn't like long long constants > >such as -2251799813685248LL > > If you want to make this particular compiler happy, use > -2251799813685248i64. > Thanks, this helped when I tested this scenario. Interesting that while searching and replacing I found a comment existing in 3.6.10 version (the earliest I have) and probably prior that addressed the problems with exposing 64-bit constants to compilers > /* Some compilers complain about constants of the form 0x7fff. > ** Others complain about 0x7i64. The following macro seems > ** to provide the constant while making all compilers happy. >*/ > # define MAX_ROWID (i64)( (((u64)0x7fff)<<32) | (u64)0x ) So probably it's hard to keep making all compilers (old and young) happy all the time :). Thanks to Richard and the team for at least trying to make them as happy as it's possible. Max ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] [PATCH] Bug fix - Makefile.in: add extension 'BEXE' to prerequisite 'lemon' (for platforms with non-void BEXE)
I hope sending to this list an email with the patch in the email body is OK. Thanks, J. = diff --git a/Makefile.in b/Makefile.in index 015796b65..e4b824365 100644 --- a/Makefile.in +++ b/Makefile.in @@ -1177,7 +1177,7 @@ FTS5_SRC = \ $(TOP)/ext/fts5/fts5_varint.c \ $(TOP)/ext/fts5/fts5_vocab.c \ -fts5parse.c: $(TOP)/ext/fts5/fts5parse.y lemon +fts5parse.c: $(TOP)/ext/fts5/fts5parse.y lemon$(BEXE) cp $(TOP)/ext/fts5/fts5parse.y . rm -f fts5parse.h ./lemon$(BEXE) $(OPTS) -S fts5parse.y ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
