Re: [sqlite] Causal profiling
D Burgess, on Wednesday, January 1, 2020 08:19 PM, wrote... > > > I’ve spent too much time lately trying to figure out or debug > hellacious C spaghetti code > > And I’ve spent too much time lately trying to figure out or debug > hellacious C++ spaghetti code > > Someone who writes bad C, will write even worse C++ Very true. And for that matter, any other language too. Except assembler. ;-) josé ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
Also spaghetti inheritance is a thing. On Wed, Jan 1, 2020, 19:19 D Burgess wrote: > > I’ve spent too much time lately trying to figure out or debug hellacious > C spaghetti code > > And I’ve spent too much time lately trying to figure out or debug > hellacious C++ spaghetti code > > Someone who writes bad C, will write even worse C++ > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
> I’ve spent too much time lately trying to figure out or debug hellacious C > spaghetti code And I’ve spent too much time lately trying to figure out or debug hellacious C++ spaghetti code Someone who writes bad C, will write even worse C++ ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
> On Dec 31, 2019, at 7:03 AM, Richard Damon wrote: > > 1) The C API between separate compilations is very well established, and > fully documented in most environments. Agreed. APIs between library boundaries generally need to be C. This is pretty easy to do though (I've done it multiple times in my current job.) You just write some C wrappers for the public methods. > On Dec 31, 2019, at 7:19 AM, Simon Slavin wrote: > > 5) SQLite has to work on your set top box. On your Wifi base station. On > your Home hub. On the machine that runs the car park. All these things have > C compilers. Not all these things have C++ compilers. This may have been an issue ten years ago, but is it still? I did some R on embedded systems last year, and C++ support looked pretty ubiquitous. The heftier embedded boards run Linux, the middleweight ones have C++-friendly environments like mbedOS or ESP32-IDF, and even tiny 8-bit Arduino microcontrollers have an OO C++ API, even though they call it C to avoid scaring the newbies. —Jens ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
On 31 Dec 2019, at 3:03pm, Richard Damon wrote: > I will give a short answer, in my opinion, of a few reasons on why C is still > a viable language. 5) SQLite has to work on your set top box. On your Wifi base station. On your Home hub. On the machine that runs the car park. All these things have C compilers. Not all these things have C++ compilers. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
On 12/31/19 3:13 AM, Jens Alfke wrote: On Dec 30, 2019, at 6:23 PM, Alexander Vega wrote: -fdevirtualize -fdevirtualize-speculatively I believe those are C++ optimizations, specific to C++ virtual methods. What SQLite is doing is similar to a class with virtual methods, but done “by hand” in plain C, so the optimizer doesn’t know how to apply that optimization. (I’m fighting the urge to write an off-topic rant about why anyone still programs in C when they could write roughly the same code much more simply and cleanly in C++ using only the most basic features like classes and methods. I’ve spent too much time lately trying to figure out or debug hellacious C spaghetti code that’s awkwardly reinventing wheels like inheritance and constructors/destructors.) And in any case, de-virtualization only works when a virtual method has only one implementation, I.e. is monomorphic, and that isn’t true here. —Jens I will give a short answer, in my opinion, of a few reasons on why C is still a viable language. 1) The C API between separate compilations is very well established, and fully documented in most environments. C++ is less so, where sometimes important details like how to mangle names and occasionally handle exceptions not fully standardized between compilers on a given platform. 2) The way language support packages work, it is fully defined to have a C library in a C++ program, but it might not work to have a C++ library in a C program, you need to have a C++ compatible startup package (to run the constructors etc.). Changing a library from C to C++ is thus a backwards breaking change unless EVERYONE has migrated to C++. 3) The C language is much more stable. While some new features have been added to C in the last decade, the language has been very stable (yes, some legacy features that were included in C90 were later removed but that was for things that weren't recommended anyway). It is quite possible to have a code base designed to build on a C99 implementation (or even C90, perhaps with a few replacement headers like stdint.h for some very useful additions from C99) and have it still work with a totally up to date system. The C++ language has been much more fluid in language definition. 4) There still exists some embedded system that don't have a free C++ compiler, so such a change restricts it from some environment (admittedly, many of those may not have the resources to use SQLite well). As an aside for that, SQLites 'virtual' functions aren't implementing something easy to do in C++ (except by doing the same thing), as to do what is being done you would have to change the 'type' of the SQLite 'object'. -- Richard Damon ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
> On Dec 30, 2019, at 6:23 PM, Alexander Vega wrote: > > -fdevirtualize > -fdevirtualize-speculatively I believe those are C++ optimizations, specific to C++ virtual methods. What SQLite is doing is similar to a class with virtual methods, but done “by hand” in plain C, so the optimizer doesn’t know how to apply that optimization. (I’m fighting the urge to write an off-topic rant about why anyone still programs in C when they could write roughly the same code much more simply and cleanly in C++ using only the most basic features like classes and methods. I’ve spent too much time lately trying to figure out or debug hellacious C spaghetti code that’s awkwardly reinventing wheels like inheritance and constructors/destructors.) And in any case, de-virtualization only works when a virtual method has only one implementation, I.e. is monomorphic, and that isn’t true here. —Jens ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
On 31 Dec 2019, at 2:21am, Alexander Vega wrote: > Could there be some compile time option to force Linux or Windows > statically? One of the problems with this is that there are a lot of utility libraries built around SQLite. These are libraries provided for general-purpose use, often a language shim to make it easy to call SQLite from some language which isn't C or C++. Compiler options which make SQLite behave significantly different generate fault reports (support calls, messages to this list, etc.). Because someone gets the 'different' version of SQLite and wonders why it doesn't do what the version they're used to does. If SQLite3 ever moves to SQLite4, a big change in behaviour may be acceptable: people are used to that in full version increases. But while SQLite remains version 3.x.x, an option like that would have to be thought through very carefully. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
I watched the video and wondered about the virtual table calls within sqlite as well. Without them you would have no VFS though... so they are needed. Could there be some compile time option to force Linux or Windows statically? Maybe. I do not know the effect of collecting debug information using -g, but if using -O2 to compile sqlite with GCC the following optimizations from https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#Optimize-Options are enabled: -fdevirtualize -fdevirtualize-speculatively So depending on what compiler used and version etc, those virtual calls could all disappear anyway. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
> On Dec 30, 2019, at 7:19 AM, Doug wrote: > > I am suggesting that if the we added the global calls to the underlying > functions to the API - that is, the functions that are called by the function > table indirection - then one could code the application to call the > underlying functions. What you're describing is basically the SQLITE_CONFIG_SINGLETHREAD compile option. In that configuration SQLite does not use any mutexes, and you're responsible for "call[ing] the underlying functions" (the mutex lock/unlock functions) yourself. However, in this configuration any client code that makes concurrent calls to SQLite — and there is a lot of code like that — will crash and burn. > If you are going to tell me that you need to maintain two versions of your > application if you run it in a single-thread environment or a multi-thread > environment I don't know if someone else was saying that; what I was pointing out earlier is that many clients of SQLite link to it as a shared library provided by the OS, either built-in (as on iOS and macOS) or installed by a central package manager (Linux). This means they don't specify compile-time options, they implicitly get the behavior configured by the OS or package vendor, which I'd guess is probably MULTITHREAD. If the SQLite they link with changes its behavior to avoid mutexes, many of these programs will break. > Thinking about it, I'm surprised that the C API isn't just a set of macros > already. What you're describing is basically the effect of link-time optimization (LTO). The optimizer runs over the entire program's compiled code (not just one compilation unit) and applies optimizations such as inlining. This can indeed have a big impact. —Jens ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
On 12/30/19 10:19 AM, Doug wrote: OK, I get that the definition of the SQLite API is a (large) set of C function calls. And that changing the way they work under the covers would break something. (Check out the IDEA below!) I'm wondering if your use case is - in the same application - that you dynamically change from option SQLITE_CONFIG_SINGLETHREAD to SQLITE_CONFIG_MULTITHREAD to SQLITE_CONFIG_SERIALIZED while your application is running? If that is the case, then your application knows which option to use dynamically; Otherwise, your application doesn't know and doesn't care which option is in effect. I am suggesting that if the we added the global calls to the underlying functions to the API - that is, the functions that are called by the function table indirection - then one could code the application to call the underlying functions. If the application knows it's single-thread, then code it that way and get a 25% improvement (see the talk). If the application makes the choice of thread option dynamically, then the penalty for single-thread is at least double (application choice, SQLite indirection), so calling the (new) underlying function once you made the choice, performs better for that path. I grant you that probably will see little improvement on the threaded path. If you are going to tell me that you need to maintain two versions of your application if you run it in a single-thread environment or a multi-thread environment, then let's define the (new) API to use a preprocessor macro to generate the right code for the option selected based on the values of SQLITE_CONFIG_SINGLETHREAD, el al. So now you have a single source but multiple executables corresponding to that source. And the choice of which executable to use becomes a configuration problem at application deployment time, or at application run time. IDEA! Thinking about it, I'm surprised that the C API isn't just a set of macros already. I can visualize a C API composed of a set of macro definitions _identical_ to the current C function calls. They just use the extra knowledge of SQLITE_THREADSAFE and other SQLite compiler options to decide what application code to generate. Then the formal API doesn't change from a coding point of view. The generated code calls a set of under-the-cover functions which are not part of the API. The change doesn't require a new layer of testing; presumably, we already have test cases that test the same code using different compiler options. What about that? Best, Doug One BIG reason the C API can't be made into a set of macros is that the C API is basically the multi-language API for using the SQLite shared library in other languages. This requires that the C API functions be REAL functions that provide entry points into the shared library. It also, perhaps as a much more minor point, prevents taking the address of those functions to use in the application either to pass SQLite routines as call backs or make build your own virtual functions (admittedly, I don't know of a case where you would really want to do that). I suspect that by far the vast majority of SQLite uses don't bundle the SQLite source code into the project, but link to it as an external resource. Yes, there is perhaps an option to provide some specific configuration macros to allow SQLite to be optimized when included statically in a project, but those options shouldn't change the API. -- Richard Damon ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
On Wed, Dec 25, 2019 at 4:25 PM Richard Hipp wrote: > > The video provided details on what they did. I could not find any > performance improvement by making mutexes direct calls instead of > indirect calls. This kind of optimization improves performance by reducing pressure on the CPU's branch branch-target address caching and prediction. > Maybe they are measuring performance differently than > me. (I use CPU cycle counts measured by valgrind.) > By default, valgrind doesn't model either branch predictors or branch-target address caches. Its model is somewhat primitive[0], but it is available through command-line option `--branch-sim` [1,2]. When you performance tested this change, did you enable that option? tool/run-speed-test.sh certainly doesn't. Of course, since those structures are shared globally, the performance benefit for de-virtualizing any given function call is highly context dependent. Sincerely, -- Jonathan Brandmeyer [0]: https://valgrind.org/docs/manual/cg-manual.html#branch-sim [1]: https://www.valgrind.org/docs/manual/cl-manual.html#cl-manual.options.simulation [2]: https://valgrind.org/docs/manual/cg-manual.html#cg-manual.cgopts ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
OK, I get that the definition of the SQLite API is a (large) set of C function calls. And that changing the way they work under the covers would break something. (Check out the IDEA below!) I'm wondering if your use case is - in the same application - that you dynamically change from option SQLITE_CONFIG_SINGLETHREAD to SQLITE_CONFIG_MULTITHREAD to SQLITE_CONFIG_SERIALIZED while your application is running? If that is the case, then your application knows which option to use dynamically; Otherwise, your application doesn't know and doesn't care which option is in effect. I am suggesting that if the we added the global calls to the underlying functions to the API - that is, the functions that are called by the function table indirection - then one could code the application to call the underlying functions. If the application knows it's single-thread, then code it that way and get a 25% improvement (see the talk). If the application makes the choice of thread option dynamically, then the penalty for single-thread is at least double (application choice, SQLite indirection), so calling the (new) underlying function once you made the choice, performs better for that path. I grant you that probably will see little improvement on the threaded path. If you are going to tell me that you need to maintain two versions of your application if you run it in a single-thread environment or a multi-thread environment, then let's define the (new) API to use a preprocessor macro to generate the right code for the option selected based on the values of SQLITE_CONFIG_SINGLETHREAD, el al. So now you have a single source but multiple executables corresponding to that source. And the choice of which executable to use becomes a configuration problem at application deployment time, or at application run time. IDEA! Thinking about it, I'm surprised that the C API isn't just a set of macros already. I can visualize a C API composed of a set of macro definitions _identical_ to the current C function calls. They just use the extra knowledge of SQLITE_THREADSAFE and other SQLite compiler options to decide what application code to generate. Then the formal API doesn't change from a coding point of view. The generated code calls a set of under-the-cover functions which are not part of the API. The change doesn't require a new layer of testing; presumably, we already have test cases that test the same code using different compiler options. What about that? Best, Doug > -Original Message- > From: sqlite-users > On Behalf Of Jens Alfke > Sent: Friday, December 27, 2019 2:22 PM > To: SQLite mailing list > Subject: Re: [sqlite] Causal profiling > > > > > On Dec 26, 2019, at 3:45 PM, Doug wrote: > > > > Can you explain what the API is that you use to "set concurrency > levels per connection", please? Is it a parameter on open() or its > own function call? And how would the API break, exactly? > > sqlite3_config(), specifically the SQLITE_CONFIG_SINGLETHREAD, > SQLITE_CONFIG_MULTITHREAD, SQLITE_CONFIG_SERIALIZED options. > > This API would break because configuring those options at runtime > would have no effect on behavior; the only thing that would change > threading behavior would be the compile-time flags > SQLITE_MUTEX_OMIT, etc. > > (This is actually global, not per-connection, but that doesn't > invalidate what I said.) > > > The talk suggested removing the SQLite virtual table of > functions (specifically the call to free a mutex). The user calls > the function directly. How does that break an API? > > If SQLite's implementation directly called the mutex lock/unlock > functions, instead of indirecting, then there would be no way to > control whether or not mutexes were used. In other words, it would > be impossible to change any of the above options at runtime. > > > The talk suggested removing the SQLite virtual table of > functions (specifically the call to free a mutex). The user calls > the function directly. How does that break an API? > > No, the user does not call those functions directly. The code > shown in the video is deep inside SQLite itself and not visible > through the API. (You say you're using a TCL wrapper … so you may > not be aware of what is or isn't in the C API. Trust me, I use the > C API a lot.) > > —Jens > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite- > users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
> On Dec 26, 2019, at 3:45 PM, Doug wrote: > > Can you explain what the API is that you use to "set concurrency levels per > connection", please? Is it a parameter on open() or its own function call? > And how would the API break, exactly? sqlite3_config(), specifically the SQLITE_CONFIG_SINGLETHREAD, SQLITE_CONFIG_MULTITHREAD, SQLITE_CONFIG_SERIALIZED options. This API would break because configuring those options at runtime would have no effect on behavior; the only thing that would change threading behavior would be the compile-time flags SQLITE_MUTEX_OMIT, etc. (This is actually global, not per-connection, but that doesn't invalidate what I said.) > The talk suggested removing the SQLite virtual table of functions > (specifically the call to free a mutex). The user calls the function > directly. How does that break an API? If SQLite's implementation directly called the mutex lock/unlock functions, instead of indirecting, then there would be no way to control whether or not mutexes were used. In other words, it would be impossible to change any of the above options at runtime. > The talk suggested removing the SQLite virtual table of functions > (specifically the call to free a mutex). The user calls the function > directly. How does that break an API? No, the user does not call those functions directly. The code shown in the video is deep inside SQLite itself and not visible through the API. (You say you're using a TCL wrapper … so you may not be aware of what is or isn't in the C API. Trust me, I use the C API a lot.) —Jens ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
On Fri, 27 Dec 2019 at 06:11, Jens Alfke wrote: > > > On Dec 25, 2019, at 2:53 PM, Doug wrote: > > > > I wrote an application in Qt which uses SQLite. Therefore, I invoke > SQLite functions with some wrapper. For a 9% performance improvement in > SQLite using the direct call versus indirect call (as discussed in the > talk), cannot the wrapper functions be changed so my application doesn't > know the difference? > > This change would break the API that lets you set concurrency levels per > connection; instead, the concurrency would be hardcoded at compile time. > _You_ may not be using this API, but there are definitely developers who do. > Note that API is already inherently unreliable though, as compiling with -DSQLITE_THREADSAFE=0 implies -DSQLITE_MUTEX_OMIT which eliminates the mutex calls entirely. Attempting to change the concurrency level at runtime via sqlite3_config() against such a binary will have no effect. (this is explained in the documentation for sqlite3_threadsafe()) -Rowan ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
> -Original Message- > From: sqlite-users > On Behalf Of Jens Alfke > Sent: Thursday, December 26, 2019 3:11 PM > To: SQLite mailing list > Subject: Re: [sqlite] Causal profiling > > > > > On Dec 25, 2019, at 2:53 PM, Doug wrote: > > > > I wrote an application in Qt which uses SQLite. Therefore, I > invoke SQLite functions with some wrapper. For a 9% performance > improvement in SQLite using the direct call versus indirect call > (as discussed in the talk), cannot the wrapper functions be > changed so my application doesn't know the difference? > > This change would break the API that lets you set concurrency > levels per connection; instead, the concurrency would be hardcoded > at compile time. _You_ may not be using this API, but there are > definitely developers who do. Can you explain what the API is that you use to "set concurrency levels per connection", please? Is it a parameter on open() or its own function call? And how would the API break, exactly? The talk suggested removing the SQLite virtual table of functions (specifically the call to free a mutex). The user calls the function directly. How does that break an API? > This would especially be a problem for environments where SQLite > is provided as a shared library in the operating system (e.g. iOS, > macOS). The concurrency level would then be up to the platform > vendor, not the application developer. (Unless they bundle their > own copy of SQLite in their app.) > > One possible workaround would be a compile option that enables the > direct calls, but which is off by default. People who wanted > mutexes but with direct calls could then set that option. > > [deletia] > > —Jens > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite- > users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
> On Dec 25, 2019, at 2:53 PM, Doug wrote: > > I wrote an application in Qt which uses SQLite. Therefore, I invoke SQLite > functions with some wrapper. For a 9% performance improvement in SQLite using > the direct call versus indirect call (as discussed in the talk), cannot the > wrapper functions be changed so my application doesn't know the difference? This change would break the API that lets you set concurrency levels per connection; instead, the concurrency would be hardcoded at compile time. _You_ may not be using this API, but there are definitely developers who do. This would especially be a problem for environments where SQLite is provided as a shared library in the operating system (e.g. iOS, macOS). The concurrency level would then be up to the platform vendor, not the application developer. (Unless they bundle their own copy of SQLite in their app.) One possible workaround would be a compile option that enables the direct calls, but which is off by default. People who wanted mutexes but with direct calls could then set that option. > On Dec 25, 2019, at 3:25 PM, Richard Hipp wrote: > > By "untestable" I mean that we would be unable to obtain 100% MC/DC > (essentually 100% branch test coverage) in an SQLite compiled as for > delivery. Because you'd only be testing one concurrency mode? But there are plenty of existing compile-time options in SQLite; don't you have to test multiple builds of the library to test those? How would this one be different? —Jens ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
On 12/25/19, Doug wrote: > Richard, can you please explain each of these? > > 1. API break > I wrote an application in Qt which uses SQLite. Therefore, I invoke SQLite > functions with some wrapper. For a 9% performance improvement in SQLite > using the direct call versus indirect call (as discussed in the talk), > cannot the wrapper functions be changed so my application doesn't know the > difference? You can completely disable all of the mutexes by compiling with SQLITE_THREADSAFE=0. That works fine, as long as you don't use any SQLite API in more than one thread at a time. And it does, definitely, make SQLite run faster. If you feel like you have to use threads, then you can run SQLite in multi-thread mode and most of the mutex calls will be omitted. Multi-thread mode allows multiple threads to use SQLite at the same time, as long as every thread is using a different database connection. If you run in serialized threading mode, then there will be many mutex calls. There is no way around that. Threading modes described here: https://www.sqlite.org/threadsafe.html > > 2. Render SQLITE untestable > Does that mean that you are doing whitebox testing? Surely, all the > thousands of queries vs responses are blackbox, not whitebox. Why would > changing indirect calls to direct calls render SQLite untestable? By "untestable" I mean that we would be unable to obtain 100% MC/DC (essentually 100% branch test coverage) in an SQLite compiled as for delivery. We go by the philosophy that "If it isn't tested, then it doesn't work" and so if there are branches that are unreachable by our tests, then SQLite is "untestable". We also go by "fly what you test and test what you fly", so adding a compile-time option that allows mutex calls to be intercepted and redirected in testing builds but not in release builds won't work for us. > > 3. Unable to replicate performance gains > This says to me you actually made the change suggested. And then you ran a > test suite against the amalgamation. And you actually measured the result. > How can you have done that if such a change renders SQLite untestable? And > (sneaking a peak at the talk again re performance measurements), what did > you use to measure the results? > The video provided details on what they did. I could not find any performance improvement by making mutexes direct calls instead of indirect calls. Maybe I did something wrong. Maybe it depends on your compiler and or optimization options and I didn't use the right combination. Maybe they are measuring performance differently than me. (I use CPU cycle counts measured by valgrind.) Maybe the team that studied this made a mistake in their testing and using indirect calls to mutexes really doesn't matter that much after all. Maybe the problem is some combination of all of the above. I don't know. If you want to try to replicate the performance improvement yourself, and report your detailed findings on this mailing list, you are welcomed to do so. If you have a reproducible test case - perhaps we will go in and provide a compile-time option that makes SQLite run faster in intensely multi-threaded applications at the expense of also rendering it untestable -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
Richard, can you please explain each of these? 1. API break I wrote an application in Qt which uses SQLite. Therefore, I invoke SQLite functions with some wrapper. For a 9% performance improvement in SQLite using the direct call versus indirect call (as discussed in the talk), cannot the wrapper functions be changed so my application doesn't know the difference? 2. Render SQLITE untestable Does that mean that you are doing whitebox testing? Surely, all the thousands of queries vs responses are blackbox, not whitebox. Why would changing indirect calls to direct calls render SQLite untestable? 3. Unable to replicate performance gains This says to me you actually made the change suggested. And then you ran a test suite against the amalgamation. And you actually measured the result. How can you have done that if such a change renders SQLite untestable? And (sneaking a peak at the talk again re performance measurements), what did you use to measure the results? Best, Doug > -Original Message- > From: sqlite-users > On Behalf Of Richard Hipp > Sent: Wednesday, December 25, 2019 3:18 PM > To: SQLite mailing list > Subject: Re: [sqlite] Causal profiling > > On 12/25/19, sky5w...@gmail.com wrote: > > Thanks for sharing! > > Did his suggested optimization make it to a commit? > > No. That would be an API break, and would also render SQLite > untestable. Furthermore, we have been unable to replicate the > performance gains. > -- > D. Richard Hipp > d...@sqlite.org > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite- > users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
On 12/25/19, sky5w...@gmail.com wrote: > Thanks for sharing! > Did his suggested optimization make it to a commit? No. That would be an API break, and would also render SQLite untestable. Furthermore, we have been unable to replicate the performance gains. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Causal profiling
Thanks for sharing! Did his suggested optimization make it to a commit? On Wed, Dec 25, 2019 at 10:46 AM Craig H Maynard wrote: > All, > > Just watched an interesting lecture by UMass professor Emery Berger on > improving software performance: > > https://www.youtube.com/watch?v=r-TLSBdHe1A > > SQLite is discussed in the section on causal profiling, which begins at > 34:12. > > Craig > > -- > Craig H Maynard > Rhode Island, USA > 401.413.2376 > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Causal profiling
All, Just watched an interesting lecture by UMass professor Emery Berger on improving software performance: https://www.youtube.com/watch?v=r-TLSBdHe1A SQLite is discussed in the section on causal profiling, which begins at 34:12. Craig -- Craig H Maynard Rhode Island, USA 401.413.2376 ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
