Re: [sqlite] Size of the SQLite library
On 7 Jun 2018, at 5:25am, Dianne Dunn wrote: > Hey there do you know how I can get off this list.?? Click the link that appears at the bottom of every post. Simon. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
Have you tried the link at the end of every message? --- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume. >-Original Message- >From: sqlite-users [mailto:sqlite-users- >boun...@mailinglists.sqlite.org] On Behalf Of Dianne Dunn >Sent: Wednesday, 6 June, 2018 22:25 >To: SQLite mailing list >Subject: Re: [sqlite] Size of the SQLite library > >Hey there do you know how I can get off this list.?? > >Sent from my iPad > >> On Jun 5, 2018, at 3:50 AM, Robert M. Münch > wrote: >> >>> On 31 May 2018, at 19:15, Richard Hipp wrote: >>> >>> But more recently, mobile phone designers are telling me things >like >>> "try to keep the size under 5 megabytes, if you can, please." >>> >>> Based on those more recent conversations, I'm thinking that we >have >>> more headroom that we have had historically, and so I have >recently >>> been allowing new features to start creeping into the core. >> >> Size matters IMO, it’s a sign of good design and less is more WRT >errors etc. >> >> >>> Size is still important. But having useful features is important >too. >> >> True, and we all know that most features are not used. Do you have >an idea what features are used by ratio? Maybe adding a „report back >feature collector“ might be an idea, for those wanting to support the >feature selection process. >> >> >>> I'm continuing to work to find the right balance between these >>> competing goals. >> >> Keeping things configurable as it is, is a very good approach, >please keep it. >> >> -- >> >> Robert M. Münch, CEO >> M: +41 79 65 11 49 6 >> >> Saphirion AG >> smarter | better | faster >> >> http://www.saphirion.com >> http://www.nlpp.ch >> ___ >> sqlite-users mailing list >> sqlite-users@mailinglists.sqlite.org >> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite- >users > >___ >sqlite-users mailing list >sqlite-users@mailinglists.sqlite.org >http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
Hey there do you know how I can get off this list.?? Sent from my iPad > On Jun 5, 2018, at 3:50 AM, Robert M. Münch > wrote: > >> On 31 May 2018, at 19:15, Richard Hipp wrote: >> >> But more recently, mobile phone designers are telling me things like >> "try to keep the size under 5 megabytes, if you can, please." >> >> Based on those more recent conversations, I'm thinking that we have >> more headroom that we have had historically, and so I have recently >> been allowing new features to start creeping into the core. > > Size matters IMO, it’s a sign of good design and less is more WRT errors etc. > > >> Size is still important. But having useful features is important too. > > True, and we all know that most features are not used. Do you have an idea > what features are used by ratio? Maybe adding a „report back feature > collector“ might be an idea, for those wanting to support the feature > selection process. > > >> I'm continuing to work to find the right balance between these >> competing goals. > > Keeping things configurable as it is, is a very good approach, please keep it. > > -- > > Robert M. Münch, CEO > M: +41 79 65 11 49 6 > > Saphirion AG > smarter | better | faster > > http://www.saphirion.com > http://www.nlpp.ch > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On 06/06/18 09:24, Bob Friesenhahn wrote: > A local tool which makes it easy to configure sqlite from local files > sounds useful ... It already exists. It is what the SQLite team uses to produce the amalgamations etc, and is part of the SQLite code base. > but depending on a "web site" (baby-bird model) ... Note that behind the scenes the existing tools would be used with the relevant results zipped up and downloadable. No one is advocating getting rid of the command line tools, just a web front end. > There is already far too much dependence on what what > happens to get served up at the time and too much dependence on a live > connection to the "Internet" ... You and Warren comprehensively describe best practises and why. You are both right. It is what developers *should* do for repeatable reliable builds. But it is a lot of friction. And not every developer follows best practise. And developers start out investigating and playing around with potential solutions, and then adopt the appropriate ones. If you are trying out a "hello world" quick test, then the best practises are a lot of friction, and a few web page tickboxes are the least. The more friction there is, the fewer people will try non-default configurations. But that also locks SQLite into a pessimistic legacy configuration going forward. For example default enabling STAT4 or disabling deprecated API could not be done, ever. Roger signature.asc Description: OpenPGP digital signature ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
I recommend a mixture of the following two solutions: On 6 Jun 2018, at 5:05pm, Roger Binns wrote: > That is why I advocate a web site where the user (un)ticks what they > want, and the web site provides a correctly configured download. 6 Jun 2018, at 5:24pm, Bob Friesenhahn wrote: > A local tool which makes it easy to configure sqlite from local files sounds > useful but depending on a "web site" (baby-bird model) does not sound good to > me. There is already far too much dependence on what what happens to get > served up at the time and too much dependence on a live connection to the > "Internet" with a naive expectation what what was produced yesterday will > continue to be produced tomorrow. To mix them, you put one configuration on the web. The current one (call it "most useful for most people") is fine. This configuration can be changed by editing one file. For a C project this would be a "sqlite3config.h" file with lots of "#define" lines. This file has nothing in except for configuration settings and some (but not long and exhaustive) comments on what they do. No macros or function definitions. Experts can read the documentation in the file and set the definitions themselves. But you also put up an online configuration web page, which works using JavaScript. The web page has GUI features up top: popup menus, checkboxes, radio-buttons, whatever. At the bottom of the page is a text field containing the entire contents for an ".h" file which configures the compilation according to those settings. You can change the GUI settings and see how that changes the file in real time. It's up to the user to copy that text and past it into a new "sqlite3config.h" file or whatever it is. Here's the good part: because the web page works using just JavaScript (rather than PHP, ASP, node, whatever) you can include a copy of it with the distribution and any user can run it in their favourite browser without needing internet access, or perhaps on a smartphone. Simon. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On Wednesday, 6 June, 2018 10:24, Bob Friesenhahn wrote: > The build strategy for the Python APSW extension is an > example of unwanted dependency and loss of control. > Building of software from source code should always be > under the complete control of the person who is performing > the build and should inherently support use of local files > which may contain local changes. I build APSW this way and it uses a completely customized version of the sqlite3.c amalgamation that I also build into its own sqlite3 executables and DLLs, it is really not that difficult. Mind you, I extract the latest APSW sources from the ZIP archive and have built my own build-scripts to do this. Because I use the mingw-w64 compiler toolchain I also have to slightly modify the default Python library cygwincompiler.py and have made my own customized APSW setup.py with a different name (based on the distributed source setup.py) that automates the whole process. Fossil is used to manage the local repositories that are created from the distribution source (I have another Linux machine that builds the amalgamation source from the full sources and I copy that to use as my base amalgamation for generating the executables, DLLs, and APSW). --- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On Wed, 6 Jun 2018, Roger Binns wrote: That is why I advocate a web site where the user (un)ticks what they want, and the web site provides a correctly configured download. This will also tell the SQLite developers what features are configured. (eg if everyone turns off virtual tables that is useful feedback, as would the opposite.) A local tool which makes it easy to configure sqlite from local files sounds useful but depending on a "web site" (baby-bird model) does not sound good to me. There is already far too much dependence on what what happens to get served up at the time and too much dependence on a live connection to the "Internet" with a naive expectation what what was produced yesterday will continue to be produced tomorrow. The build strategy for the Python APSW extension is an example of unwanted dependency and loss of control. Building of software from source code should always be under the complete control of the person who is performing the build and should inherently support use of local files which may contain local changes. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On 05/06/18 15:07, Warren Young wrote:
> All right, so include [multi-component source control and build process] ...
I'm still not sure what point you are trying to make. Yes *you* can do
that. Should *every* SQLite user who wants non-default options *have*
to go through a similar amount of friction?
SQLite currently only has one distribution. This distribution has to
fit most user needs regarding backwards and forwards compatibility
(including query plans), functionality, size etc.
*If* SQLite wants to step away from one size/configuration fits most,
then there needs to be way less friction in getting the alternate
configurations. One solution is a small number of alternate downloads
("presets"), although it is hard to know what configurations they should
have.
That is why I advocate a web site where the user (un)ticks what they
want, and the web site provides a correctly configured download. This
will also tell the SQLite developers what features are configured. (eg
if everyone turns off virtual tables that is useful feedback, as would
the opposite.)
> Thus the need for curated collections of build options, since a jQuery UI
> like tool that assumes the options are all orthogonal would frequently
> produce unbuildable output.
Huh? No one is advocating a SQLite web tool that produces unbuildable
output, or offers every possible combination of options. It would need
to be useful, and can start simple.
Roger
signature.asc
Description: OpenPGP digital signature
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On Jun 5, 2018, at 2:24 PM, Roger Binns wrote: > > For example to exclude virtual tables from SQLite, you can't just add a > compile time option and be done. You have to regenerate from the > grammar (so it is no longer valid SQL syntax and no longer has calls to > virtual table relevant functions). And then you almost certainly want > to use the tool to make the amalgamation from the updated grammar. And > then you need to make sure your Makefile or equivalent passes in the > omit flag too. All right, so include a clone of the SQLite Fossil repository in your application tree instead of the amalgamation and write a script to do all of this as part of the build process. That is, make the existence of sqlite3.c dependent on a successful run of this script. You can find many open source projects built this way, most often seen when the top-level configure script calls a dependency’s configure script, then the top-level Makefile calls the dependency’s Makefile at a time determined by the top-level’s declared deps. >> Contrast a language like JavaScript, where you can ship a program that has >> calls to functions that don’t exist, and as long as you continue to not call >> those functions, the JS VM won’t balk. > > You can do lazy runtime linking in some operating systems …leaving out platforms where it doesn’t work. I believe macOS and iOS are that way, for instance. Of the platforms I’ve used, the only one I’m sure allows it is Linux, and I believe I’m remembering that because of all the times it’s masked problems that showed up when I went to port some bit of software to another platform. > But in any event JS code is not distributed how you think. Ah, so? What I actually think is that one of my primary development products is based on jQuery UI, which in turn leads me to think I know how JS code is distributed in general, and how jQuery UI is distributed specifically. :) One of the options you have when using jQuery UI’s custom distributions is to include the necessary modules in your project’s tree unminified and unmerged so that you can do your own local merging and minification. Which I do. If you then mistakenly leave out a needed module, you get mysterious run-time failures. (Ask me how I know.) “Do I need focusable.js? Hmmm, I guess I’d better just try all of the UI functions and see if I get complaints in the JS error console, because the nonexistent static linker sure isn’t going to tell me!” I’ve even seen such run-time failures happen with calls that are *internal* to the merged and minified artifact because some bit of dynamic JS was being sufficiently clever that the tools couldn’t see that there was an unresolved call. >> 2. There are ways around this with C, > > My point is that it isn't. You cannot add / remove defines to the > amalgamation to omit most features. Heck if you try it just won't > compile. More work has to be done. The mailing list archives have many > messages where people tried a few compile flags and it didn't work. Yes, I’ve run into such situations several times myself. Every time it’s happened to me, I just drop the now-problematic OMIT option(s) and move on. The most recent such regression (?) for me was finding that shell.c wouldn’t build to a usable SQLite CLI binary with -DSQLITE_OMIT_AUTOINIT because it is now depending on autoinit. Thus the need for curated collections of build options, since a jQuery UI like tool that assumes the options are all orthogonal would frequently produce unbuildable output. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On 01/06/18 13:46, Warren Young wrote: > Your jQuery example later on doesn’t much apply here, for several reasons: Note that I was showing how the site let you choose whatever features you want, and then gave you a download matching exactly that. > 1. JavaScript is a dynamic language, while C is a statically-compiled > language. Your comments while correct don't actually apply to SQLite. SQLite is not a C file. It is many C source files, many headers, a grammar (not in C), and various tools (typically TCL). For example to exclude virtual tables from SQLite, you can't just add a compile time option and be done. You have to regenerate from the grammar (so it is no longer valid SQL syntax and no longer has calls to virtual table relevant functions). And then you almost certainly want to use the tool to make the amalgamation from the updated grammar. And then you need to make sure your Makefile or equivalent passes in the omit flag too. The web site doing all that work for you, and getting it right every time does have value IMHO. It also makes it easier for SQLite to have bigger or smaller presets to address the varying developer needs. And the team will have some idea of what OMITs are used, where testing should check etc. > That means that all of the symbols needed to link the program ... That was nothing to do with the issue. To be very clear: * SQLite has a way of omiting functionality * Other than a few special cases, you must use the SQLite source (not the amalgamation) to regenerate what you finally use * Doing this is difficult and error prone > Contrast a language like JavaScript, where you can ship a program that has > calls to functions that don’t exist, and as long as you continue to not call > those functions, the JS VM won’t balk. You can do lazy runtime linking in some operating systems so functions to calls that don't exist are ok (until you call them). But in any event JS code is not distributed how you think. Minified source is usually used, and works best if run through dead code elimination first (called "tree shaking" in the JS world). ie the distribution isn't that different to SQLite (amalgamation). > 2. There are ways around this with C, My point is that it isn't. You cannot add / remove defines to the amalgamation to omit most features. Heck if you try it just won't compile. More work has to be done. The mailing list archives have many messages where people tried a few compile flags and it didn't work. > One could write a variant of cpp that would run on the sqlite3.c amalgamation > ... It won't work for anything grammar related. And the project has tools like you describe (eg it is how the amalgamation is produced). Those tools are in TCL and know about the structure and coding patterns of SQLite. Roger signature.asc Description: OpenPGP digital signature ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On 31 May 2018, at 19:15, Richard Hipp wrote: > But more recently, mobile phone designers are telling me things like > "try to keep the size under 5 megabytes, if you can, please." > > Based on those more recent conversations, I'm thinking that we have > more headroom that we have had historically, and so I have recently > been allowing new features to start creeping into the core. Size matters IMO, it’s a sign of good design and less is more WRT errors etc. > Size is still important. But having useful features is important too. True, and we all know that most features are not used. Do you have an idea what features are used by ratio? Maybe adding a „report back feature collector“ might be an idea, for those wanting to support the feature selection process. > I'm continuing to work to find the right balance between these > competing goals. Keeping things configurable as it is, is a very good approach, please keep it. -- Robert M. Münch, CEO M: +41 79 65 11 49 6 Saphirion AG smarter | better | faster http://www.saphirion.com http://www.nlpp.ch signature.asc Description: OpenPGP digital signature ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
"You are soo, bloated," said Java. On Thu, May 31, 2018, 11:58 R Smith wrote: > > On 2018/05/31 5:17 PM, ven...@intouchmi.com wrote: > > I have to agree with Bob! > > > > We have considered SQLITE for our project. Going over 500Kbytes puts it > > just beyond the size of our Flash - the current Firmware. > > I stand corrected! It seems the embedded systems with still an extremely > limited memory footprint size may not be as thin on the ground as I > imagined, and I regret trying to categorize all embedded systems under > the same ideal - apologies for that. > > Towards my point though, both Bob and Vance, would you be especially > swayed if the marketing slogan had said "under a megabyte" as opposed to > "under half a megabyte"? I still feel that this level of embedded > system is not common, and even where it might be common, I bet that > slogan is not the catch phrase that got you interested in SQLite (or > would sway you from choosing it). > > It's however clear my view may not be 100% representative, so perhaps > the KiB or 0.5 MiB route has its place. > > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > -- Cheers, Chris ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On May 31, 2018, at 6:32 PM, Roger Binns wrote: > > On 31/05/18 10:15, Richard Hipp wrote: >> Size is still important. But having useful features is important too. >> I'm continuing to work to find the right balance between these >> competing goals. > > A pattern used in other projects is to have standard downloads, as well > as custom ones. Your jQuery example later on doesn’t much apply here, for several reasons: 1. JavaScript is a dynamic language, while C is a statically-compiled language. That means that all of the symbols needed to link the program need to be available at link time in order to produce a binary. To achieve that with C, you’d have to do things like create mock modules that export an API but don’t implement it, merely to placate the linker. Contrast a language like JavaScript, where you can ship a program that has calls to functions that don’t exist, and as long as you continue to not call those functions, the JS VM won’t balk. 2. There are ways around this with C, such as with the plugin pattern — which is in fact already being used in SQLite’s VFS layer — but it carries an indirection overhead. jQuery is a bad exemplar here because it’s a solution for implementing user-facing actions, which means that as long as each group of actions implemented using it take under 100 ms or so, it’s fast enough. For SQLite, though, adding layers of abstraction merely for the programmer’s convenience means slowing it down materially, which can turn a viable solution into failure. Some sage once opined that any problem in computer science can be solved by adding another layer of abstraction, but there is one that can’t: “The software is too slow, and we’re unwilling to buy more hardware.” 3. SQLite already has a way to generate multiple versions of the source code in a programmatic way: #ifdefs. That’s precisely what the C preprocessor does. JavaScript has no equivalent mechanism, so someone had to go an factor jQuery into modules by hand and rely on the user to provide the correct subset of modules needed by their software. One could write a variant of cpp that would run on the sqlite3.c amalgamation to produce predigested subsets without bringing in all of the #includes, but all that’s really needed here are curated sets of -DSQLITE_* flags, since then the end user can use them with the C preprocessor they already have. I’ve a feeling that I’m missing more reasons, but those will suffice. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
1. Define in documentation as < 1Mb. (Don't have to visit again.) 2. Continue to strive to keep in the 0.5-1MB range. 3. Add some information on building a MINIMUM size for those concerned that is relatively easy to accomplish without a lot of expertise if possible. danap. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On 31/05/18 10:15, Richard Hipp wrote: > Size is still important. But having useful features is important too. > I'm continuing to work to find the right balance between these > competing goals. A pattern used in other projects is to have standard downloads, as well as custom ones. With the latter you can include or exclude additional components and features. You can already do this with SQLite, but it requires several more command line tools, programming languages, and comprehensive reading of the documentation. Perhaps a custom download web page that gives you some presets (smallest, default, everything) or lets you choose your own settings. It would then produce known good source files, and users would be happy. Here is an example page for a Javascript project: https://jqueryui.com/download/ On the balance side, STAT4 is a good example. I think it would benefit the majority of SQLite users if it was enabled by default. But making only that change could change query plans for existing users. (Many users also don't compile SQLite itself - they get binaries from the platform, or language bindings.) Roger signature.asc Description: OpenPGP digital signature ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
>On Thursday, 31 May, 2018 10:19, Dominique Devienne said: >Given where the conversation is going, let me point out that many do >not care one bit about the lib's size :) >I'd much rather have an SQLite with tons of features, than forego >those in the name saving a few bytes, to save a few bucks on the >embedded chip and flash for commercial products that don't even >pay for SQLite. >SQLite is already amazingly small for the value it brings. And if >people want "smaller", they can still stick >with older leaner versions of SQLite too. My $0.02... --DD The custom version of the library that I build which contains *all* features and extensions built-in, and then some, compiled with MinGW-w64 GCC 7.1.0 on Windows using -m64 -O3, and statically linked with all the GCC libraries (__float128, runtime, threading, etc., so no dependancies other than to the subsystem runtime (MSVCRT) and the Windows DLLs actually used) comes in around 2 MB ... APSW a wee bit bigger ... 2018-05-30 16:40 2,173,456 sqlite3.dll 2018-05-30 16:43 2,326,544 apsw.cp36-win_amd64.pyd Of course, this includes almost all the extensions, all the math library, many Windows APIs, and a bunch of aggregates/functions/collations to handle IP Addresses (v4 and v6), some running statistics, proper rounding (Half-Even), all the hash functions (MD4/MD5/SHA/SHA1/SHA2(256/384/512)/SHA3(224/256/384/512)) and a few other odds and sods. If it were for a computer with "limited resources" I would par it down a lot, but having everything automatically available is very nice ... and I am not CPU/Memory/IO constained (though having all NVMe drives does make me have to "fix" things from time to time to stay within the constraints of spinning rust (3 GB/s I/O can become very addictive). --- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
I totally agree with that. On most systems it is much more important to have a feature-rich library than a very small one. Any application where a few bytes more or less matter should be written in pure assembler anyway. - Original Message - From: Dominique Devienne To: General Discussion of SQLite Database Sent: Thursday, May 31, 2018, 18:18:51 Subject: [sqlite] Size of the SQLite library On Thu, May 31, 2018 at 3:44 PM Richard Hipp wrote: > For many years, we have boasted that the size of the SQLite library is > "less than half a megabyte". Given where the conversation is going, let me point out that many do not care one bit about the lib's size :) I'd much rather have an SQLite with tons of features, than forego those in the name saving a few bytes, to save a few bucks on the embedded chip and flash for commercial products that don't even pay for SQLite. SQLite is already amazingly small for the value it brings. And if people want "smaller", they can still stick with older leaner versions of SQLite too. My $0.02... --DD ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On 5/31/18, Simon Slavin wrote: > > Did you know that less than half of SQLite installations are on desktop > computers ? My guess is that mobile phones are now the biggest category of > devices. They run off battery power. They have firmware on chips. The > more chips they have to keep powered-up, the more battery power they use, > the physically bigger the phone has to be to hold not just the chips but the > bigger battery, the heavier and more expensive it is. > Back in the day, Motorola and Symbian used to really pressure us to keep the library footprint down. Every byte mattered. But more recently, mobile phone designers are telling me things like "try to keep the size under 5 megabytes, if you can, please." Based on those more recent conversations, I'm thinking that we have more headroom that we have had historically, and so I have recently been allowing new features to start creeping into the core. Size is still important. But having useful features is important too. I'm continuing to work to find the right balance between these competing goals. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On 31 May 2018, at 5:18pm, Dominique Devienne wrote: > On Thu, May 31, 2018 at 3:44 PM Richard Hipp wrote: > >> For many years, we have boasted that the size of the SQLite library is >> "less than half a megabyte". > > Given where the conversation is going, let me point out that many do not > care one bit about the lib's size :) Did you know that less than half of SQLite installations are on desktop computers ? My guess is that mobile phones are now the biggest category of devices. They run off battery power. They have firmware on chips. The more chips they have to keep powered-up, the more battery power they use, the physically bigger the phone has to be to hold not just the chips but the bigger battery, the heavier and more expensive it is. Not to mention a whole fleet of handheld safety-testing equipment with different configurations for different installations, and those gadgets parking inspectors carry around to keep track of which cars parked when. Might be interesting to find out what proportion of SQLite devices are mains-powered vs. battery-powered. Although whether one should class an Airbus A350 XWB as "battery-powered" I am not certain. Simon. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On May 31, 2018 12:18:51 PM EDT, Dominique Devienne wrote: >On Thu, May 31, 2018 at 3:44 PM Richard Hipp wrote: > >> For many years, we have boasted that the size of the SQLite library >is >> "less than half a megabyte". >> > >Given where the conversation is going, let me point out that many do >not >care one bit about the lib's size :) > >I'd much rather have an SQLite with tons of features, than forego those >in >the name saving a few bytes, >to save a few bucks on the embedded chip and flash for commercial >products >that don't even pay for SQLite. > >SQLite is already amazingly small for the value it brings. And if >people >want "smaller", they can still stick >with older leaner versions of SQLite too. My $0.02... --DD >___ >sqlite-users mailing list >sqlite-users@mailinglists.sqlite.org >http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users I agree with this sentiment. I mostly use SQLite in PHP, where it is awkward to customize SQLite, and all but impossible to rely on features not included in the standard build when distributing to others. A more powerful default configuration would be very beneficial, and a less powerful one possibly crippling. -- J. King ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On Thu, May 31, 2018 at 3:44 PM Richard Hipp wrote: > For many years, we have boasted that the size of the SQLite library is > "less than half a megabyte". > Given where the conversation is going, let me point out that many do not care one bit about the lib's size :) I'd much rather have an SQLite with tons of features, than forego those in the name saving a few bytes, to save a few bucks on the embedded chip and flash for commercial products that don't even pay for SQLite. SQLite is already amazingly small for the value it brings. And if people want "smaller", they can still stick with older leaner versions of SQLite too. My $0.02... --DD ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On Thu, May 31, 2018 at 11:38 AM, Richard Hipp wrote: > [...] > By using multiple SQLITE_OMIT compile-time options to leave out > features, I can get the size down to 308,189 bytes using gcc-7 -Os > -m32. @Richard can you elaborate some more on how you make this kind of a build? I wouldn't mind if we drop some more less-used options from the default build to keep the standard size "less than half a megabyte". Also -1 for kibibyte/mebibyte wording on my part. On Thu, May 31, 2018 at 11:57 AM, Christian Schmitz wrote: > [...] > Maybe your graph should have three lines. +1 would be nice, not major though (I think) On Thu, May 31, 2018 at 11:58 AM, R Smith wrote: > [...] > Towards my point though, both Bob and Vance, would you be especially swayed > if the marketing slogan had said "under a megabyte" as opposed to "under > half a megabyte"? I think Vance already gave the answer (negative). Would it be an idea to have size slogans for both regular and embedded builds? ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On 2018/05/31 5:17 PM, ven...@intouchmi.com wrote: I have to agree with Bob! We have considered SQLITE for our project. Going over 500Kbytes puts it just beyond the size of our Flash - the current Firmware. I stand corrected! It seems the embedded systems with still an extremely limited memory footprint size may not be as thin on the ground as I imagined, and I regret trying to categorize all embedded systems under the same ideal - apologies for that. Towards my point though, both Bob and Vance, would you be especially swayed if the marketing slogan had said "under a megabyte" as opposed to "under half a megabyte"? I still feel that this level of embedded system is not common, and even where it might be common, I bet that slogan is not the catch phrase that got you interested in SQLite (or would sway you from choosing it). It's however clear my view may not be 100% representative, so perhaps the KiB or 0.5 MiB route has its place. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
> > See https://sqlite.org/tmp/size-20180531.jpg for the library size > trend over 5 years. Maybe your graph should have three lines. 1. Minimum SQLite with all off 2. Default SQLite 3. Maximum SQLite with all on Sincerely Christian -- Read our blog about news on our plugins: http://www.mbsplugins.de/ ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On 5/31/18, ven...@intouchmi.com wrote: > > We have considered SQLITE for our project. Going over 500Kbytes puts it > just beyond the size of our Flash - the current Firmware. By using multiple SQLITE_OMIT compile-time options to leave out features, I can get the size down to 308,189 bytes using gcc-7 -Os -m32. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
I have to agree with Bob! We have considered SQLITE for our project. Going over 500Kbytes puts it just beyond the size of our Flash - the current Firmware. Vance On 2018-05-31 11:04, Bob Friesenhahn wrote: > On Thu, 31 May 2018, R Smith wrote: > >> Nice idea, but to be honest, I can't remember when last someone cared about >> "Kilobytes", and I mean embedded people, not big OSes. > > I work on embedded projects and we do definitely worry about "kilobytes". > This is even though our embedded projects have large resources compared with > many other embedded projects. The firmware image for some of our products is > consuming all available Flash pages, (except for spares for > wear-leveling/repair). > > Many embedded projects are very cost-sensitive since they sell into > hyper-competitive markets where being a bit more expensive than the > competition results in a lack of sales. > >> The measure of importance is how expensive the DATA storing is, both in size >> and write-frequency, when committed to some hardware NANDs. The code store >> section of even the smallest modern embedded system will be designed to fit >> things many megabytes more than SQLite requires (exceptions may exist, but >> are really thin on the ground). So then, whether the operating code is given >> in KB or MiB or KiB is, to my mind, not very relevant - and it too will >> become untrue in a non-too-distant future. > > Your experience is different than mine. What NOR or NAND Flash chip are you > using on your PCB? If you are not using a single soldered chip with a > specialized flash filesystem (e.g. JFFS2, UBIFS, squashfs on UBI or bare) > then perhaps you are just using a small form factor PC which uses components > common in laptop PCs. > > Bob ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On Thu, 31 May 2018, R Smith wrote: Nice idea, but to be honest, I can't remember when last someone cared about "Kilobytes", and I mean embedded people, not big OSes. I work on embedded projects and we do definitely worry about "kilobytes". This is even though our embedded projects have large resources compared with many other embedded projects. The firmware image for some of our products is consuming all available Flash pages, (except for spares for wear-leveling/repair). Many embedded projects are very cost-sensitive since they sell into hyper-competitive markets where being a bit more expensive than the competition results in a lack of sales. The measure of importance is how expensive the DATA storing is, both in size and write-frequency, when committed to some hardware NANDs. The code store section of even the smallest modern embedded system will be designed to fit things many megabytes more than SQLite requires (exceptions may exist, but are really thin on the ground). So then, whether the operating code is given in KB or MiB or KiB is, to my mind, not very relevant - and it too will become untrue in a non-too-distant future. Your experience is different than mine. What NOR or NAND Flash chip are you using on your PCB? If you are not using a single soldered chip with a specialized flash filesystem (e.g. JFFS2, UBIFS, squashfs on UBI or bare) then perhaps you are just using a small form factor PC which uses components common in laptop PCs. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Size of the SQLite library
On 2018/05/31 3:44 PM, Richard Hipp wrote: For many years, we have boasted that the size of the SQLite library is "less than half a megabyte". That will likely still be true in the 3.24.0 release, though just barely. Compiling with gcc 5.4.0 and -Os on ubuntu, I get 499,709 bytes. With gcc 7.1.0 and -Os I get 496,399 bytes. The library is, of course, larger if you enable optional features such as full-text search and/or use compiler optimizations like -O3 which enable loop unrolling and function inlining. And most people do compile it that way. So "less than a megabyte" might be a more accurate description of SQLite in practice. But the default configuration compiled with -Os is a good metric for comparison. See https://sqlite.org/tmp/size-20180531.jpg for the library size trend over 5 years. The measurements in the graph were done with gcc 5.4.0 and -Os on ubuntu. As you can see, we have held the line below 500,000 bytes for a long time. But the recent addition of new features (ex: UPSERT) has caused a slight uptick in the library size. As further new features are in the pipeline, the upcoming 3.24.0 release will probably be the last for which the library size comes in at less than 500,000 bytes. For this reason, I will probably change the size bullet point to say "less than 500 kibibytes (KiB)" or "less than 0.5 mebibytes (MiB)", as "less than 600KB" does not have quite the same emotional impact. You will notice that the graph linked above is calibrated in mebibytes. Nice idea, but to be honest, I can't remember when last someone cared about "Kilobytes", and I mean embedded people, not big OSes. The measure of importance is how expensive the DATA storing is, both in size and write-frequency, when committed to some hardware NANDs. The code store section of even the smallest modern embedded system will be designed to fit things many megabytes more than SQLite requires (exceptions may exist, but are really thin on the ground). So then, whether the operating code is given in KB or MiB or KiB is, to my mind, not very relevant - and it too will become untrue in a non-too-distant future. May I propose, if changing is on the table, to rather update to a more current universal notion (and modern embedded capacities) and make it leaner by removing one word and simply call it: "less than a megabyte" This has much the same emotional impact, still is downright amazing, doesn't require naming shenanigans, is very TRUE, even with a few funny switches compiled-in, AND will remain true for many years to come, possibly to the end of the SQLite lifecycle ~30 years hence. My 2c... Ryan ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
