Re: [sqlite] Setting auto_vacuum=2 doesn't work after setting journal_mode=WAL

2020-02-21 Thread Keith Medcalf
On Friday, 21 February, 2020 19:36, Simon Slavin wrote: >On 22 Feb 2020, at 2:28am, Keith Medcalf wrote: >> When a database is to be created these commands must be given BEFORE >any command which opens or creates the database: >> >> pragma auto_vacuum >> pragma encoding >> pragma page_size >>

Re: [sqlite] Setting auto_vacuum=2 doesn't work after setting journal_mode=WAL

2020-02-21 Thread Simon Slavin
On 22 Feb 2020, at 2:28am, Keith Medcalf wrote: > When a database is to be created these commands must be given BEFORE any > command which opens or creates the database: > > pragma auto_vacuum > pragma encoding > pragma page_size > pragma data_store_directory > > The issuance (or preparation)

Re: [sqlite] Setting auto_vacuum=2 doesn't work after setting journal_mode=WAL

2020-02-21 Thread Keith Medcalf
There are three pragma's which affect the "organization" of a newly created database. When a database is to be created these commands must be given BEFORE any command which opens or creates the database: pragma auto_vacuum pragma encoding pragma page_size pragma data_store_directory The

[sqlite] Setting auto_vacuum=2 doesn't work after setting journal_mode=WAL

2020-02-21 Thread Jens Alfke
I just discovered that none of our databases are auto-vacuuming, even though we set auto_vacuum to 'incremental' when a database is created, and periodically call 'pragma incremental_vacuum'. If I go into the CLI, open a database and run "pragma auto_vacuum", it returns 0. After some

[sqlite] Record serialization infrastructure of SQLITE

2020-02-21 Thread Deon Brewis
Inspired by the json encoding thread. We have a need for multi-dimensional data (like json), and currently use a record format for that that is derived from the SQLITE format with Serial Type/Size stored in varint Huffman encoding of twos-complement. It is a great fast, compact storage format.

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Martin Raiber
On 21.02.2020 15:03 Richard Hipp wrote: > On 2/21/20, Wout Mertens wrote: >> The idea is that upon storing the JSON >> data, the JSON1 extension parses it, extracts the layouts recursively, >> stores them when they are not known yet, and then only stores the >> values in the binary format with

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Jens Alfke
> On Feb 21, 2020, at 4:20 AM, Wout Mertens wrote: > > I was wondering if the JSON extension could not do the same thing: for each > table, keep a hidden stash of object layouts, and store the values as > sqlite primitives. (you'd be able to disable this, in case the layouts > rarely repeat)

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Jens Alfke
> On Feb 21, 2020, at 4:20 AM, Wout Mertens wrote: > > In JavaScript, objects are key-value collections with unique keys, where the > order of the keys is important. JSON is not JavaScript. The order of keys is NOT significant in JSON, and many, many JSON implementations parse JSON objects

Re: [sqlite] Performance Issue on Large Table

2020-02-21 Thread Jens Alfke
> On Feb 20, 2020, at 5:24 PM, Chip Beaulieu wrote: > > CREATE INDEX `INX_Verses_BID` ON `Verses` (`BibleID`) > CREATE INDEX `INX_Verses_BID_B_C` ON `Verses` (`BibleID`,`Book`,`Chapter`) > CREATE INDEX `INX_Verses_BI_B_C_V` ON `Verses` > (`BibleID`,`Book`,`Chapter`,`VerseNumber`) I believe

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Carl Edquist
If you have example code for a mechanism that is more space efficient and/or faster, please share it with us. "Bencode" is approximately the same space-wise as JSON, but encoding/decoding is potentially faster since it doesn't have to do any escaping for strings:

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Wout Mertens
On Fri, Feb 21, 2020 at 3:03 PM Richard Hipp wrote: > If you > have example code for a mechanism that is more space efficient and/or > faster, please share it with us. I'll see if I can prototype something in JS - I'd be keeping the layouts in a helper table, and I wouldn't be storing the values

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Richard Hipp
On 2/21/20, Wout Mertens wrote: > The idea is that upon storing the JSON > data, the JSON1 extension parses it, extracts the layouts recursively, > stores them when they are not known yet, and then only stores the > values in the binary format with the layout identifiers. I experimented with a

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Wout Mertens
On Fri, Feb 21, 2020 at 2:37 PM Warren Young wrote: > > On Feb 21, 2020, at 5:20 AM, Wout Mertens wrote: > > Queries can go faster, because a query like `where json_extract(json, > > '$.foo') = 'bar'` can first check the layouts to see which ones apply, > > SQLite’s JSON1 extension is a storage

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Warren Young
On Feb 21, 2020, at 5:20 AM, Wout Mertens wrote: > > In JavaScript, objects are key-value collections with unique keys, where > the order of the keys is important. ECMAScript §13.7.5.15 (2019 edition) says, "The mechanics and order of enumerating the properties is not specified but must

Re: [sqlite] Performance Issue on Large Table

2020-02-21 Thread Luuk
On 21-2-2020 02:24, Chip Beaulieu wrote: I have a table with 4.5 million records with full text indexing. Reads are very fast, but deleting / inserting / updating takes on average about 50 seconds per record. I often do batches of 30,000 deletes / inserts at a time. The last batch took 10

[sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Wout Mertens
Hi, I use SQLite as a MaybeSQL store, mixing fixed columns with schemaless JSON columns. It's really great. In JavaScript, objects are key-value collections with unique keys, where the order of the keys is important. Most JSVMs store them as a pointer to a layout and then the values. The layout

Re: [sqlite] Is randomblob(16) a good guid generation across multiple computers?

2020-02-21 Thread Richard Hipp
On 2/21/20, Andy KU7T wrote: > Are you saying the PRNG on Windows is not good enough to use > randomblob(16) in Sqlite? All I need is a reasonable assurance that is are > unique... The default PRNG on Windows is fine for generating globally unique identifiers. The complaint is that the seeding