Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-26 Thread Paul van Helden
> > > I experimented with a number of similar ideas for storing JSON when I > was first designing the JSON components for SQLite. I was never able > to find anything that was as fast or as compact as just storing the > original JSON text. > I've also done a lot of experiments and was surprised a

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-25 Thread Jens Alfke
> On Feb 25, 2020, at 6:12 AM, J Decker wrote: > > other than that; if space is really a concern, maybe a zip layer? In my experience, the concern is more about speed than size. Given the raw string/blob data from a SQLite column, and a specific property name/path, how fast can you find its

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-25 Thread J Decker
On Fri, Feb 21, 2020 at 6:03 AM Richard Hipp wrote: > On 2/21/20, Wout Mertens wrote: > > The idea is that upon storing the JSON > > data, the JSON1 extension parses it, extracts the layouts recursively, > > stores them when they are not known yet, and then only stores the > > values in the bina

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Martin Raiber
On 21.02.2020 15:03 Richard Hipp wrote: > On 2/21/20, Wout Mertens wrote: >> The idea is that upon storing the JSON >> data, the JSON1 extension parses it, extracts the layouts recursively, >> stores them when they are not known yet, and then only stores the >> values in the binary format with the

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Jens Alfke
> On Feb 21, 2020, at 4:20 AM, Wout Mertens wrote: > > I was wondering if the JSON extension could not do the same thing: for each > table, keep a hidden stash of object layouts, and store the values as > sqlite primitives. (you'd be able to disable this, in case the layouts > rarely repeat) W

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Jens Alfke
> On Feb 21, 2020, at 4:20 AM, Wout Mertens wrote: > > In JavaScript, objects are key-value collections with unique keys, where the > order of the keys is important. JSON is not JavaScript. The order of keys is NOT significant in JSON, and many, many JSON implementations parse JSON objects i

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Carl Edquist
If you have example code for a mechanism that is more space efficient and/or faster, please share it with us. "Bencode" is approximately the same space-wise as JSON, but encoding/decoding is potentially faster since it doesn't have to do any escaping for strings: https://en.wikipedia

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Wout Mertens
On Fri, Feb 21, 2020 at 3:03 PM Richard Hipp wrote: > If you > have example code for a mechanism that is more space efficient and/or > faster, please share it with us. I'll see if I can prototype something in JS - I'd be keeping the layouts in a helper table, and I wouldn't be storing the values

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Richard Hipp
On 2/21/20, Wout Mertens wrote: > The idea is that upon storing the JSON > data, the JSON1 extension parses it, extracts the layouts recursively, > stores them when they are not known yet, and then only stores the > values in the binary format with the layout identifiers. I experimented with a nu

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Wout Mertens
On Fri, Feb 21, 2020 at 2:37 PM Warren Young wrote: > > On Feb 21, 2020, at 5:20 AM, Wout Mertens wrote: > > Queries can go faster, because a query like `where json_extract(json, > > '$.foo') = 'bar'` can first check the layouts to see which ones apply, > > SQLite’s JSON1 extension is a storage a

Re: [sqlite] more efficient JSON encoding: idle musing

2020-02-21 Thread Warren Young
On Feb 21, 2020, at 5:20 AM, Wout Mertens wrote: > > In JavaScript, objects are key-value collections with unique keys, where > the order of the keys is important. ECMAScript §13.7.5.15 (2019 edition) says, "The mechanics and order of enumerating the properties is not specified but must conform