On 10/17/23 13:20, Walter Underwood wrote:
Gzipping the JSON can be a big win, especially if there are lots of repeated keys, like in state.json. Gzip has the advantage that some editors can natively unpack it.
It may save you some transfer time, provided the transport subsystem doesn't compress on the fly, but with JSON being all-or-nothing format, your problem's going to be RAM for the string representation plus RAM for the decoded object representation, of the entire store.
If you want it scalable, you want an "incremental" format like asn.1, protocol buffers, or avro.
Dima
