Okay, I'm back and think that I get it now. And, yeah, storage space with 4D object fields is definitely a planning issue. Headline result: Copy and paste a 1MB JSON from a text editor and save it in 4D and it's likely to take about 2MB to store. More below. Corrections welcomed. I've also posted my 'findings' (current notions) on the forums on France where, hopefully, an errors I've made will be addressed. (I've experienced some vigorous correction already on that thread, so clearly folks aren't being shy about telling me when I'm wrong.)
On this thread, I'm also adding a story and some comments at the bottom for background and thoguht. The take-away from that is if you've got a small system, you'll probably never care about this and that if you have a big system you very likely might care a lot. Oh, and I posted a link to a feature request I made but Cannon Smith made a better one already, please vote for it: Option To Compress Object (and Text) Fields http://forums.4d.fr/Post/FR/17748608/1/17748609#17748609 Okay, as far as I can tell, the answer I was after is really simple: Question: How much room do objects require for storage in an object field? Answer: Very nearly the amount the object-as-JSON-without-whitespace would take stored as UTF-16. Put another way, if you take a 1MB JSON in a text editor, paste it into an object field and save it, it's likely to take roughly 2MB to store. Below are several points that I think are right (or close to it), but improvements and corrections are welcome: * Object fields don't store straight JSON, they use some kind of binary format. * The binary format seems to take up very nearly as much space as the raw JSON would, stripped of whitespace. (Padding between elements.) * It is likely that the storage space is double what you would expect because 4D tables always use UTF-16. If you need to store large character data, then UTF-16 is what you need. If you don't need to store large character data, then UTF-16 doubles the storage requirements for all your alpha/text/object data. If you're using something like SQLLite, MySQL/MariaDB, or PostgreSQL, you can specify the character set on a per-table basis. (All support UTF8, all but SQLite support even smaller character encodings.) So, if you're dealing with pure ASCII-like data, you could potentially save it without compression in another character set that takes 1/4 or 1/2 as much space as UTF-16. In 4D, I think you're always using UTF-16, is that right? * I don't care what the binary format is. No one else cares either ;-) * The binary format may change in future versions. Again, no one minds or cares about the details. * There are a lot of ways to format and store JSON. Some are inefficient (like the example I was using), some are more compact. This is a pretty well-travelled subject in the world as JSON is so common, regardless of language. I'll skip discussing how to pack and organize JSON as it's a different subject entirely from how the JSON is stored and handled by the DB engine. * If you want to use 4D's object field magic features, your data needs to be laid out in name-value pairs. And the name-value pairs are *stored in the data without compression.* You can see them right in the hex of the 4DD. * If you don't need to use 4D's object field magic features, you can rewrite your JSON to be more compact. Or, for that matter, you can convert it to something even more compact than that and store it as text. It would be great to have the option to compress JSON stored by 4D, so long as it allows indexed searching to work properly. The idea is that if you're using 4D to as a repository for flabby data (logs, instrumentation reports, etc.) or pre-rendered JSON for export/serving, you don't typically need to search into it, indexes are just fine. There's a cost to the compression/decompression, but that could pay for itself quickly in some situations. I put in a request like this and then Cannon Smith posted a link to a better-written existing request: Option To Compress Object (and Text) Fields http://forums.4d.fr/Post/FR/17748608/1/17748609#17748609 Please vote for Cannon's request. ********************************************************************** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:[email protected] **********************************************************************
