I tend to do this a lot to minimize the cost of fetching data; if I need all the information about a Person then I get all that info by loading a single entity, usually pulled from memcache. Most of my analytic processes look like map/reduce so indexing these complex structures wouldn't really help.
If you're in Javaland you can use Objectify4 and annotate a field with @Serialize(zip=true) to get it automatically compressed. I haven't compared resulting entity sizes, however. Jeff On Tue, Jan 17, 2012 at 1:27 PM, Brandon Wirtz <[email protected]> wrote: > Have you weighed Compression vs uncompressed in terms of speed, and cost? > > I'll end up testing to see, but I like to know the answers beforehand :-) > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of bFlood > Sent: Tuesday, January 17, 2012 4:54 AM > To: Google App Engine > Subject: [google-appengine] Re: Do you Serialize your Data? > > hi brandon > > I do this with a bunch of models, keep a few properties for search and then > dump the bulk of data into a compressed, json text property. > Obviously the json part might be overkill but its a lot more flexible going > forward if (err, when) you need to add more data (especially nested data) > > reads and writes are faster, YMMV > > YMMV > b > > On Jan 17, 4:44 am, "Brandon Wirtz" <[email protected]> wrote: >> We are looking at restructuring some of our calls, optimizing again >> for speed, we are thinking that rather than having a datastore call >> for a single cell, we'd serialize things. >> >> This makes searching suck, and we have had a few issues where >> serialized and unserialized data didn't quite match when we were done >> (because of weird encoded characters from sites) >> >> Serializing also locks you in to a single data structure for the most > part. >> >> We are also just considering: >> data as a zipped, delimited array >> uncompressed, delimited >> Double storing, so that we pay twice for writes, and can run >> analytics, but so that reads are against the serialized data, and faster > and cost less. >> >> What are anyone's thoughts? Have you had these discussions with your >> dev team? > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
