You wrote "How to Use GAE to Get Laid" and didn't tell anyone? Evil.
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Jeff Schnitzer Sent: Tuesday, January 17, 2012 6:10 AM To: [email protected] Subject: Re: [google-appengine] Re: Do you Serialize your Data? In fact, I wrote up an explanation of this 6 months ago: http://blog.similarity.com/post/7541938593/how-to-build-an-online-dating-sit e-nosql-edition Jeff On Tue, Jan 17, 2012 at 2:06 PM, Jeff Schnitzer <[email protected]> wrote: > I tend to do this a lot to minimize the cost of fetching data; if I > need all the information about a Person then I get all that info by > loading a single entity, usually pulled from memcache. Most of my > analytic processes look like map/reduce so indexing these complex > structures wouldn't really help. > > If you're in Javaland you can use Objectify4 and annotate a field with > @Serialize(zip=true) to get it automatically compressed. I haven't > compared resulting entity sizes, however. > > Jeff > > On Tue, Jan 17, 2012 at 1:27 PM, Brandon Wirtz <[email protected]> wrote: >> Have you weighed Compression vs uncompressed in terms of speed, and cost? >> >> I'll end up testing to see, but I like to know the answers beforehand >> :-) >> >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of bFlood >> Sent: Tuesday, January 17, 2012 4:54 AM >> To: Google App Engine >> Subject: [google-appengine] Re: Do you Serialize your Data? >> >> hi brandon >> >> I do this with a bunch of models, keep a few properties for search >> and then dump the bulk of data into a compressed, json text property. >> Obviously the json part might be overkill but its a lot more flexible >> going forward if (err, when) you need to add more data (especially >> nested data) >> >> reads and writes are faster, YMMV >> >> YMMV >> b >> >> On Jan 17, 4:44 am, "Brandon Wirtz" <[email protected]> wrote: >>> We are looking at restructuring some of our calls, optimizing again >>> for speed, we are thinking that rather than having a datastore call >>> for a single cell, we'd serialize things. >>> >>> This makes searching suck, and we have had a few issues where >>> serialized and unserialized data didn't quite match when we were >>> done (because of weird encoded characters from sites) >>> >>> Serializing also locks you in to a single data structure for the >>> most >> part. >>> >>> We are also just considering: >>> data as a zipped, delimited array >>> uncompressed, delimited >>> Double storing, so that we pay twice for writes, and can run >>> analytics, but so that reads are against the serialized data, and >>> faster >> and cost less. >>> >>> What are anyone's thoughts? Have you had these discussions with your >>> dev team? >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to [email protected]. >> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. >> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
