Round one of my testing says that Serializing via JSON, or Pickle is SLOWER than just doing the reads from the datastore.
I'm reading up on model_to_protobuf Which is supposed to be faster. I'm a little sad. It would be awesome if there was a .put(key,serialize(stuff_i_would_write) -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of bFlood Sent: Tuesday, January 17, 2012 6:52 AM To: Google App Engine Subject: [google-appengine] Re: Do you Serialize your Data? hi brandon compression - not really, at least not with HRD. I remember it being useful in M/S but only for larger strings. I'm using this python property now, it only compresses over a certain size http://atastypixel.com/blog/wp-content/uploads/2011/01/compressible_text_pro perty.py_.txt great article jeff, I know I've read of few of your older posts about this in the past cheers brian On Jan 17, 9:21 am, "Brandon Wirtz" <[email protected]> wrote: > You wrote "How to Use GAE to Get Laid" and didn't tell anyone? Evil. > > > > > > > > -----Original Message----- > From: [email protected] > > [mailto:[email protected]] On Behalf Of Jeff Schnitzer > Sent: Tuesday, January 17, 2012 6:10 AM > To: [email protected] > Subject: Re: [google-appengine] Re: Do you Serialize your Data? > > In fact, I wrote up an explanation of this 6 months ago: > > http://blog.similarity.com/post/7541938593/how-to-build-an-online-dat... > e-nosql-edition > > Jeff > > On Tue, Jan 17, 2012 at 2:06 PM, Jeff Schnitzer <[email protected]> wrote: > > I tend to do this a lot to minimize the cost of fetching data; if I > > need all the information about a Person then I get all that info by > > loading a single entity, usually pulled from memcache. Most of my > > analytic processes look like map/reduce so indexing these complex > > structures wouldn't really help. > > > If you're in Javaland you can use Objectify4 and annotate a field > > with > > @Serialize(zip=true) to get it automatically compressed. I haven't > > compared resulting entity sizes, however. > > > Jeff > > > On Tue, Jan 17, 2012 at 1:27 PM, Brandon Wirtz <[email protected]> > wrote: > >> Have you weighed Compression vs uncompressed in terms of speed, and cost? > > >> I'll end up testing to see, but I like to know the answers > >> beforehand > >> :-) > > >> -----Original Message----- > >> From: [email protected] > >> [mailto:[email protected]] On Behalf Of bFlood > >> Sent: Tuesday, January 17, 2012 4:54 AM > >> To: Google App Engine > >> Subject: [google-appengine] Re: Do you Serialize your Data? > > >> hi brandon > > >> I do this with a bunch of models, keep a few properties for search > >> and then dump the bulk of data into a compressed, json text property. > >> Obviously the json part might be overkill but its a lot more > >> flexible going forward if (err, when) you need to add more data > >> (especially nested data) > > >> reads and writes are faster, YMMV > > >> YMMV > >> b > > >> On Jan 17, 4:44 am, "Brandon Wirtz" <[email protected]> wrote: > >>> We are looking at restructuring some of our calls, optimizing > >>> again for speed, we are thinking that rather than having a > >>> datastore call for a single cell, we'd serialize things. > > >>> This makes searching suck, and we have had a few issues where > >>> serialized and unserialized data didn't quite match when we were > >>> done (because of weird encoded characters from sites) > > >>> Serializing also locks you in to a single data structure for the > >>> most > >> part. > > >>> We are also just considering: > >>> data as a zipped, delimited array > >>> uncompressed, delimited > >>> Double storing, so that we pay twice for writes, and can run > >>> analytics, but so that reads are against the serialized data, and > >>> faster > >> and cost less. > > >>> What are anyone's thoughts? Have you had these discussions with > >>> your dev team? > > >> -- > >> You received this message because you are subscribed to the Google > >> Groups "Google App Engine" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]. > >> For more options, visit this group at > >>http://groups.google.com/group/google-appengine?hl=en. > > >> -- > >> You received this message because you are subscribed to the Google > >> Groups > "Google App Engine" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > [email protected]. > >> For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google > Groups "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
