Ah... Working off old docs that said cPickle didn't work so I didn't even try it.
I'll try that. I moved on to put_async which looks like other low hanging fruit for my app. Especially since Cache Misses are already the slowest of my pages, (and cache miss = write) -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Brian Quinlan Sent: Wednesday, January 18, 2012 4:33 PM To: [email protected] Subject: Re: [google-appengine] Re: Do you Serialize your Data? Hi Brandon, On Thu, Jan 19, 2012 at 11:25 AM, Brandon Wirtz <[email protected]> wrote: > Round one of my testing says that Serializing via JSON, or Pickle is > SLOWER than just doing the reads from the datastore. This might be different in the Python 2.7 runtime, where cPickle and C-accelerated JSON are available. Cheers, Brian > I'm reading up on model_to_protobuf > Which is supposed to be faster. > > I'm a little sad. It would be awesome if there was a > .put(key,serialize(stuff_i_would_write) > > > > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of bFlood > Sent: Tuesday, January 17, 2012 6:52 AM > To: Google App Engine > Subject: [google-appengine] Re: Do you Serialize your Data? > > hi brandon > > compression - not really, at least not with HRD. I remember it being > useful in M/S but only for larger strings. I'm using this python > property now, it only compresses over a certain size > http://atastypixel.com/blog/wp-content/uploads/2011/01/compressible_te > xt_pro > perty.py_.txt > > great article jeff, I know I've read of few of your older posts about > this in the past > > cheers > brian > > > On Jan 17, 9:21 am, "Brandon Wirtz" <[email protected]> wrote: >> You wrote "How to Use GAE to Get Laid" and didn't tell anyone? Evil. >> >> >> >> >> >> >> >> -----Original Message----- >> From: [email protected] >> >> [mailto:[email protected]] On Behalf Of Jeff >> Schnitzer >> Sent: Tuesday, January 17, 2012 6:10 AM >> To: [email protected] >> Subject: Re: [google-appengine] Re: Do you Serialize your Data? >> >> In fact, I wrote up an explanation of this 6 months ago: >> >> http://blog.similarity.com/post/7541938593/how-to-build-an-online-dat... >> e-nosql-edition >> >> Jeff >> >> On Tue, Jan 17, 2012 at 2:06 PM, Jeff Schnitzer <[email protected]> > wrote: >> > I tend to do this a lot to minimize the cost of fetching data; if I >> > need all the information about a Person then I get all that info by >> > loading a single entity, usually pulled from memcache. Most of my >> > analytic processes look like map/reduce so indexing these complex >> > structures wouldn't really help. >> >> > If you're in Javaland you can use Objectify4 and annotate a field >> > with >> > @Serialize(zip=true) to get it automatically compressed. I haven't >> > compared resulting entity sizes, however. >> >> > Jeff >> >> > On Tue, Jan 17, 2012 at 1:27 PM, Brandon Wirtz >> > <[email protected]> >> wrote: >> >> Have you weighed Compression vs uncompressed in terms of speed, >> >> and > cost? >> >> >> I'll end up testing to see, but I like to know the answers >> >> beforehand >> >> :-) >> >> >> -----Original Message----- >> >> From: [email protected] >> >> [mailto:[email protected]] On Behalf Of bFlood >> >> Sent: Tuesday, January 17, 2012 4:54 AM >> >> To: Google App Engine >> >> Subject: [google-appengine] Re: Do you Serialize your Data? >> >> >> hi brandon >> >> >> I do this with a bunch of models, keep a few properties for search >> >> and then dump the bulk of data into a compressed, json text property. >> >> Obviously the json part might be overkill but its a lot more >> >> flexible going forward if (err, when) you need to add more data >> >> (especially nested data) >> >> >> reads and writes are faster, YMMV >> >> >> YMMV >> >> b >> >> >> On Jan 17, 4:44 am, "Brandon Wirtz" <[email protected]> wrote: >> >>> We are looking at restructuring some of our calls, optimizing >> >>> again for speed, we are thinking that rather than having a >> >>> datastore call for a single cell, we'd serialize things. >> >> >>> This makes searching suck, and we have had a few issues where >> >>> serialized and unserialized data didn't quite match when we were >> >>> done (because of weird encoded characters from sites) >> >> >>> Serializing also locks you in to a single data structure for the >> >>> most >> >> part. >> >> >>> We are also just considering: >> >>> data as a zipped, delimited array uncompressed, delimited Double >> >>> storing, so that we pay twice for writes, and can run analytics, >> >>> but so that reads are against the serialized data, and faster >> >> and cost less. >> >> >>> What are anyone's thoughts? Have you had these discussions with >> >>> your dev team? >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups "Google App Engine" group. >> >> To post to this group, send email to [email protected]. >> >> To unsubscribe from this group, send email to >> >> [email protected]. >> >> For more options, visit this group at >> >>http://groups.google.com/group/google-appengine?hl=en. >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> "Google App Engine" group. >> >> To post to this group, send email to [email protected]. >> >> To unsubscribe from this group, send email to >> [email protected]. >> >> For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google > Groups "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to [email protected]. > For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
