Re: Serialization, in case this isn't common knowledge: When we recently switched to python 2.7, I did some serialization speed tests and cPickle is much faster than protobuf and pickle in that environment. We've been using keys only queries and memcache multi gets with cPickle and have seen significant speed ups. There are pickle issues if the data model changes, so you have to be sure to clear the cache in that case. Oh and we're not using NDB.
On Jan 29, 2013, at 2:11 PM, Jeff Schnitzer <[email protected]> wrote: > That is only relevant with *very* low hit rates. A read op is 7 times > more expensive than a small op. From a pure op-cost perspective, hit > rates greater than about 15% are a win. And while every app is > different, I disagree strongly with "very easy to miss the cache"; > usually there is some sort of working set, and 15% is a very modest > goal. > > I have discussed this exact issue with Guido (he can chime in if he's > still reading). As you see from those benchmarks, anything less than > a very high hit rate causes a significant speed degradation. And even > 100% hit rates aren't much faster than a normal query. Guido thought > this was due to Python serialization overhead, and I expect he would > know. > > Java serialization is _much_ faster. One to two orders of magnitude > faster. The relative latency of serialization vs the RPC is slanted > towards the later, and makes hybrid queries more generally useful. > But still, in the rare case where you don't want this behavior, you > can call hybrid(false) on your query command. > > Jeff > > On Tue, Jan 29, 2013 at 12:28 PM, alex <[email protected]> wrote: >> I was talking about low cache hit rates, which doesn't depend on >> serialization performance. >> >> say, if you have a 1m entities and do a query with limit 100, it's >> very easy to miss the cache. in this case query(keys only) + get_multi >> will cost you more than just a single query. >> >> On Tue, Jan 29, 2013 at 4:57 PM, Jeff Schnitzer <[email protected]> wrote: >>> NDB's primary problem is that python serialization costs are severe; >>> even cache hits don't help performance much. Java does not share this >>> problem. >>> >>> Jeff >>> >>> On Tue, Jan 29, 2013 at 3:13 AM, alex <[email protected]> wrote: >>>> That is not always optimal as you might easily get into low cache hit >>>> rate. For instance, that is the reason why NDB doesn't do that >>>> automatically (which it could). >>>> >>>> You can find more info/benchmark here: >>>> http://code.google.com/p/appengine-ndb-experiment/issues/detail?id=118#c6 >>>> >>>> On Mon, Jan 28, 2013 at 10:58 PM, Jeff Schnitzer <[email protected]> >>>> wrote: >>>>> Answered on the ofy mailing list, but just in case anyone else >>>>> stumbles across this: >>>>> >>>>> Objectify automatically converts queries for cacheable entities into a >>>>> keys-only query followed by a batch get. With cache hits, you will >>>>> pay for small operations instead of read operations. >>>>> >>>>> Jeff >>>>> >>>>> On Mon, Jan 28, 2013 at 5:48 AM, Marco Cuccato <[email protected]> wrote: >>>>>> Hello to all, >>>>>> i have a ununderstandable (for me) problem. >>>>>> >>>>>> My data model has very few entity kinds and very few entities instances >>>>>> at >>>>>> all (< 10). >>>>>> My app is deployed on appspot.com, so i can see resource status dashboard >>>>>> and i noticed this thing: no read and write operation is counted, but >>>>>> only >>>>>> small datastore operation. >>>>>> Unfortunately, this counter reach it's (unbilled) limit in a bounce of >>>>>> requests of this (Objectify) kind: >>>>>> >>>>>> ofy.load().type(Profile.class).list(); >>>>>> >>>>>> This kind of request is in my mind a Read Operation, not a Small (?) >>>>>> one... >>>>>> or am i wrong? >>>>>> This make Small Operations counter to reach limit with 5 requests >>>>>> (Profile >>>>>> has 4 instances only and it's cached - 100% cache hit on a subsequent >>>>>> call >>>>>> -). >>>>>> >>>>>> What is the problem? Why my app appears to do only small operations? >>>>>> Thanks >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google Groups >>>>>> "Google App Engine" group. >>>>>> To post to this group, send email to [email protected]. >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected]. >>>>>> Visit this group at >>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Google App Engine" group. >>>>> To post to this group, send email to [email protected]. >>>>> To unsubscribe from this group, send email to >>>>> [email protected]. >>>>> Visit this group at http://groups.google.com/group/google-appengine?hl=en. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Google App Engine" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/google-appengine?hl=en. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/google-appengine?hl=en. >>> For more options, visit https://groups.google.com/groups/opt_out. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/google-appengine?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/google-appengine?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
