On Sat, Jun 25, 2011 at 5:36 PM, Jeff Schnitzer <[email protected]> wrote:
> On Sat, Jun 25, 2011 at 3:26 PM, Alfred Fuller < > [email protected]> wrote: > >> This makes sense, and encourages more use of memcache. to hold entities. >>> One question that I've been wondering a while - presuming no caching, does >>> this query-keys+batch-get approach produce higher latency than a simple >>> query, and if so, by how much? >>> >> >> a db.Get will force strong consistency in the High Replication Datastore, >> which will introduce higher latency depending on how many entity groups you >> are fetching from (see other threads about this problem). If you set the >> read_policy to EVENTUAL_CONSISTENCY (or you are still using M/S Datastore) >> you will only pay the additional RPC latency. This is the not taking into >> account the 1000 entities vs 500 entities part of this question, which is >> hard to predict. >> > > For fetching let's say 100 entities, what % of the cost of the fetch is > just the RPC? Is this fairly insignificant compared to the cost of pulling > 100 entities from the datastore? > > Also, when you say RPC, I presume you mean the RPC from our appserver to > whatever system manages datastore requests, right? Because there is likely > a handful of requests from that system out to the individual tablet servers > that hold the entity data, right? > The only difference between querying keys then manually getting the entities (with EVENTUAL_CONSISTENCY in HRD) is: - Higher likelihood of seeing inconsistent results (entities that no longer match query) - Additional appserver -> datastore RPC overhead (you will have to run performance tests to see what this is, might be able to hid it with async calls) - Additional small op charge per key fetched otherwise they function identically. > > Also, is there any way we can get the transaction timestamp out on >>> datastore writes? This would *dramatically* improve the robustness of code >>> that tries to keep memcache in sync with the datastore during contention. >>> I've spoken with Alfred and Max about this, but I don't know if it's a >>> priority. This could potentially reduce datastore bills by orders of >>> magnitude. >>> >> >> :-), Ya it will save you the gets to populate the cache. >> > > A couple of us in Objectify-land tried to come up with a way to keep the > memcache synchronized with the datastore during contended writes, and we > failed to come up with a good answer. I'm fairly certain that without > access to the write timestamp, there's no way to guarantee synchronization. > > Jeff > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
