Thank Nick and troberti. Great suggestions indeed. And in fact I had both: app stat and cloud trace. App stats record a bit more stuff but I found cloud trace a good compromise for the rough analysis.
I investigated a bit more and I found 4 major problem: 1 - memcached layer due to the usage pattern in ndb if start to fail or slow down will case a huge impact (even if batched). But that should be expected. I need to find if mc is down or slow and disable the usage in ndb context probably during that period. 2 - my app had a bug, due to the usage of a Reader/Writer lock I wrote (I tested it with up to 50 threads but it seem that the test is not always enough). I removed that part and the threads started to work better (I will ask suggestion for an usage pattern in a different thread) 3 - while I used a lot ndb.multi, async call and I cached in practice all the static data in memory (when the app start) an F1 machine wasn’t enough to granted more than 3/4 concurrent threads while keeping the latency under 1/1.5 seconds. (now it is time to understand why, because the code of the handlers is quite simple to be honest) 4 - I was mixing some slow work from task queue to the same instances and because they get the thread for a while, they could slow down a lot the other handler in queue 5 - I noticed that sometime the datastore transactions and put have great spike (in term of time). Normally they take 0.2 / 0.5 second and something they need 6 second to complete. I’m worried a bit about the entity ids. There are many entity (that belong to different ancestor) that use the same name, maybe are used or close to the same db shard. As usual, we learn by doing thing :) PS: troberti, are you from firigames ? So I have to thank you for both Phoenix HD and for your great BTree library! I use it a lot for my leaderboards ;) On Thursday, July 16, 2015 at 9:40:31 AM UTC+2, Cristian Marastoni wrote: > > My app is experiencing latency problem (up to 5 times). I noticed many > problem also accessing and writing memcache stuff. > Are there any issue reported? > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/59b19a27-226c-400e-8b69-fe6fc1b1e8ba%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
