Thanks for the reply! I did some additional profiling and it turns out Datastore is not the issue but setting cache entries to Redis. I have a RedisLabs Cloud deployment and my redis_client.get commands are taking <1s but my redis_client.set commands are taking 40 seconds, these used to be on the order of 1s.
I've looked at the Redis SlowLog but my worst latencies are 10ms. The objects I'm caching are 3MB or less and I'm well below my size and connection limits. The .set() command is called once every few minutes. My app is located in us-central and my Redis Labs instance is deployed to us-central1-1.gce.cloud.redislabs.com. Note that I am not using Memorystore for Redis. I am connecting to a RedisLabs cloud instance and billed directly by Redis Labs. Any advice on troubleshooting? It feels like a networking problem. On Fri, Jun 26, 2020 at 8:02 AM 'Olu' via Google App Engine < [email protected]> wrote: > Hello, Chase > > I understand you observed increased Latency in your App Engine > application, which you found out was relating to the latency is from reads > to your Datastore. There are a number of reasons that may be responsible > for Datastore latency including hotspotting and ramping up traffic to a > kind too rapidly, not following 500/50/5[1]. To determine specifically the > root cause of the latency issue, I recommend that you contact the GCP > Support Engineers[2][3]. This way a GCP Support engineer would be able to > work closely with you, reviewing your application configuration and > determine exactly what may be the exact cause of the long latency on reads > to your Datastore. > > Checking our internal issue tracker, I can confirm to you that there seems > to be no incident relating to latency on Datastore. So, this issue might be > specific to your setup, which would be better supported by a GCP Support > Engineer collaborating with you to find out the cause. > > [1] > https://cloud.google.com/datastore/docs/best-practices#ramping_up_traffic > [2]https://cloud.google.com/support/docs#contacting_technical_support > [3]https://cloud.google.com/support-hub > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Google App Engine" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/google-appengine/jy3tAGsERJY/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/google-appengine/d9b5e9ce-448c-4ab7-ac60-9b8507563650o%40googlegroups.com > <https://groups.google.com/d/msgid/google-appengine/d9b5e9ce-448c-4ab7-ac60-9b8507563650o%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAB0C9cnStiskUN-4wpiFXMxVquqvvreGHVKyfL53ksaBQUXCVw%40mail.gmail.com.
