I tried using Memorystore for Redis with the VPC connector and I'm still
seeing the same issue. If I try running the same commands from the cloud
console I have no issue. Something is going wrong inside the App Engine
environment for me.
Here is the psuedo code:
# see if data is cached
cached = redis_client.get('my_key')
if cached is not None:
return cached
# data not cached, query from datastore
q = datastore_client.query(kind='MyKind')
result = list(q.fetch())
# cache data for next query
t = time.time()
success = redis_client.set('my_key', result, ex=7200)
logging.info(f'redis.set took {time.time() - t} seconds') # this takes 35
seconds with memorystore for redis, <1 s from cloud console
On Friday, June 26, 2020 at 2:15:22 PM UTC-5 Chase Slater wrote:
> Thanks for the reply!
>
> I did some additional profiling and it turns out Datastore is not the
> issue but setting cache entries to Redis. I have a RedisLabs Cloud
> deployment and my redis_client.get commands are taking <1s but my
> redis_client.set commands are taking 40 seconds, these used to be on the
> order of 1s.
>
> I've looked at the Redis SlowLog but my worst latencies are 10ms. The
> objects I'm caching are 3MB or less and I'm well below my size and
> connection limits. The .set() command is called once every few minutes.
>
> My app is located in us-central and my Redis Labs instance is deployed to
> us-central1-1.gce.cloud.redislabs.com.
>
> Note that I am not using Memorystore for Redis. I am connecting to a
> RedisLabs cloud instance and billed directly by Redis Labs. Any advice on
> troubleshooting? It feels like a networking problem.
>
>
> On Fri, Jun 26, 2020 at 8:02 AM 'Olu' via Google App Engine <
> [email protected]> wrote:
>
>> Hello, Chase
>>
>> I understand you observed increased Latency in your App Engine
>> application, which you found out was relating to the latency is from reads
>> to your Datastore. There are a number of reasons that may be responsible
>> for Datastore latency including hotspotting and ramping up traffic to a
>> kind too rapidly, not following 500/50/5[1]. To determine specifically the
>> root cause of the latency issue, I recommend that you contact the GCP
>> Support Engineers[2][3]. This way a GCP Support engineer would be able to
>> work closely with you, reviewing your application configuration and
>> determine exactly what may be the exact cause of the long latency on reads
>> to your Datastore.
>>
>> Checking our internal issue tracker, I can confirm to you that there
>> seems to be no incident relating to latency on Datastore. So, this issue
>> might be specific to your setup, which would be better supported by a GCP
>> Support Engineer collaborating with you to find out the cause.
>>
>> [1]
>> https://cloud.google.com/datastore/docs/best-practices#ramping_up_traffic
>> [2]https://cloud.google.com/support/docs#contacting_technical_support
>> [3]https://cloud.google.com/support-hub
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Google App Engine" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/google-appengine/jy3tAGsERJY/unsubscribe
>> .
>> To unsubscribe from this group and all its topics, send an email to
>> [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/google-appengine/d9b5e9ce-448c-4ab7-ac60-9b8507563650o%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/google-appengine/d9b5e9ce-448c-4ab7-ac60-9b8507563650o%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/db6b618f-46d0-4dc1-89f5-da7c4eb5e487n%40googlegroups.com.
