I’m confused - this is “low-volume data” but you're worried about exceeding 4000 connections? Maybe I’m missing something in translation; how many QPS do you expect?
4000 active connections is pretty crazy, even for a high-traffic system. http://www.mysqlcalculator.com/ expects that to consume about 11G on the server. Just use a connection pool with a hard cap of 12 and a modest timeout. You want to avoid leaving a lot of idle connections around, but if you have 4000 actually busy connections I would be worried more about your single node melting down. You probably want to consider federation or alternative database architectures. Jeff On Thu, May 25, 2017 at 1:14 PM, 'Nicholas (Google Cloud Support)' via Google App Engine <google-appengine@googlegroups.com> wrote: > Those are legitimate concerns when dealing with Cloud SQL at this scale. > I would recommend as you've suggested that you keep the > max_concurrent_requests sufficiently low to avoid the *per instance limit*. > This can also tend to keep your instances somewhat more responsive in > general. > > Regarding the 4000 connection limit, this limit may not seem like much at > a large scale but it can be stretched. Depending on the nature of what > those connections are doing, you may be able to offload read-only > operations to a read replica > <https://cloud.google.com/sql/docs/mysql/replication/create-replica#create-2nd-gen>, > likely reducing the number of concurrent connections required for the > master and read replica(s). > > Note that there are also other storage services > <https://cloud.google.com/storage-options/> that may be more appropriate > depending on the type of data stored, the volume of data, the availability > you need and the scale. For low-volume data (like user profiles), Cloud > Datastore may be better suited. Not knowing exactly what your needs are, > I'd recommend reviewing each storage and its limitations to decide what's > best for your use case. > > On Tuesday, May 23, 2017 at 8:17:49 AM UTC-4, Attila-Mihaly Balazs wrote: >> >> I have an app (Python 2.7, App Engine standard, EU region) which uses >> CloudSQL to store some low-volume data (like user profiles). Unfortunately >> CloudSQL has two limitations with regards to the maximum number of >> connections: >> >> - a global limit of 4000 (https://cloud.google.com/sql/faq - for >> second-generation instances) >> - a per instance limit of 12 (from the same URL as above - "Each App >> Engine instance running in a standard environment cannot have more than 12 >> concurrent connections to a Google Cloud SQL instance") >> >> To make sure that I respect this I can take two approaches, neither of >> them foolproof: >> >> - use frontend (F) instances with automatic scaling and set >> "max_concurrent_requests" sufficiently low. This solves the second issue, >> but if there are sufficient requests such that GAE scales my app to many >> instances, I can break the global connection limit (and there is no >> "max_num_instances" or something similar for the auto scaler) >> - use backaend (B) instances where I can specify "max_instances", however >> I don't know how many requests are routed to one instance (when >> "threadsafe: true" is set in app.yaml) - though I suspect not many - and I >> frequently get the "Request was aborted after waiting too long to attempt >> to service your request." error because I have too few instances (I'm >> running in the EU region so can't start more than 25 instances with the >> basic scaler :( ). >> >> One quick hack would be to use the backend instances but also run >> multiple services (like backend0, backend1, backend2, etc) and have the >> client randomly pick one (I control both the server and the client) to work >> around the 25 instance / service limit, however that feels very much like a >> hack :(. >> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at https://groups.google.com/group/google-appengine. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/google-appengine/679e332c-21a1-4a38-b5bf- > ab7abd681437%40googlegroups.com > <https://groups.google.com/d/msgid/google-appengine/679e332c-21a1-4a38-b5bf-ab7abd681437%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CADK-0uiqJJHHC6EzDnErT5F7hf0xmM1F8RQWVp6Wn%3DU_SUVanw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
