Hey Minie, As Jeff Schnitzer has pointed out, the issue stems from the fact that the PyMongo driver is wanting to use threads (via thread.open() <https://github.com/mongodb/mongo-python-driver/blob/master/pymongo/periodic_executor.py#L68>) to manage the connection pool. The thread opened in this way will be joined to the main thread when the response is finally sent for each request. You can find the driver's source on github <https://github.com/mongodb/mongo-python-driver/>.
As they also pointed out, you can use Managed VMs <https://cloud.google.com/appengine/docs/python/managed-vms/> to run threads indefinitely, and your code will probably work without any alteration. In this regard - allowing threading, direct access to the network interface, process-control and the filesystem, Managed VMs are much more like a traditional VPS hosting your python runtime (used to run your WSGI app) rather than an App Engine instance, with the sandbox and special runtime. Now, the other connection libraries you mention, if they're based on PyMongo, will of course share the same issue with its connection pool code wanting access to threads, since they use it as a basis and probably don't write their own connection pool code. As for the coroutine-based code you posted, generators work by returning a generator object <https://www.jeffknupp.com/blog/2013/04/07/improve-your-python-yield-and-generators-explained/>, which is then iterable with reference to its stored state. Creating the DBCONN object through client.get_default_database() will mean storing a copy of that object in memory which can be accessed by calls to "next()" on the generator. The issue is, this doesn't change the fact that behind the abstraction of the client object returned by client.get_default_database(), the implementation is still opening threads. As for Task Queues, they are a means of doing computation outside the life-cycle and limitations of a request on the instance, as specified by the limitations of the App Engine runtime, however they aren't the same as persistent threads, which would hold on to an open socket connection and communicate with request-handling threads. Through reading the docs, you can see tasks are scheduled to run, and a request is handled (with different deadlines) to represent the "invocation" of the task when it reaches the front of the queue. So, the one-time-invocation of tasks would prevent this from being a solution which you might have envisioned without looking more into the specifics. It seems as though the two pieces of advice offered by Jeff are in line with what I would recommend as well. The App Engine runtime is different from bare-metal processing, and the power offered by the platform needs to be taken in mind at the same time as the limitations which make it possible. Even still, Managed VMs seem to offer the best of both worlds. There may be ways to look into the source of the mongo driver library and modify it to use sockets, instance memory, and any other services in App Engine which would then work as a persistent cross-request connection pool, although that would be up to you to look into, as it would likely be a rather complex task. As a third option which would, in line with Jeff's idea to patch the driver, break out of certain implicit limitations in the problem-space as originally laid-out, you could drop the expectation of using mongo wire protocol and instead make use of REST to interact with the DB server <https://docs.mongodb.org/ecosystem/tools/http-interfaces/>. It's really up to you which direction to move in. I hope my contributions to this thread have been useful to you in clarifying exactly where you are, and possible roads forward. On Thursday, November 5, 2015 at 3:34:52 PM UTC-5, Minie Takalova wrote: > > Hi > > Re. timh: OK, I understand that keep open connection which is based on > threads, in frontend is prohibited. So, what is the easiest and clearest > way how to solve it? > > What I tried: > > * convinced PyMongo to work without threads. Tried many parameters (from > here http://api.mongodb.org/python/current/api/pymongo/mongo_client.html > ) during conectin established but still GAE end on timeout. > > * observe another connecion service like Motor > https://motor.readthedocs.org/en/stable/ asyncmongo > https://github.com/bitly/asyncmongo micromongo > http://pythonhosted.org/micromongo/ mongoengine http://mongoengine.org/ > Most of them depend on pymongo and another ones have unsuitable atributes. > Maybe, after many hours of research some of them will be usefull. > > * Tried to encapsulate connection to co-routine functionality. This is > working, but only on development server. > def coroutine_db_connection(): > client = pymongo.MongoClient(MONGO_URI) > DBCON = client.get_default_database() > while True: > yield DBCON > > * Tried run DB connection on separate backend module. Old backend is > depricated and new one, modules based is badly documented. Cannot find > usable example :( Find > https://cloud.google.com/appengine/docs/python/modules/ > Is there some step-by-step tutorial how to made simple GAE backend module > and how to take variable (open connection handle in my case) from backend > to frontend? > > Also I'am afraid about $cost if the connection will be open in backend > thread for lifetime of instance. Any clarify of that will be helpfull. > > * Made changes in PyMongo - in progress, I explore this files now. > > > --- Oh my GOD, keep open connection to database is such a simple task, in > another environment... --- > > > > > > > > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/fb4ab6fb-5389-4763-b204-dff0871b24b8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
