Thanks for the answers. I've been using gunicorn with sync workers and it seems clear to me. If I understand correctly, up-till-some-point, everything in the script is shared across the processes, and these receive identical ids in Python. Everything which happens at import time is definitely in this "shared" memory space. Client libraries which cannot be shared, need to be regenerated / disposed / taken care of like with engine.dispose().
The fact that database engines get different ids in each worker means to me that I'm totally safe under gunicorn sync with SQLAlchemy. With waitress on the other hand, I'm surprised. I mean if waitress is multi-threaded, then how come it doesn't require every single module and sub-dependency to be thread-safe? I've only been using waitress for local development via pserve, but I see it's a popular choice for a production server as well. How can it work in a multi-threaded mode, if you have no idea about every single library's every dependency you use? Or you actually check out all your libraries? Now about SQL, why do I need any kind of transaction implementation? I mean isn't with engine.connect() as conn: conn.execute(stmt) a transaction? I'd just need to explicitly pass conn to every function which need DB access, but I don't see why is it important to do anything more than this, even with write queries. (I don't know how would I have DB access in view derivers with this pattern though). Zsolt On Wed, 1 May 2019 at 19:46, Jonathan Vanasco <[email protected]> wrote: > > > As I understand each worker needs it's own engine instance, don't they? I > > think the gunicorn behaviour is good, but I'm puzzled by the > > pserve/Waitress behaviour. Is this by design? > > Just to expand on Michael's comment on there usually only being one engine > but multiple connections... > > Worker's don't need their own engine instances, but they need their own > connection pools. gunicorn (and uwsgi) implement a forked process model, and > the database connections are not safe to move across the process boundaries > due to some things inherent to Python and/or filesystems . if you connect to > the database during setup in this model, you need to call `engine.dispose()` > when setup is complete or after the fork, or else weird and bad things will > eventually happen. see > https://docs.sqlalchemy.org/en/13/core/connections.html#engine-disposal . > waitress is multithreaded and the pool is supposed to be safe across process > boundaries. > > You should be able to just do an Engine on the request, but if you do ensure > you do the `dispose()` or set up an event to ensure process safety. it's all > the pooling docs https://docs.sqlalchemy.org/en/13/core/pooling.html > you'll also have to implement your own transaction management, which is > honestly not fun. > > if you are sure you're only doing SELECTS... you could have a dedicated > engine/connection string for a read-only database user and have that bypass > the transaction system entirely. i use that pattern a lot, but you need to > make sure you enforce it as read-only at the database credentials level or > you have the potential for a lot of issues bubbling up. > > > -- > You received this message because you are subscribed to a topic in the Google > Groups "pylons-discuss" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/pylons-discuss/8Rndsn6oLyg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pylons-discuss/0e53f127-d115-4724-95aa-5cfd14f23223%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAKw-smDgNYOx9oOxePp7vWAxVC67KzPgfmmayA5uKGcoBYKfNA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
