Thanks for the detailed explanation. Yes I was talking about console scripts.
1. First, I had my console scripts based on initializedb.py (with no explicit manager in models/__init__.py) https://github.com/Pylons/pyramid-cookiecutter-alchemy/blob/latest/%7B%7Bcookiecutter.repo_name%7D%7D/%7B%7Bcookiecutter.repo_name%7D%7D/scripts/initializedb.py 2. Then I added explicit manager to models/__init__.py which broke my console scripts running in pshell, so I reverted to implicit. https://github.com/Pylons/pyramid/issues/3083 3. I migrated my console scripts to use to a request based config, using this pattern: setup_logging(config_uri) settings = get_appsettings(config_uri) request_dummy = Request.blank('/', base_url=base_url) env = bootstrap(config_uri, request=request_dummy) request = env['request'] do_things() # using "with transaction.manager:" env['closer']() 4. Now, using this new console script structure allowed me to replace transaction.manager with request.tm and enable explicit manager. Does my new console script make any sense? Maybe it'd make sense to modernize initializedb.py like this, as most new users would probably go through the same path as I did. Zsolt On 2 April 2018 at 23:18, Michael Merickel <[email protected]> wrote: > You almost never want to use "with request.tm". This cannot be nested with > another call to request.tm.begin() and pyramid_tm already does > request.tm.begin() for you. You can basically think of pyramid_tm as doing a > "with request.tm" in a tween around all of your views except for some > exception views in very rare circumstances (read the pyramid_tm docs for > more about that). > > The explicit vs implicit transaction managers is this: If you call > tm.begin() twice in a row with the implicit transaction manager then it will > be basically like calling tm.begin(), tm.abort() tm.begin(). The abort is > implicit. With the explicit manager if you call tm.begin() twice in a row > then the second call will raise an exception. It requires an explicit call > to abort and an explicit call to begin(). This is probably what you expect, > whereas the implicit workflow is the default bw-compat behavior that is very > likely not what you expect if you're coming from a more traditional RDBMS > background. > > You'll notice that in normal operation you never call tm.begin() or > tm.abort()... pyramid_tm does this for you. The explicit manager weeds out > issues where you might use the database before or after pyramid_tm is > controlling the lifecycle whereas the implicit manager would silently hide > some bugs where you call tm.begin() twice without realizing it - thus > causing some objects to be detached from the first session because of the > tm.abort() that was called alongside the second tm.begin()... Sorry I can't > think of a more succinct way of saying it other than "just use the explicit > manager". > > The only time you really need to think about transaction stuff is when > pyramid_tm is not active which is basically at config-time and in console > scripts. > > - Michael > > On Mon, Apr 2, 2018 at 2:54 PM, Zsolt Ero <[email protected]> wrote: >> >> Thanks. So the key point for me is to just use engine.dispose() (I >> don't want to dig into gunicorn preload, this seems much cleaner). >> >> About explicit manager: in practice it's as simple as >> 1. enabling it in models/__init__.py >> 2. using "with request.tm" everywhere instead of "with >> transaction.manager", right? >> >> > This is more likely to be an issue in web requests where your entire >> > request lifecycle is not actually protected by a transaction. >> >> About your final sentence, I'm not sure I understand it. If I'm always >> using request.dbsession, by definition I'm protected, to a request's >> lifecycle am I not? >> >> Zsolt >> >> On 2 April 2018 at 21:47, Michael Merickel <[email protected]> wrote: >> > The forking issue is likely because you're using a connection pool and >> > so >> > once a connection is opened at config-time, even though the session is >> > properly closed the connection is just returned to the pool. The pool >> > here >> > is shared across the fork which is bad. The basic solution here is to >> > add >> > some sort of pre-fork hook that closes out every existing connection in >> > the >> > pool. I personally have not used forking wsgi servers in production and >> > thus >> > have not needed to deal with this problem but I'm pretty sure my advice >> > hits >> > at the core issue you're experiencing. Calling dispose() is one way to >> > make >> > sure that the underlying connection is closed which is why it's working >> > for >> > you. If I recall, gunicorn has some way to fork before config is called >> > such >> > that each subprocess is loaded independently which can help with this as >> > well... If memory serves it was something like turning off the >> > preloading >> > feature. >> > >> > The explicit manager is about weeding out situations where you use a >> > connection after it's closed [1]. Basically without it you could use >> > something joined to the "tm" after the "with" block is done and you >> > might >> > not see an error. This is more likely to be an issue in web requests >> > where >> > your entire request lifecycle is not actually protected by a >> > transaction. >> > >> > [1] >> > >> > https://docs.pylonsproject.org/projects/pyramid-tm/en/latest/#custom-transaction-managers >> > >> > On Mon, Apr 2, 2018 at 2:21 PM, Zsolt Ero <[email protected]> wrote: >> >> >> >> Michael, I've updated the code to your recommendation. >> >> >> >> https://github.com/hyperknot/pyramid_connections_bug >> >> >> >> It still requires the explicit engine.dispose() line, otherwise it >> >> does bring the connection to the forked processes. >> >> >> >> This is with and without the explicit manager. What does the explicit >> >> manager protect us from? >> >> >> >> Zsolt >> >> >> >> >> >> On 2 April 2018 at 20:11, Michael Merickel <[email protected]> wrote: >> >> > Using the pyramid-cookiecutter-alchemy setup you can access config >> >> > data >> >> > at >> >> > config time using a pattern like this: >> >> > >> >> > from transaction import TransactionManager >> >> > >> >> > from myapp.models import get_tm_session >> >> > >> >> > def main(global_config, **settings): >> >> > config = Configurator(settings=settings) >> >> > config.include('myapp.models') >> >> > >> >> > tm = TransactionManager(explicit=True) >> >> > with tm: >> >> > dbsession = >> >> > get_tm_session(config.registry['dbsession_factory'], >> >> > tm) >> >> > ... # do queries and stuff >> >> > >> >> > return config.make_wsgi_app() >> >> > >> >> > This will properly handle the lifecycle of a session for you, same as >> >> > when >> >> > serving a request. Be sure if you keep any objects around that you >> >> > expunge >> >> > them from the session and re-attach/merge them into any session where >> >> > you >> >> > use them.. ORM objects are only valid on the dbsession they were >> >> > loaded >> >> > from. >> >> > >> >> > - Michael >> >> > >> >> > >> >> > On Mon, Apr 2, 2018 at 11:59 AM, Zsolt Ero <[email protected]> >> >> > wrote: >> >> >> >> >> >> OK, I agree with that. Still, storing config values in the database >> >> >> is >> >> >> a common pattern for medium to large web apps, so it at least makes >> >> >> sense to have some kind of resource about how to do it. I hope that >> >> >> if >> >> >> nothing else at least this thread will be useful for someone in the >> >> >> future. >> >> >> >> >> >> -- >> >> >> You received this message because you are subscribed to the Google >> >> >> Groups >> >> >> "pylons-discuss" group. >> >> >> To unsubscribe from this group and stop receiving emails from it, >> >> >> send >> >> >> an >> >> >> email to [email protected]. >> >> >> To post to this group, send email to >> >> >> [email protected]. >> >> >> To view this discussion on the web visit >> >> >> >> >> >> >> >> >> https://groups.google.com/d/msgid/pylons-discuss/CAKw-smBuH5hKGpNgs-6dYkNiAU%3DGGUUOF7aLHAzHe0AE25T7qg%40mail.gmail.com. >> >> >> For more options, visit https://groups.google.com/d/optout. >> >> > >> >> > >> >> > -- >> >> > You received this message because you are subscribed to a topic in >> >> > the >> >> > Google Groups "pylons-discuss" group. >> >> > To unsubscribe from this topic, visit >> >> > >> >> > >> >> > https://groups.google.com/d/topic/pylons-discuss/_MJflNUcjdg/unsubscribe. >> >> > To unsubscribe from this group and all its topics, send an email to >> >> > [email protected]. >> >> > To post to this group, send email to [email protected]. >> >> > To view this discussion on the web visit >> >> > >> >> > >> >> > https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwGoPbgM7gDWBLxdanT%3DhcJQR8sT01E2xwWFxw1zxpCD1A%40mail.gmail.com. >> >> > >> >> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "pylons-discuss" group. >> >> To unsubscribe from this group and stop receiving emails from it, send >> >> an >> >> email to [email protected]. >> >> To post to this group, send email to [email protected]. >> >> To view this discussion on the web visit >> >> >> >> https://groups.google.com/d/msgid/pylons-discuss/CAKw-smBMmRqu2ZTPSp%3Djh%2BY0RriEwvxgdYQnwq--k4abeUkd-g%40mail.gmail.com. >> >> For more options, visit https://groups.google.com/d/optout. >> > >> > >> > -- >> > You received this message because you are subscribed to a topic in the >> > Google Groups "pylons-discuss" group. >> > To unsubscribe from this topic, visit >> > >> > https://groups.google.com/d/topic/pylons-discuss/_MJflNUcjdg/unsubscribe. >> > To unsubscribe from this group and all its topics, send an email to >> > [email protected]. >> > To post to this group, send email to [email protected]. >> > To view this discussion on the web visit >> > >> > https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwGFjP7-aYzz4NEewWv1kdbMxhH25hrX4JC3oR4ZizqYeg%40mail.gmail.com. >> > >> > For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "pylons-discuss" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/pylons-discuss/CAKw-smDuP2324T0GLi-J83UH-a5mVVvtPC3HyUAgB65aRSGQ5Q%40mail.gmail.com. >> For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to a topic in the > Google Groups "pylons-discuss" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/pylons-discuss/_MJflNUcjdg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwG8OPmtCDZ9w%3DpJdLEesCiQRbwoRFnDXYXLsyxxV20-4g%40mail.gmail.com. > > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAKw-smAHfxzAWddrRFRsmOTn8eu%2BqxmVWhtBEhdNhyECDhqoew%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
