You almost never want to use "with request.tm". This cannot be nested with another call to request.tm.begin() and pyramid_tm already does request.tm.begin() for you. You can basically think of pyramid_tm as doing a "with request.tm" in a tween around all of your views except for some exception views in very rare circumstances (read the pyramid_tm docs for more about that).
The explicit vs implicit transaction managers is this: If you call tm.begin() twice in a row with the implicit transaction manager then it will be basically like calling tm.begin(), tm.abort() tm.begin(). The abort is implicit. With the explicit manager if you call tm.begin() twice in a row then the second call will raise an exception. It requires an explicit call to abort and an explicit call to begin(). This is probably what you expect, whereas the implicit workflow is the default bw-compat behavior that is very likely not what you expect if you're coming from a more traditional RDBMS background. You'll notice that in normal operation you never call tm.begin() or tm.abort()... pyramid_tm does this for you. The explicit manager weeds out issues where you might use the database before or after pyramid_tm is controlling the lifecycle whereas the implicit manager would silently hide some bugs where you call tm.begin() twice without realizing it - thus causing some objects to be detached from the first session because of the tm.abort() that was called alongside the second tm.begin()... Sorry I can't think of a more succinct way of saying it other than "just use the explicit manager". The only time you really need to think about transaction stuff is when pyramid_tm is not active which is basically at config-time and in console scripts. - Michael On Mon, Apr 2, 2018 at 2:54 PM, Zsolt Ero <zsolt....@gmail.com> wrote: > Thanks. So the key point for me is to just use engine.dispose() (I > don't want to dig into gunicorn preload, this seems much cleaner). > > About explicit manager: in practice it's as simple as > 1. enabling it in models/__init__.py > 2. using "with request.tm" everywhere instead of "with > transaction.manager", right? > > > This is more likely to be an issue in web requests where your entire > request lifecycle is not actually protected by a transaction. > > About your final sentence, I'm not sure I understand it. If I'm always > using request.dbsession, by definition I'm protected, to a request's > lifecycle am I not? > > Zsolt > > On 2 April 2018 at 21:47, Michael Merickel <mmeri...@gmail.com> wrote: > > The forking issue is likely because you're using a connection pool and so > > once a connection is opened at config-time, even though the session is > > properly closed the connection is just returned to the pool. The pool > here > > is shared across the fork which is bad. The basic solution here is to add > > some sort of pre-fork hook that closes out every existing connection in > the > > pool. I personally have not used forking wsgi servers in production and > thus > > have not needed to deal with this problem but I'm pretty sure my advice > hits > > at the core issue you're experiencing. Calling dispose() is one way to > make > > sure that the underlying connection is closed which is why it's working > for > > you. If I recall, gunicorn has some way to fork before config is called > such > > that each subprocess is loaded independently which can help with this as > > well... If memory serves it was something like turning off the preloading > > feature. > > > > The explicit manager is about weeding out situations where you use a > > connection after it's closed [1]. Basically without it you could use > > something joined to the "tm" after the "with" block is done and you might > > not see an error. This is more likely to be an issue in web requests > where > > your entire request lifecycle is not actually protected by a transaction. > > > > [1] > > https://docs.pylonsproject.org/projects/pyramid-tm/en/ > latest/#custom-transaction-managers > > > > On Mon, Apr 2, 2018 at 2:21 PM, Zsolt Ero <zsolt....@gmail.com> wrote: > >> > >> Michael, I've updated the code to your recommendation. > >> > >> https://github.com/hyperknot/pyramid_connections_bug > >> > >> It still requires the explicit engine.dispose() line, otherwise it > >> does bring the connection to the forked processes. > >> > >> This is with and without the explicit manager. What does the explicit > >> manager protect us from? > >> > >> Zsolt > >> > >> > >> On 2 April 2018 at 20:11, Michael Merickel <mmeri...@gmail.com> wrote: > >> > Using the pyramid-cookiecutter-alchemy setup you can access config > data > >> > at > >> > config time using a pattern like this: > >> > > >> > from transaction import TransactionManager > >> > > >> > from myapp.models import get_tm_session > >> > > >> > def main(global_config, **settings): > >> > config = Configurator(settings=settings) > >> > config.include('myapp.models') > >> > > >> > tm = TransactionManager(explicit=True) > >> > with tm: > >> > dbsession = get_tm_session(config. > registry['dbsession_factory'], > >> > tm) > >> > ... # do queries and stuff > >> > > >> > return config.make_wsgi_app() > >> > > >> > This will properly handle the lifecycle of a session for you, same as > >> > when > >> > serving a request. Be sure if you keep any objects around that you > >> > expunge > >> > them from the session and re-attach/merge them into any session where > >> > you > >> > use them.. ORM objects are only valid on the dbsession they were > loaded > >> > from. > >> > > >> > - Michael > >> > > >> > > >> > On Mon, Apr 2, 2018 at 11:59 AM, Zsolt Ero <zsolt....@gmail.com> > wrote: > >> >> > >> >> OK, I agree with that. Still, storing config values in the database > is > >> >> a common pattern for medium to large web apps, so it at least makes > >> >> sense to have some kind of resource about how to do it. I hope that > if > >> >> nothing else at least this thread will be useful for someone in the > >> >> future. > >> >> > >> >> -- > >> >> You received this message because you are subscribed to the Google > >> >> Groups > >> >> "pylons-discuss" group. > >> >> To unsubscribe from this group and stop receiving emails from it, > send > >> >> an > >> >> email to pylons-discuss+unsubscr...@googlegroups.com. > >> >> To post to this group, send email to pylons-discuss@googlegroups.com > . > >> >> To view this discussion on the web visit > >> >> > >> >> https://groups.google.com/d/msgid/pylons-discuss/CAKw- > smBuH5hKGpNgs-6dYkNiAU%3DGGUUOF7aLHAzHe0AE25T7qg%40mail.gmail.com. > >> >> For more options, visit https://groups.google.com/d/optout. > >> > > >> > > >> > -- > >> > You received this message because you are subscribed to a topic in the > >> > Google Groups "pylons-discuss" group. > >> > To unsubscribe from this topic, visit > >> > > >> > https://groups.google.com/d/topic/pylons-discuss/_ > MJflNUcjdg/unsubscribe. > >> > To unsubscribe from this group and all its topics, send an email to > >> > pylons-discuss+unsubscr...@googlegroups.com. > >> > To post to this group, send email to pylons-discuss@googlegroups.com. > >> > To view this discussion on the web visit > >> > > >> > https://groups.google.com/d/msgid/pylons-discuss/ > CAKdhhwGoPbgM7gDWBLxdanT%3DhcJQR8sT01E2xwWFxw1zxpCD1A%40mail.gmail.com. > >> > > >> > For more options, visit https://groups.google.com/d/optout. > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "pylons-discuss" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to pylons-discuss+unsubscr...@googlegroups.com. > >> To post to this group, send email to pylons-discuss@googlegroups.com. > >> To view this discussion on the web visit > >> https://groups.google.com/d/msgid/pylons-discuss/CAKw- > smBMmRqu2ZTPSp%3Djh%2BY0RriEwvxgdYQnwq--k4abeUkd-g%40mail.gmail.com. > >> For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "pylons-discuss" group. > > To unsubscribe from this topic, visit > > https://groups.google.com/d/topic/pylons-discuss/_MJflNUcjdg/unsubscribe > . > > To unsubscribe from this group and all its topics, send an email to > > pylons-discuss+unsubscr...@googlegroups.com. > > To post to this group, send email to pylons-discuss@googlegroups.com. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwGFjP7- > aYzz4NEewWv1kdbMxhH25hrX4JC3oR4ZizqYeg%40mail.gmail.com. > > > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pylons-discuss+unsubscr...@googlegroups.com. > To post to this group, send email to pylons-discuss@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/pylons-discuss/CAKw-smDuP2324T0GLi-J83UH-a5mVVvtPC3HyUAgB65aRSGQ5Q% > 40mail.gmail.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscr...@googlegroups.com. To post to this group, send email to pylons-discuss@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwG8OPmtCDZ9w%3DpJdLEesCiQRbwoRFnDXYXLsyxxV20-4g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
