The best answer is to use "with request.tm" in your console script as it
will then use the same manager config as your requests... which I suggest
to be explicit of course.
We need to update the cookiecutter with a better example here. The
initialize_db script is a bad example because it is in the unique position
of preparing the schema and thus you do *not* usually want to use
bootstrap() here (imagine that your main was doing database queries
requiring a particular schema that wasn't there yet)... whereas you do in
other scripts. In a normal console script you likely want to do:
setup_logging(config_uri)
with bootstrap(config_uri) as env:
request = env['request']
with request.tm:
request.dbsession.query(...)
- Michael
On Mon, Apr 2, 2018 at 4:35 PM, Zsolt Ero <[email protected]> wrote:
> Thanks for the detailed explanation. Yes I was talking about console
> scripts.
>
> 1. First, I had my console scripts based on initializedb.py (with no
> explicit manager in models/__init__.py)
> https://github.com/Pylons/pyramid-cookiecutter-alchemy/blob/latest/%7B%
> 7Bcookiecutter.repo_name%7D%7D/%7B%7Bcookiecutter.repo_name%7D%7D/scripts/
> initializedb.py
>
> 2. Then I added explicit manager to models/__init__.py which broke my
> console scripts running in pshell, so I reverted to implicit.
> https://github.com/Pylons/pyramid/issues/3083
>
> 3. I migrated my console scripts to use to a request based config,
> using this pattern:
>
> setup_logging(config_uri)
> settings = get_appsettings(config_uri)
> request_dummy = Request.blank('/', base_url=base_url)
> env = bootstrap(config_uri, request=request_dummy)
> request = env['request']
>
> do_things() # using "with transaction.manager:"
>
> env['closer']()
>
> 4. Now, using this new console script structure allowed me to replace
> transaction.manager with request.tm and enable explicit manager.
>
> Does my new console script make any sense? Maybe it'd make sense to
> modernize initializedb.py like this, as most new users would probably
> go through the same path as I did.
>
> Zsolt
>
>
>
>
> On 2 April 2018 at 23:18, Michael Merickel <[email protected]> wrote:
> > You almost never want to use "with request.tm". This cannot be nested
> with
> > another call to request.tm.begin() and pyramid_tm already does
> > request.tm.begin() for you. You can basically think of pyramid_tm as
> doing a
> > "with request.tm" in a tween around all of your views except for some
> > exception views in very rare circumstances (read the pyramid_tm docs for
> > more about that).
> >
> > The explicit vs implicit transaction managers is this: If you call
> > tm.begin() twice in a row with the implicit transaction manager then it
> will
> > be basically like calling tm.begin(), tm.abort() tm.begin(). The abort is
> > implicit. With the explicit manager if you call tm.begin() twice in a row
> > then the second call will raise an exception. It requires an explicit
> call
> > to abort and an explicit call to begin(). This is probably what you
> expect,
> > whereas the implicit workflow is the default bw-compat behavior that is
> very
> > likely not what you expect if you're coming from a more traditional RDBMS
> > background.
> >
> > You'll notice that in normal operation you never call tm.begin() or
> > tm.abort()... pyramid_tm does this for you. The explicit manager weeds
> out
> > issues where you might use the database before or after pyramid_tm is
> > controlling the lifecycle whereas the implicit manager would silently
> hide
> > some bugs where you call tm.begin() twice without realizing it - thus
> > causing some objects to be detached from the first session because of the
> > tm.abort() that was called alongside the second tm.begin()... Sorry I
> can't
> > think of a more succinct way of saying it other than "just use the
> explicit
> > manager".
> >
> > The only time you really need to think about transaction stuff is when
> > pyramid_tm is not active which is basically at config-time and in console
> > scripts.
> >
> > - Michael
> >
> > On Mon, Apr 2, 2018 at 2:54 PM, Zsolt Ero <[email protected]> wrote:
> >>
> >> Thanks. So the key point for me is to just use engine.dispose() (I
> >> don't want to dig into gunicorn preload, this seems much cleaner).
> >>
> >> About explicit manager: in practice it's as simple as
> >> 1. enabling it in models/__init__.py
> >> 2. using "with request.tm" everywhere instead of "with
> >> transaction.manager", right?
> >>
> >> > This is more likely to be an issue in web requests where your entire
> >> > request lifecycle is not actually protected by a transaction.
> >>
> >> About your final sentence, I'm not sure I understand it. If I'm always
> >> using request.dbsession, by definition I'm protected, to a request's
> >> lifecycle am I not?
> >>
> >> Zsolt
> >>
> >> On 2 April 2018 at 21:47, Michael Merickel <[email protected]> wrote:
> >> > The forking issue is likely because you're using a connection pool and
> >> > so
> >> > once a connection is opened at config-time, even though the session is
> >> > properly closed the connection is just returned to the pool. The pool
> >> > here
> >> > is shared across the fork which is bad. The basic solution here is to
> >> > add
> >> > some sort of pre-fork hook that closes out every existing connection
> in
> >> > the
> >> > pool. I personally have not used forking wsgi servers in production
> and
> >> > thus
> >> > have not needed to deal with this problem but I'm pretty sure my
> advice
> >> > hits
> >> > at the core issue you're experiencing. Calling dispose() is one way to
> >> > make
> >> > sure that the underlying connection is closed which is why it's
> working
> >> > for
> >> > you. If I recall, gunicorn has some way to fork before config is
> called
> >> > such
> >> > that each subprocess is loaded independently which can help with this
> as
> >> > well... If memory serves it was something like turning off the
> >> > preloading
> >> > feature.
> >> >
> >> > The explicit manager is about weeding out situations where you use a
> >> > connection after it's closed [1]. Basically without it you could use
> >> > something joined to the "tm" after the "with" block is done and you
> >> > might
> >> > not see an error. This is more likely to be an issue in web requests
> >> > where
> >> > your entire request lifecycle is not actually protected by a
> >> > transaction.
> >> >
> >> > [1]
> >> >
> >> > https://docs.pylonsproject.org/projects/pyramid-tm/en/
> latest/#custom-transaction-managers
> >> >
> >> > On Mon, Apr 2, 2018 at 2:21 PM, Zsolt Ero <[email protected]>
> wrote:
> >> >>
> >> >> Michael, I've updated the code to your recommendation.
> >> >>
> >> >> https://github.com/hyperknot/pyramid_connections_bug
> >> >>
> >> >> It still requires the explicit engine.dispose() line, otherwise it
> >> >> does bring the connection to the forked processes.
> >> >>
> >> >> This is with and without the explicit manager. What does the explicit
> >> >> manager protect us from?
> >> >>
> >> >> Zsolt
> >> >>
> >> >>
> >> >> On 2 April 2018 at 20:11, Michael Merickel <[email protected]>
> wrote:
> >> >> > Using the pyramid-cookiecutter-alchemy setup you can access config
> >> >> > data
> >> >> > at
> >> >> > config time using a pattern like this:
> >> >> >
> >> >> > from transaction import TransactionManager
> >> >> >
> >> >> > from myapp.models import get_tm_session
> >> >> >
> >> >> > def main(global_config, **settings):
> >> >> > config = Configurator(settings=settings)
> >> >> > config.include('myapp.models')
> >> >> >
> >> >> > tm = TransactionManager(explicit=True)
> >> >> > with tm:
> >> >> > dbsession =
> >> >> > get_tm_session(config.registry['dbsession_factory'],
> >> >> > tm)
> >> >> > ... # do queries and stuff
> >> >> >
> >> >> > return config.make_wsgi_app()
> >> >> >
> >> >> > This will properly handle the lifecycle of a session for you, same
> as
> >> >> > when
> >> >> > serving a request. Be sure if you keep any objects around that you
> >> >> > expunge
> >> >> > them from the session and re-attach/merge them into any session
> where
> >> >> > you
> >> >> > use them.. ORM objects are only valid on the dbsession they were
> >> >> > loaded
> >> >> > from.
> >> >> >
> >> >> > - Michael
> >> >> >
> >> >> >
> >> >> > On Mon, Apr 2, 2018 at 11:59 AM, Zsolt Ero <[email protected]>
> >> >> > wrote:
> >> >> >>
> >> >> >> OK, I agree with that. Still, storing config values in the
> database
> >> >> >> is
> >> >> >> a common pattern for medium to large web apps, so it at least
> makes
> >> >> >> sense to have some kind of resource about how to do it. I hope
> that
> >> >> >> if
> >> >> >> nothing else at least this thread will be useful for someone in
> the
> >> >> >> future.
> >> >> >>
> >> >> >> --
> >> >> >> You received this message because you are subscribed to the Google
> >> >> >> Groups
> >> >> >> "pylons-discuss" group.
> >> >> >> To unsubscribe from this group and stop receiving emails from it,
> >> >> >> send
> >> >> >> an
> >> >> >> email to [email protected].
> >> >> >> To post to this group, send email to
> >> >> >> [email protected].
> >> >> >> To view this discussion on the web visit
> >> >> >>
> >> >> >>
> >> >> >> https://groups.google.com/d/msgid/pylons-discuss/CAKw-
> smBuH5hKGpNgs-6dYkNiAU%3DGGUUOF7aLHAzHe0AE25T7qg%40mail.gmail.com.
> >> >> >> For more options, visit https://groups.google.com/d/optout.
> >> >> >
> >> >> >
> >> >> > --
> >> >> > You received this message because you are subscribed to a topic in
> >> >> > the
> >> >> > Google Groups "pylons-discuss" group.
> >> >> > To unsubscribe from this topic, visit
> >> >> >
> >> >> >
> >> >> > https://groups.google.com/d/topic/pylons-discuss/_
> MJflNUcjdg/unsubscribe.
> >> >> > To unsubscribe from this group and all its topics, send an email to
> >> >> > [email protected].
> >> >> > To post to this group, send email to pylons-discuss@googlegroups.
> com.
> >> >> > To view this discussion on the web visit
> >> >> >
> >> >> >
> >> >> > https://groups.google.com/d/msgid/pylons-discuss/
> CAKdhhwGoPbgM7gDWBLxdanT%3DhcJQR8sT01E2xwWFxw1zxpCD1A%40mail.gmail.com.
> >> >> >
> >> >> > For more options, visit https://groups.google.com/d/optout.
> >> >>
> >> >> --
> >> >> You received this message because you are subscribed to the Google
> >> >> Groups
> >> >> "pylons-discuss" group.
> >> >> To unsubscribe from this group and stop receiving emails from it,
> send
> >> >> an
> >> >> email to [email protected].
> >> >> To post to this group, send email to [email protected]
> .
> >> >> To view this discussion on the web visit
> >> >>
> >> >> https://groups.google.com/d/msgid/pylons-discuss/CAKw-
> smBMmRqu2ZTPSp%3Djh%2BY0RriEwvxgdYQnwq--k4abeUkd-g%40mail.gmail.com.
> >> >> For more options, visit https://groups.google.com/d/optout.
> >> >
> >> >
> >> > --
> >> > You received this message because you are subscribed to a topic in the
> >> > Google Groups "pylons-discuss" group.
> >> > To unsubscribe from this topic, visit
> >> >
> >> > https://groups.google.com/d/topic/pylons-discuss/_
> MJflNUcjdg/unsubscribe.
> >> > To unsubscribe from this group and all its topics, send an email to
> >> > [email protected].
> >> > To post to this group, send email to [email protected].
> >> > To view this discussion on the web visit
> >> >
> >> > https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwGFjP7-
> aYzz4NEewWv1kdbMxhH25hrX4JC3oR4ZizqYeg%40mail.gmail.com.
> >> >
> >> > For more options, visit https://groups.google.com/d/optout.
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "pylons-discuss" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an
> >> email to [email protected].
> >> To post to this group, send email to [email protected].
> >> To view this discussion on the web visit
> >> https://groups.google.com/d/msgid/pylons-discuss/CAKw-
> smDuP2324T0GLi-J83UH-a5mVVvtPC3HyUAgB65aRSGQ5Q%40mail.gmail.com.
> >> For more options, visit https://groups.google.com/d/optout.
> >
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "pylons-discuss" group.
> > To unsubscribe from this topic, visit
> > https://groups.google.com/d/topic/pylons-discuss/_MJflNUcjdg/unsubscribe
> .
> > To unsubscribe from this group and all its topics, send an email to
> > [email protected].
> > To post to this group, send email to [email protected].
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwG8OPmtCDZ9w%
> 3DpJdLEesCiQRbwoRFnDXYXLsyxxV20-4g%40mail.gmail.com.
> >
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/pylons-discuss/CAKw-smAHfxzAWddrRFRsmOTn8eu%
> 2BqxmVWhtBEhdNhyECDhqoew%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwHkje_98Yz%2BCMx1S5CTUMi-Jae23WWUhyuBd%3D6QpHauVg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
