Re: [openstack-dev] [cinder][db] lazy loading of an attribute impossible
On 09/30/2016 10:54 AM, Roman Podoliaka wrote:
Michał,
You are absolutely right: this exception is raised when you try to
lazy-load instance attributes outside a Session scope. There is an
obvious problem with that - instances do not communicate with a DB on
their own - it's left up to Session [1].
Unfortunately, it does not play nicely with the "classic" DB access
layer we have in Cinder and other projects, when you have a notion of
pluggable DB APIs and SQLAlchemy implementation that looks like:
@require_context
@handle_db_data_error
def snapshot_create(context, values):
values['snapshot_metadata'] = _metadata_refs(values.get('metadata'),
models.SnapshotMetadata)
if not values.get('id'):
values['id'] = str(uuid.uuid4())
session = get_session()
with session.begin():
snapshot_ref = models.Snapshot()
snapshot_ref.update(values)
session.add(snapshot_ref)
return _snapshot_get(context, values['id'], session=session)
In this case a Session (and transaction) scope is bound to "public" DB
API functions. There are a few problems with this:
1) once a public DB function returns an instance, it becomes prone to
lazy-load errors, as the corresponding session (and DB transaction) is
already gone and it's not possible to load missing data (without
establishing a new session/transaction)
2) you have to carefully pass a Session object when doing calls to
"private" DB API functions to ensure they all participate in the very
same DB transaction. Otherwise snapshot_get() above would not see the
row created by snapshot_create() due to isolation of transactions in
RDBMS
3) if you do multiple calls to "public" DB API functions when handling
a single HTTP request it's not longer easy to do a rollback as every
function creates its own DB transaction
Mixing of Session objects creation with the actual business logic is
considered to be an anti-pattern in SQLAlchemy [2] due to problems
mentioned above.
At this point I suggest you take a look at [3] and start using in
Cinder: in Kilo we did a complete redesign of EngineFacade in oslo.db
- it won't solve all you problems with lazy-loading automatically, but
what it can do is provide a tool for declarative definition of session
(and transaction) scope, so that it's not longer limited to one
"public" DB API function and you can extend it when needed: you no
longer create a Session object explicitly, but rather mark methods
with a decorator, that will inject a session into the context, and all
callees will participate in the established session (thus, DB
transaction) rather than create a new one (my personal opinion is that
for web-services it's preferable to bind session/transaction scope to
the scope of one HTTP request, so that it's easy to roll back changes
on errors - we are not there yet, but some projects like Nova are
already moving the session scope up the stack, e.g. to objects layer).
+1 thanks Roman !
Thanks,
Roman
[1]
http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#what-does-the-session-do
[2]
http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it
[3]
https://specs.openstack.org/openstack/oslo-specs/specs/kilo/make-enginefacade-a-facade.html
On Thu, Sep 22, 2016 at 4:45 PM, Michał Dulko wrote:
Hi,
I've just noticed another Cinder bug [1], similar to past bugs [2], [3].
All of them have a common exception causing them:
sqlalchemy.orm.exc.DetachedInstanceError: Parent instance
<{$SQLAlchemyObject} at {$MemoryLocation}> is not bound to a Session;
lazy load operation of attribute '{$ColumnName}' cannot proceed
We've normally fixed them by simply making the $ColumnName eager-loaded,
but as there's another similar bug report, I'm starting to think that we
have some issue with how we're managing our DB connections and
SQLAlchemy objects are losing their sessions too quickly, before we'll
manage to lazy-load required stuff.
I'm not too experienced with SQLAlchemy session management, so I would
welcome any help with investigation.
Thanks,
Michal
[1] https://bugs.launchpad.net/cinder/+bug/1626499
[2] https://bugs.launchpad.net/cinder/+bug/1517763
[3] https://bugs.launchpad.net/cinder/+bug/1501838
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack
Re: [openstack-dev] [cinder][db] lazy loading of an attribute impossible
Michał,
You are absolutely right: this exception is raised when you try to
lazy-load instance attributes outside a Session scope. There is an
obvious problem with that - instances do not communicate with a DB on
their own - it's left up to Session [1].
Unfortunately, it does not play nicely with the "classic" DB access
layer we have in Cinder and other projects, when you have a notion of
pluggable DB APIs and SQLAlchemy implementation that looks like:
@require_context
@handle_db_data_error
def snapshot_create(context, values):
values['snapshot_metadata'] = _metadata_refs(values.get('metadata'),
models.SnapshotMetadata)
if not values.get('id'):
values['id'] = str(uuid.uuid4())
session = get_session()
with session.begin():
snapshot_ref = models.Snapshot()
snapshot_ref.update(values)
session.add(snapshot_ref)
return _snapshot_get(context, values['id'], session=session)
In this case a Session (and transaction) scope is bound to "public" DB
API functions. There are a few problems with this:
1) once a public DB function returns an instance, it becomes prone to
lazy-load errors, as the corresponding session (and DB transaction) is
already gone and it's not possible to load missing data (without
establishing a new session/transaction)
2) you have to carefully pass a Session object when doing calls to
"private" DB API functions to ensure they all participate in the very
same DB transaction. Otherwise snapshot_get() above would not see the
row created by snapshot_create() due to isolation of transactions in
RDBMS
3) if you do multiple calls to "public" DB API functions when handling
a single HTTP request it's not longer easy to do a rollback as every
function creates its own DB transaction
Mixing of Session objects creation with the actual business logic is
considered to be an anti-pattern in SQLAlchemy [2] due to problems
mentioned above.
At this point I suggest you take a look at [3] and start using in
Cinder: in Kilo we did a complete redesign of EngineFacade in oslo.db
- it won't solve all you problems with lazy-loading automatically, but
what it can do is provide a tool for declarative definition of session
(and transaction) scope, so that it's not longer limited to one
"public" DB API function and you can extend it when needed: you no
longer create a Session object explicitly, but rather mark methods
with a decorator, that will inject a session into the context, and all
callees will participate in the established session (thus, DB
transaction) rather than create a new one (my personal opinion is that
for web-services it's preferable to bind session/transaction scope to
the scope of one HTTP request, so that it's easy to roll back changes
on errors - we are not there yet, but some projects like Nova are
already moving the session scope up the stack, e.g. to objects layer).
Thanks,
Roman
[1]
http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#what-does-the-session-do
[2]
http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it
[3]
https://specs.openstack.org/openstack/oslo-specs/specs/kilo/make-enginefacade-a-facade.html
On Thu, Sep 22, 2016 at 4:45 PM, Michał Dulko wrote:
> Hi,
>
> I've just noticed another Cinder bug [1], similar to past bugs [2], [3].
> All of them have a common exception causing them:
>
> sqlalchemy.orm.exc.DetachedInstanceError: Parent instance
> <{$SQLAlchemyObject} at {$MemoryLocation}> is not bound to a Session;
> lazy load operation of attribute '{$ColumnName}' cannot proceed
>
> We've normally fixed them by simply making the $ColumnName eager-loaded,
> but as there's another similar bug report, I'm starting to think that we
> have some issue with how we're managing our DB connections and
> SQLAlchemy objects are losing their sessions too quickly, before we'll
> manage to lazy-load required stuff.
>
> I'm not too experienced with SQLAlchemy session management, so I would
> welcome any help with investigation.
>
> Thanks,
> Michal
>
>
> [1] https://bugs.launchpad.net/cinder/+bug/1626499
> [2] https://bugs.launchpad.net/cinder/+bug/1517763
> [3] https://bugs.launchpad.net/cinder/+bug/1501838
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [cinder][db] lazy loading of an attribute impossible
Hi,
I've just noticed another Cinder bug [1], similar to past bugs [2], [3].
All of them have a common exception causing them:
sqlalchemy.orm.exc.DetachedInstanceError: Parent instance
<{$SQLAlchemyObject} at {$MemoryLocation}> is not bound to a Session;
lazy load operation of attribute '{$ColumnName}' cannot proceed
We've normally fixed them by simply making the $ColumnName eager-loaded,
but as there's another similar bug report, I'm starting to think that we
have some issue with how we're managing our DB connections and
SQLAlchemy objects are losing their sessions too quickly, before we'll
manage to lazy-load required stuff.
I'm not too experienced with SQLAlchemy session management, so I would
welcome any help with investigation.
Thanks,
Michal
[1] https://bugs.launchpad.net/cinder/+bug/1626499
[2] https://bugs.launchpad.net/cinder/+bug/1517763
[3] https://bugs.launchpad.net/cinder/+bug/1501838
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
