[sqlalchemy] Temporarily disable/intercept ORM events on mutation
When customers call our JSON:API API, they can use an "include" parameter
to specify related objects to be appended to the response.
However, requests to include are not always satisfiable (e.g. if
job.supervisor=null, include=supervisor is ignored).
In order to prevent Marshmallow from trying to load nonexistent related
objects to append to our API responses, we need to tell it when to ignore a
relationship attribute, such as via setting
job.supervisor=marshmallow.missing (if it sees job.supervisor=None, it will
attempt (and fail) to load the null supervisor object, so we cannot just
leave it as-is).
Unfortunately, this causes problems as SQLAlchemy attempts to handle the
new value:
Error Traceback (most recent call last): File
"/Users/lucadou/IdeaProjects/person-api/api/unit_tests/test_person_service.py",
line 601, in test_get_people_include_job response =
self.person_service.get_people(QueryParameters({"include": "jobs"}))
File
"/Users/lucadou/IdeaProjects/person-api/api/src/person_service.py", line
61, in get_people response = person_schema.dump(people, many=True)
^ File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/schema.py",
line 557, in dump result = self._serialize(processed_obj, many=many)
^ File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/schema.py",
line 519, in _serialize return [ ^ File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/schema.py",
line 520, in self._serialize(d, many=False) File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/schema.py",
line 525, in _serialize value = field_obj.serialize(attr_name, obj,
accessor=self.get_attribute)
File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow_jsonapi/fields.py",
line 248, in serialize return super().serialize(attr, obj, accessor)
^^ File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/fields.py",
line 344, in serialize return self._serialize(value, attr, obj, **kwargs)
^^^ File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow_jsonapi/fields.py",
line 274, in _serialize self._serialize_included(item) File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow_jsonapi/fields.py",
line 280, in _serialize_included result = self.schema.dump(value)
^^^ File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/schema.py",
line 551, in dump processed_obj = self._invoke_dump_processors(
^ File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/schema.py",
line 1068, in _invoke_dump_processors data = self._invoke_processors(
File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/schema.py",
line 1225, in _invoke_processors data = processor(data, many=many, **kwargs)
File
"/Users/lucadou/IdeaProjects/person-api/api/src/model/schema/job_schema.py",
line 62, in set_null_supervisor job.supervisor = missing ^^ File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/sqlalchemy/orm/attributes.py",
line 536, in __set__ self.impl.set( File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/sqlalchemy/orm/attributes.py",
line 1466, in set value = self.fire_replace_event(state, dict_, value, old,
initiator) File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/sqlalchemy/orm/attributes.py",
line 1505, in fire_replace_event value = fn( ^^^ File
"/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/sqlalchemy/orm/attributes.py",
line 2167, in emit_backref_from_scalar_set_event instance_state(child),
^ AttributeError: '_Missing' object has no attribute
'_sa_instance_state'
Is there any way to temporarily disable ORM event listeners when we mutate
objects and have no intention of saving the changes/do not intend for the
ORM to act on them?
--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable
Example. See http://stackoverflow.com/help/mcve for a full description.
---
You received this message because you are subscribed to the
Re: [sqlalchemy] sqlalchemy.orm.exc.FlushError on subclass
Thanks. We went with the helper class route, and it seems to be working
much better than attempting to use inheritance in a manner that seems
unsupported.
On Wednesday, September 6, 2023 at 1:24:04 PM UTC-5 Mike Bayer wrote:
> if you can't correct this model to apply the persistence details to the
> concrete class you wish to persist and query, then you'd do the suggested
> "enable_typechecks=False". There is no attribute in SQLAlchemy named
> "meta" and no stack trace is given here so I dont know to what that refers.
>
> Overall I'm not sure how this API_Person class is useful because you can't
> query for them. I would think that if you cant change the original
> model then you'd have this API_Person as a series of helper functions that
> accept a Person as their argument.
>
>
>
> On Wed, Sep 6, 2023, at 1:56 PM, 'Luna Lucadou' via sqlalchemy wrote:
>
> The project I am working on is split up into several modules. Previously,
> each module had its own ORM classes.
> However, due to several bugs arising from forgetting to update each
> module's ORM classes in lock step when adding new functionality, we have
> decided it would be best to extract the ORM classes which interact with our
> DB into their own module and make that available to the other modules via
> pip.
>
> One of these modules, which monitors for changes in an upstream DB and
> applies them to ours, has some methods which are not present in the other
> modules and which cannot be easily extracted out due to its dependencies on
> upstream DB functionality.
>
> As such, in this module, we must subclass the ORM models which interact
> with our DB:
>
> models.apimodels.db.person.py:
> #...
> @dataclass(init=False, eq=True, unsafe_hash=True)
> class Person(Base):
> __tablename__ = "person"
>
> id: Mapped[int] = mapped_column(primary_key=True)
> first_name: Mapped[str]
> last_name: Mapped[str]
> email_address: Mapped[str]
> office_address: Mapped[str]
> office_phone_number: Mapped[str]
>
> # ...
>
> etl.models.api_db.api_person.py:
> #...
> from apimodels.db.person import Person as PersonBase
> # ...
> class API_Person(PersonBase):
> __tablename__ = "person"
> __table_args__ = {"keep_existing": True}
>
> def get_pvi(self):
> # ...
>
> def get_historical_pvis(self) -> list[str]:
> # ...
>
> def __eq__(self):
> # ...
>
> def __hash__(self):
> # ...
>
> @staticmethod
> def from_upstream_hub_person(
> uh_person: Optional[UH_Person],
> ) -> Optional["API_Person"]:
> # ...
>
> Of note is that this subclass does not add any new attributes or modify
> existing ones, it merely adds some helper methods related to identifying
> primary key changes in the upstream DB. (This is also why we override the
> eq and hash methods provided by dataclasses - incoming changesets have to
> be matched against existing records, even when primary keys change
> upstream.)
>
> This is effectively single-table inheritance, but it is not a good fit for
> polymorphic_identity since it is not a distinct class, merely adding
> module-specific helper methods, and if I am reading the documentation
> correctly, using polymorphic_identity would mean any records touched by
> the API_Person subclass (which is all of them) would no longer be usable
> by other modules, which do not extend any of the models.
>
> From what I have read, it seems like the keep_existing param should be of
> use here, but neither it nor extend_existing set to True help with the
> errors I am seeing when attempting to interact with this subclass in any
> way:
>
> sqlalchemy.orm.exc.FlushError: Attempting to flush an item of type 'model.api_db.api_person.API_Person'> as a member of collection
> "Identifier.person". Expected an object of type 'apimodels.db.person.Person'> or a polymorphic subclass of this type. If
> is a subclass of 'apimodels.db.person.Person'>, configure mapper "Mapper[Person(person)]" to
> load this subtype polymorphically, or set enable_typechecks=False to allow
> any subtype to be accepted for flush.
>
> I did try setting enable_typechecks to False, but this results in a
> different error when attempting to use getattr on the subclass:
>
> AttributeError: 'Person' object has no attribute 'meta'
>
> Is there a better way of doing this?
>
>
> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Comple
[sqlalchemy] sqlalchemy.orm.exc.FlushError on subclass
The project I am working on is split up into several modules. Previously,
each module had its own ORM classes.
However, due to several bugs arising from forgetting to update each
module's ORM classes in lock step when adding new functionality, we have
decided it would be best to extract the ORM classes which interact with our
DB into their own module and make that available to the other modules via
pip.
One of these modules, which monitors for changes in an upstream DB and
applies them to ours, has some methods which are not present in the other
modules and which cannot be easily extracted out due to its dependencies on
upstream DB functionality.
As such, in this module, we must subclass the ORM models which interact
with our DB:
models.apimodels.db.person.py:
#...
@dataclass(init=False, eq=True, unsafe_hash=True)
class Person(Base):
__tablename__ = "person"
id: Mapped[int] = mapped_column(primary_key=True)
first_name: Mapped[str]
last_name: Mapped[str]
email_address: Mapped[str]
office_address: Mapped[str]
office_phone_number: Mapped[str]
# ...
etl.models.api_db.api_person.py:
#...
from apimodels.db.person import Person as PersonBase
# ...
class API_Person(PersonBase):
__tablename__ = "person"
__table_args__ = {"keep_existing": True}
def get_pvi(self):
# ...
def get_historical_pvis(self) -> list[str]:
# ...
def __eq__(self):
# ...
def __hash__(self):
# ...
@staticmethod
def from_upstream_hub_person(
uh_person: Optional[UH_Person],
) -> Optional["API_Person"]:
# ...
Of note is that this subclass does not add any new attributes or modify
existing ones, it merely adds some helper methods related to identifying
primary key changes in the upstream DB. (This is also why we override the
eq and hash methods provided by dataclasses - incoming changesets have to
be matched against existing records, even when primary keys change
upstream.)
This is effectively single-table inheritance, but it is not a good fit for
polymorphic_identity since it is not a distinct class, merely adding
module-specific helper methods, and if I am reading the documentation
correctly, using polymorphic_identity would mean any records touched by the
API_Person subclass (which is all of them) would no longer be usable by
other modules, which do not extend any of the models.
>From what I have read, it seems like the keep_existing param should be of
use here, but neither it nor extend_existing set to True help with the
errors I am seeing when attempting to interact with this subclass in any
way:
sqlalchemy.orm.exc.FlushError: Attempting to flush an item of type as a member of collection
"Identifier.person". Expected an object of type or a polymorphic subclass of this type. If
is a subclass of , configure mapper "Mapper[Person(person)]" to
load this subtype polymorphically, or set enable_typechecks=False to allow
any subtype to be accepted for flush.
I did try setting enable_typechecks to False, but this results in a
different error when attempting to use getattr on the subclass:
AttributeError: 'Person' object has no attribute 'meta'
Is there a better way of doing this?
--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable
Example. See http://stackoverflow.com/help/mcve for a full description.
---
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sqlalchemy/bff6d34f-ea35-4aba-ba94-5ae1f29154fan%40googlegroups.com.
