here are a few questions: 1. is mailman using the version_id mapper feature with these mappings ?
2. do mailman mappings use natural primary keys? in the above example are the primary keys of objects changing? On Sun, Jan 21, 2018 at 8:24 AM, Mike Bayer <[email protected]> wrote: > On Sat, Jan 20, 2018 at 8:19 PM, Mark Sapiro <[email protected]> wrote: >> Gnu Mailman 3 uses SQL Alchemy to access SQL databases. We have an issue >> with one of or unit tests which passes with SQLAlchemy <= 1.1.15 but which >> fails with SQLAlchemy >= 1.2. The failure is independent of the backend >> (sglite, mysql or pgsql). Here's a traceback from the failure. >> >> Traceback (most recent call last): >> File "/builds/mailman/mailman/src/mailman/model/tests/test_user.py", line >> 184, in test_absorb_memberships >> all_users = list(self._manager.users) >> File "/builds/mailman/mailman/src/mailman/model/usermanager.py", line 102, >> in users >> yield from store.query(User).order_by(User.id).all() >> File >> "/builds/mailman/mailman/.tox/py36-nocov/lib/python3.6/site-packages/sqlalchemy/orm/query.py", >> line 2726, in all >> return list(self) >> File >> "/builds/mailman/mailman/.tox/py36-nocov/lib/python3.6/site-packages/sqlalchemy/orm/query.py", >> line 2877, in __iter__ >> self.session._autoflush() >> File >> "/builds/mailman/mailman/.tox/py36-nocov/lib/python3.6/site-packages/sqlalchemy/orm/session.py", >> line 1434, in _autoflush >> self.flush() >> File >> "/builds/mailman/mailman/.tox/py36-nocov/lib/python3.6/site-packages/sqlalchemy/orm/session.py", >> line 2243, in flush >> self._flush(objects) >> File >> "/builds/mailman/mailman/.tox/py36-nocov/lib/python3.6/site-packages/sqlalchemy/orm/session.py", >> line 2369, in _flush >> transaction.rollback(_capture_exception=True) >> File >> "/builds/mailman/mailman/.tox/py36-nocov/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", >> line 66, in __exit__ >> compat.reraise(exc_type, exc_value, exc_tb) >> File >> "/builds/mailman/mailman/.tox/py36-nocov/lib/python3.6/site-packages/sqlalchemy/util/compat.py", >> line 187, in reraise >> raise value >> File >> "/builds/mailman/mailman/.tox/py36-nocov/lib/python3.6/site-packages/sqlalchemy/orm/session.py", >> line 2333, in _flush >> flush_context.execute() >> File >> "/builds/mailman/mailman/.tox/py36-nocov/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", >> line 391, in execute >> rec.execute(self) >> File >> "/builds/mailman/mailman/.tox/py36-nocov/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", >> line 542, in execute >> persistence.post_update(self.mapper, states, uow, cols) >> File >> "/builds/mailman/mailman/.tox/py36-nocov/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", >> line 234, in post_update >> mapper, table, update) >> File >> "/builds/mailman/mailman/.tox/py36-nocov/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", >> line 982, in _emit_post_update_statements >> (table.description, len(records), rows)) >> sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'user' expected >> to update 1 row(s); 0 were matched. > > > so StaleDataError is thrown when the ORM seeks to update or delete a > row based on primary key, and the row which that statement would match > is no longer present. This is essentially an assertion that the state > of the database is what the ORM expects based on its internal state. > The "stale" name refers to the typical case that some other > transaction has changed the state of the database while our > transaction is proceeding, however it is often the case that some > combination of how the mappings are set up combined with what the > application is doing causes the ORM to not have a row available which > should. > > In this case, the operation is specifically in relationship to the > usage of the "post_update" feature of mappings > (http://docs.sqlalchemy.org/en/latest/orm/relationship_persistence.html#post-update) > , since we can see that in the stack trace. The difference between > 1.1 and 1.2 is that the StaleDataError assertions have been added to > the post update system: > http://docs.sqlalchemy.org/en/latest/changelog/migration_12.html#post-update-integrates-with-orm-versioning > which is almost definitely part of the reason you get an error in 1.2 > and not in 1.1. > > What remains to be understood is whether your test produced an UPDATE > that didn't match a row in any case, and in 1.2, we just happen to be > raising an error for it, whereas under 1.1 this error condition > silently passed. Or, if the change in 1.2 itself has a regression and > the steps / SQL emitted are erroneously different. > > Things that would help us to see would be: 1. a look at the models > in question and how post_update is being set up and 2. when the > test runs, grab the full SQL output, using something like echo='debug' > on create_engine() or logging the "sqlalchemy.engine" logger. We > want to look at the exact sequence of SQL emitted both for the 1.1 > series and the 1.2 series, to see what the nature of the UPDATE > against a missing row is. If it looks like things work exactly the > same in 1.1 then that means your post_update has been missing this row > the whole time. > > This is just the top level information, I haven't looked at your > actual code yet just the stack trace. I'm not available today but > later in the week if you can produce self-contained reproducing > examples then I can further examine if we have a regression on the > SQLAlchemy side, or if this is expected behavior. > > > > > > >> >> >> The test that fails sets up 3 mailing lists and subscribes 2 users (Anne and >> Bart) to some or all of the lists with different roles. Then user Anne >> "absorbs" user Bart so user Bart is deleted and user Anne assumes all Bart's >> memberships. The actual code in the test may not be intelligible without >> more Mailman knowledge, but it is: >> def test_absorb_memberships(self): >> # When a user is absorbed, all of their user-subscribed memberships >> # are relinked to the absorbing user. >> mlist2 = create_list('[email protected]') >> mlist3 = create_list('[email protected]') >> with transaction(): >> # This has to happen in a transaction so that both the user and >> # the preferences objects get valid ids. >> bart = self._manager.create_user('[email protected]', 'Bart >> Person') >> set_preferred(bart) >> # Subscribe both users to self._mlist. >> self._mlist.subscribe(self._anne, MemberRole.member) >> self._mlist.subscribe(bart, MemberRole.moderator) >> # Subscribe only Bart to mlist2. >> mlist2.subscribe(bart, MemberRole.owner) >> # Subscribe only Bart's address to mlist3. >> mlist3.subscribe(bart.preferred_address, MemberRole.moderator) >> # There are now 4 memberships, one with Anne two with Bart's user >> and >> # one with Bart's address. >> all_members = list(self._manager.members) >> self.assertEqual(len(all_members), 4, all_members) >> # Do the absorption. >> self._anne.absorb(bart) >> # The Bart user has been deleted, leaving only the Anne user in the >> # user manager. >> all_users = list(self._manager.users) >> self.assertEqual(len(all_users), 1) >> self.assertEqual(all_users[0], self._anne) >> # There are no leftover memberships for user Bart. Anne owns all >> the >> # memberships. >> all_members = list(self._manager.members) >> self.assertEqual(len(all_members), 4, all_members) >> self.assertEqual(self._anne.memberships.member_count, 4) >> memberships = {(member.list_id, member.role): member >> for member in self._anne.memberships.members} >> # Note that Anne is now both a member and moderator of the test >> list. >> self.assertEqual(set(memberships), set([ >> ('test.example.com', MemberRole.member), >> ('test.example.com', MemberRole.moderator), >> ('test2.example.com', MemberRole.owner), >> ('test3.example.com', MemberRole.moderator), >> ])) >> # Both of Bart's previous user subscriptions are now transferred to >> # the Anne user. >> self.assertEqual( >> memberships[('test.example.com', MemberRole.moderator)].address, >> self._anne.preferred_address) >> self.assertEqual( >> memberships[('test2.example.com', MemberRole.owner)].address, >> self._anne.preferred_address) >> # Bart's address was subscribed; it must not have been changed. Of >> # course, Anne now controls [email protected]. >> key = ('test3.example.com', MemberRole.moderator) >> self.assertEqual(memberships[key].address.email, '[email protected]') >> self.assertEqual(self._manager.get_user('[email protected]'), >> self._anne) >> >> >> >> The start of the traceback is the statement >> # The Bart user has been deleted, leaving only the Anne user in the >> # user manager. >> all_users = list(self._manager.users) >> Versions older than 1.2 do not throw the sqlalchemy.orm.exc.StaleDataError >> exception at this point. >> >> If I attempt to fix the test by doing the `self._anne.absorb(bart)` in a >> transaction that gets committed on completion, the same exception is thrown >> on the commit. >> >> Does anyone have any ideas about this? >> >> -- >> SQLAlchemy - >> The Python SQL Toolkit and Object Relational Mapper >> >> http://www.sqlalchemy.org/ >> >> To post example code, please provide an MCVE: Minimal, Complete, and >> Verifiable Example. See http://stackoverflow.com/help/mcve for a full >> description. >> --- >> You received this message because you are subscribed to the Google Groups >> "sqlalchemy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/sqlalchemy. >> For more options, visit https://groups.google.com/d/optout. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
