this will be moving to a github issue, and also I will attempt to create a proof of concept.
Everything that happens inside of _memoized_attr__simple_lazy_clause involves creating copies of things and the state of the loader strategy is not modified until it's done. There is one "unique=False" flag set on existing bindparam objects, however if multiple threads all did that at the same time, it doesn't change the outcome. The two elements of the tuple returned, "crtierion" and "params", need to match, however these are returned as a tuple and that's what's assigned; if two different threads both ran the method at the same time, one would replace the other....but I guess the baked query is getting the state of the earlier one cached in some way, but, I don't see how the two generations are meaningfully different....but that's where something would need to be failing. Best option would be if it's not a problem for this code to run concurrently, however we can use the mapper._CONFIGURE_MUTEX for this, but there are probably a lot of other places where this kind of thing can happen if the introduction of baked + memoized attrs has this issue. On Tue, Feb 19, 2019 at 4:03 PM Krzysztof Sulejczak <[email protected]> wrote: > > Hi, > > I'm maintaining a multi-threaded application which uses sqlalchemy to access > mysql db. Some time ago we decided to upgrade > sqlalchemy version from 1.1.18 to 1.2.14 and since then we started observing > strange problems, the application started > throwing IntegrityError in code like that: > > translation = item.translation # translation is a relation > if translation is None: > session.add(Translation(item=item)) # this throws error > > Strange thing was that these errors happened rather rarely, there were a few > hundred application processes running and only > a few of them were throwing such errors. Restarting those processed made the > errors vanish. > I started debugging faulty processes and noticed that sql query generated > looked like this: > > SELECT <truncated list of columns> > FROM ebay_translations > WHERE NULL = ebay_translations.item_id > > Further debugging led me to LazyLoader._emit_lazyload and BackedQuery thingy > where I believe problem lays: > if a thread switch occurs inside _memoized_attr__simple_lazy_clause it could > happen that BindParameter keys and > cached relation query will not match each other. > > Since that discovery I've been running the application with patched version > of sqlalchemy for two months now, > IntegrityErrors are gone. > > Can you please take a look at this patch? Or maybe this issue is known and > there exists workaround for it although I searched > the mailing list archive and found nothing. > > best regards > Krzysztof Sulejczak > > -- > SQLAlchemy - > The Python SQL Toolkit and Object Relational Mapper > > http://www.sqlalchemy.org/ > > To post example code, please provide an MCVE: Minimal, Complete, and > Verifiable Example. See http://stackoverflow.com/help/mcve for a full > description. > --- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/sqlalchemy. > For more options, visit https://groups.google.com/d/optout. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
