On Thu, May 30, 2019, at 9:21 AM, Mike Bayer wrote: > > > On Thu, May 30, 2019, at 8:42 AM, Chris Wilson wrote: >> Hi Mike, >> >> On Wed, 29 May 2019 at 15:30, Mike Bayer wrote: >>> Note that we suggested approaches that work in conjunction with the >>> serialized approach you gave, the @property approach and the mapper.load() >>> event approach. >> >> Unfortunately I think they would both require massive code changes. >>
re: "massive code changes", note that changing the TypeDecorator or TypeEngine API would mean that thousands of people would need to change their custom datatypes and third party dialects to accommodate for this change, which would be a deprecation warning as well as additional overhead. I believe we can adjust your application without a "massive" code change and that the new patterns can be installed using up front automation. However, consider that asking me to change the libraries' API so that you don't have to change your approach is still adding great inconvenience to other people not involved with this problem. >> >> >> Everywhere that we use Column(CompressedBinary), we'd have to rename the >> column and add property getters and setters. Unless there's a way to hook >> into the instrumentation machinery to do that automatically? > > All ORM events support propagation across all mappers, for InstanceEvents of > wihch load() is a member this is documented at > https://docs.sqlalchemy.org/en/13/orm/events.html?highlight=event%20load#instance-events > > > The load() event would be applied to all mapped classes, or to a specific > hierarchy / mixin hierarchy, if you prefer: > > @event.listens_for(Base, "load", propagate=True) > def my_event(...): > > > The load event is per instance so you would need to emit your Session.query > within the event. There are many ways that Core and ORM events are applied > automatically so this is not a problem. > > > As for the Column(), this is where the proposed pattern is the most broken, > that a database Column would return an ORM mapped object bound to the Session > directly. This is not how the ORM works and I can't support this as a pattern > going forward. When an ORM object links to another object or a collection, > relationship() is used for that. The whole world of folks that have ORM > objects that link to other ORM objects are using a Column to set up the > database-level linkage and relationship() to represent the load of the > object. > > In this case, you need something slightly different than a relationship(), > you probably want to just have a plain list of some kind to which your load() > handler appends, the addition of the @property can also be automated by using > the mapper_configured event. try using relationship with viewonly=True and lazy="noload". have your load() handler assign a list to this relationship attribute at load time using attributes.set_committed_value(). on the persistence side, use a before_insert() / before_update() event to read the contents of this list and update the column. > > > >> >>> However, if you are storing the fully serialized object in the column, like >>> the whole Cat object, you don't need to emit a SQL query to restore it, for >>> caching objects in serialized form you'd want to merge() it back into the >>> Session with load=False so that no SQL is emitted. >> >> We are not serializing the whole object, only the PK, but merging it back >> into the session with all its attributes expired, so that any attempt to >> access them triggers a load. That works well for us, unless the current >> session changes in the mean time (not the problem that I originally asked >> about, but a related one, that luckily isn't biting us right now). >> >>> You still need your Session though and of course, using a threadlocal >>> variable is the best way to make that happen right now without changing the >>> type API. >> >> The problem isn't that we're passing objects between Threads, it's that we >> can use multiple sessions in the same thread. I noticed it while trying to >> create a reproducible test case in the debugger, which was switching to our >> test/scratch database, setting a local variable, and then exiting the >> context (back to our main database) with that local variable still in scope >> (with expired attributes). When the debugger rendered the repr() of the >> local variable, it causes its relationships to be loaded from the live >> database, which didn't compare equal to objects in the test database, so I >> had to restart the debugger every time this happened. > > > There are other ways to get at the context. You can use the before_execute() > and after_execute() events, scan the compiled objects for the tables that > contain your datatypes in question, assign the connection being invoked to > the global variable, within the connection.info you would have the Session. > This is many steps but can be done. Unfortunately I don't have time today to > write a proof of concept for this, however I'd sooner write one for you using > the load() event, but again, I don't have time today to attend to this. > > >> >> It's not critical but it was annoying because it made debugging much harder >> and slower than I thought it could/should be. I admit that this is a niche >> use case, so I consider this a feature request instead of a bug. > > There has yet to be a mention of what "the feature" is exactly, I know how > such a thing would have to work but I assume you have a different idea of it. > >>>> > >> >>> The context available is the ExecutionContext, however this isn't passed to >>> the TypeEngine bind/result processor methods right now. That might not be a >>> bad idea in the future but for the moment would require a major breaking >>> API change that cannot be made quickly or trivially. >> >> Could it be added as an optional argument that is only passed if the >> recipient method is expecting it? > > that's how the change would have to be which involves using getargspec() way > ahead of time, emitting deprecation warnings, documenting, testing, all of > that. Additionally, this is a feature that nobody will ever use , this is the > worst case kind of situation where the feature is breaking, complex, and it > is for exactly one application that is doing something which we disagree is > even a good idea. > > SQLAlchemy is moving away from adding APIs that support broken use cases as > they spread confusion and maintenance burden so I really don't think this is > a good idea. You need at least the instance.load() event and at best you need > some new hooks to emit queries at the end of an ORM load, such as, you could > gather up all the Cat ids in one sweep at the end of a query and load them > all. An "after_query_results_loaded" event would be *VASTLY* more feasible. > > > > > >> >>> An example of passing information between a Session and the >>> execution-level context is at >>> https://github.com/sqlalchemy/sqlalchemy/wiki/SessionModifiedSQL but this >>> doesn't give you a way to get inside the TypeDecorator methods without >>> using a global threadlocal. >> >> Unfortunately even that doesn't help, I think, because the TypeDecorator is >> called after the statement has been executed, so there's no concept of a >> "current" load statement, only the last one, and I don't know for sure if >> that was really the same context/object/session that loaded the data being >> processed. >> >> I'm looking at whether something like the mutable extension would have >> access to the parent object, to coerce data structures containing SQLAlchemy >> objects into serializable form on the way into the database, and coerce them >> back to SQLAlchemy objects after a load event. >> >> Thanks again, Chris. >> >> -- >> SQLAlchemy - >> The Python SQL Toolkit and Object Relational Mapper >> >> http://www.sqlalchemy.org/ >> >> To post example code, please provide an MCVE: Minimal, Complete, and >> Verifiable Example. See http://stackoverflow.com/help/mcve for a full >> description. >> --- >> You received this message because you are subscribed to the Google Groups >> "sqlalchemy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to sqlalchemy+unsubscr...@googlegroups.com. >> To post to this group, send email to sqlalchemy@googlegroups.com. >> Visit this group at https://groups.google.com/group/sqlalchemy. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/sqlalchemy/81bb9420-3f48-408e-b5d4-04ebc3a6ea01%40googlegroups.com >> >> <https://groups.google.com/d/msgid/sqlalchemy/81bb9420-3f48-408e-b5d4-04ebc3a6ea01%40googlegroups.com?utm_medium=email&utm_source=footer>. >> For more options, visit https://groups.google.com/d/optout. > > > -- > SQLAlchemy - > The Python SQL Toolkit and Object Relational Mapper > > http://www.sqlalchemy.org/ > > To post example code, please provide an MCVE: Minimal, Complete, and > Verifiable Example. See http://stackoverflow.com/help/mcve for a full > description. > --- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sqlalchemy+unsubscr...@googlegroups.com. > To post to this group, send email to sqlalchemy@googlegroups.com. > Visit this group at https://groups.google.com/group/sqlalchemy. > To view this discussion on the web visit > https://groups.google.com/d/msgid/sqlalchemy/d50a0395-6116-4bdd-8354-2724e51c944d%40www.fastmail.com > > <https://groups.google.com/d/msgid/sqlalchemy/d50a0395-6116-4bdd-8354-2724e51c944d%40www.fastmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/354b44b6-de1f-4cd1-a9e4-a93674288892%40www.fastmail.com. For more options, visit https://groups.google.com/d/optout.
