Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Fri, May 30, 2008 at 01:55:44PM -0400, Benji York wrote: > On Fri, May 30, 2008 at 1:40 PM, Brian Sutherland > <[EMAIL PROTECTED]> wrote: > > I've just decided to jettison IAbsoluteURL and make a new interface > > ICanonicalURL. The adapters for ICanonicalURL are available anywhere > > without specially wrapping the object in a location proxy. > > IAbsoluteURL doesn't neccesarily require a location proxy, it's just that the > default implementations require objects to implement ILocation. > > No need to define your own interface, just override the IAbsoluteURL > implementation(s) you don't like to behave differently (or better yet, don't > let them get registered in the first place). One thing I don't like about the default implementation is that if you override IAbsoluteURL for a container, it has no effect for contained objects. I suppose there's a good reason for that -- efficiency -- but it often makes custom IAbsoluteURL adapters pointless. LocationProxy helps then. Marius Gedminas -- Photons have energy, and trying to cram too many into too small of a space can cause a black hole to form, which is, needless to say, not a desirable trait for an optical computer. -- http://scottaaronson.com/blog/?p=261#comment-13693 signature.asc Description: Digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Fri, May 30, 2008 at 3:03 PM, Martijn Faassen <[EMAIL PROTECTED]> wrote: > Benji York wrote: >> I have an app that allows the same object to be reachable via several >> URLs. When we wanted one of the URLs to be preferred (none are really >> "cononical" in my case), then we did just as you suggest above and added >> a way for the IAbsoluteURL adapter(s) to know which URL was preferred. >> It worked out quite well. > > Cool. Where does the adapter get the information to make a decision from? I > mean, how does it know to use URL A and not URL B? Nothing fancy: we adapt the object to (names changed to clarify the example) IPreferredLocations to get back a list ordered by how preferred the locations are (from most-preferred to least), we then simply iterate over the list (it's always short) until we find the first place the object is visible, and use it. The location ordering is specified by the user (more or less, there are details I'm leaving out). -- Benji York Senior Software Engineer Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On May 30, 2008, at 9:44 AM, Martijn Faassen wrote: Hey, Michael Bayer wrote: [snip discussion on BoundMetadata I don't comprehend yet but probably should :)] As far as ScopedSession, it's really just a thread local variable. Like any global threadlocal, you can stick whatever you want on it at the start of a request, and tear it down at the end. Is it really needed to set things up at the start and the request and tear things down on the end, or can a session be retained between requests? I.e. is the intent to recreate a Session class each time a request is issued? in the 0.4 series, the Session is "sticky" in that whatever is in it retains the state that was last loaded - 0.4 does not go back to the database to re-fetch the state of an object unless its explicitly told to do so. When an HTTP request ends, assuming all objects in the session are marked as "clean", they're weakly referenced and will fall out of scope assuming those objects were only referenced by the request. So in that sense, you can just leave the Session alone and it'll work just fine for the next request...but any pending changes in the Session that weren't flushed for some reason would get carried over, as well as anything that might be strongly referenced elsewhere. So we always recommended in 0.4 to at least issue a session.clear() at the end of a request to just empty it out (but you can still reuse that session). Other options included scopedsession.remove() which tears the whole session down, the advantage there being that the new request could configure the next session differently (as in, binding it to something else). in 0.5, Session has been changed to be less reluctant to go and "re- fetch" data (which is contrary to the particular background I came from, but since then I've learned to see a broader scope of use cases). In 0.5, after a commit() or rollback(), the Session still may be holding on to objects, but their state is expired such that it will all be re-loaded upon access, and all "pending" and "deleted" states are reverted. So 0.5's Session, when configured in the default way, makes it very hard to get at "stale" state, so in that sense you can just re-use a session wihtout worrying much about what may have been left over. Then again it's also reasonable that you might want to have individual ScopedSessions for each application instance within a multi-app process, that way the burden of setting up/tearing down is reduced or removed. This indicates to me is possible to retain a Session object between two requests? I.e. it's a worthwhile goal to reduce the amount of Session configuration going on, right? Its not a strong argument either way to reuse a session or just make a new one on each request. In 0.4, making a brand new session on each request does have a "cleaner" feel to it since theres no chance of any state left hanging around. Its not an expensive operation. A single ScopedSession for a multi-app process is like a one- dimensional approach where both current thread and current app are identified by the current "get()" of the registry; a collection of ScopedSessions is more like a two-dimensional approach where the first level of registry (i.e. which ScopedSession do I choose) distinguishes between app instance, and the second level (i.e. what Session is bound to this thread ID) distinguishes between threads. All of that said I think it can work either way but I think the latter approach might have the explicitness you're looking for. I'm trying to understand why you think so. I am looking for a way to let developers use SQLAlchemy in a straightforward way. They should be able to import 'Session', instantiate session in their app, and everything works as expected. The framework takes care of making you get the appropriately configured Session. they can in fact do this without any issue.The question is, when someone writes the call "s = Session()" three times within one request, do you want each "s" to all reference the *same* set of objects ? that was the issue scoped_session() was meant to solve. Configuration is easy since you just configure a "Session" callable for that application. But you guys have this issue of "multiple applications in the same process" at play, which each talk to a different database. So thats where a decision has to be made how to deal with that complexity, either configure engine on one scoped_session per request, or configure multiple scoped_sessions, one per engine. The latter approach seems "easier" to me since you don't actually have to do anything on a per-request basis assuming 0.5 usage. Neither of these make any difference to the end user who sees the exact same usage pattern. (I'd still absolutely like to avoid passing in context explicitly each time you need a session; it puts unnecessary burden on t
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Fri, May 30, 2008 at 2:35 PM, Martijn Faassen <[EMAIL PROTECTED]> wrote: > I guess one difference here is that it seems more common in the RDB scenario > to have multiple ways to reach the same object, and even if you just publish > one, you can get to the actual object in multiple ways. > > In this case it'd be nice to convince IAbsoluteURL to somehow know which URL > option is the best, or alternatively, give the object somehow its > 'canonical' location (parent, etc) even if you don't get it that way. I have an app that allows the same object to be reachable via several URLs. When we wanted one of the URLs to be preferred (none are really "cononical" in my case), then we did just as you suggest above and added a way for the IAbsoluteURL adapter(s) to know which URL was preferred. It worked out quite well. -- Benji York Senior Software Engineer Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Fri, May 30, 2008 at 1:40 PM, Brian Sutherland <[EMAIL PROTECTED]> wrote: > I've just decided to jettison IAbsoluteURL and make a new interface > ICanonicalURL. The adapters for ICanonicalURL are available anywhere > without specially wrapping the object in a location proxy. IAbsoluteURL doesn't neccesarily require a location proxy, it's just that the default implementations require objects to implement ILocation. No need to define your own interface, just override the IAbsoluteURL implementation(s) you don't like to behave differently (or better yet, don't let them get registered in the first place). -- Benji York Senior Software Engineer Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Fri, May 30, 2008 at 03:49:46PM +0200, Martijn Faassen wrote: > Brian Sutherland wrote: > [snip] >> I'll note that there is a difference between where things are published >> (IAbsoluteURL) and where you should go to find their canonical >> representation. >> >> One problem I've found with using IAbsoluteURL and location proxies is >> getting the object directly from the database. In this situation you >> don't have an LocationProxy and IAbsoluteURL doesn't work. >> >> It's pretty nasty when IAbsoluteURL sometimes works and sometimes >> doesn't for what essentially is the same object. > > Agreed, we should ideally always get our objects from the db already > properly wrapped in a location proxy (or providing ILocation directly). I > think we can do this with containers based on relations; those containers > are under our control after all. I've just decided to jettison IAbsoluteURL and make a new interface ICanonicalURL. The adapters for ICanonicalURL are available anywhere without specially wrapping the object in a location proxy. > Is a direct one on one relation possible in SQLAlchemy? I.e. an object foo > that has an attribute 'bar' that is another ORM-ed object. If so we also > need to take care of that case somehow. This was a problem with sqlos containers as well. It gets nasty quickly. Say you have 2 containers, of people and addresses. Also people can have an address. So you can get to an address in 2 different ways: /adresses/address1 /people/person1/address1 What should the url for address1 be? I'd vote for /adresses/address1. But due to the LocationProxy, you'll get different results for different IAbsoluteURL calls depending on how you traversed to the object. > This is the "it should work like ZODB-backed objects" pattern I've been > trying to follow as a design guideline. Since for contained objects I can > always get the URL, it should work that way for RDB-based contained objects > too. I'm not sure it's worthwhile following the "it should work like ZODB-backed objects" pattern. > > Regards, > > Martijn > > ___ > Zope-Dev maillist - Zope-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope ) -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Thu, May 29, 2008 at 07:10:16PM +0200, Martijn Faassen wrote: > [snip] > > I'm not sure whether it would be a good idea to wrap this in a session > > property, or just register it as an adapter. The only other object that > > would need access to the session (either as a property or through > > adaption) would be the application instance root object. Something like: > > > > @adapter(MyApp) > > @provider(ISession) > > def root_session(context): > > return context._sessioncontext() > > This looks to me like it could be a simple function that looks up a local > utility instead: > > def session(): >return component.getUtility(ISessionContext)() > > We get the right context from the site that way. I don't see the point in > trying to re-implement local utilities with adapters while zope.component > already does something quite similar for you. That said, I still have hope > we can used ScopedSession and forgo a utility lookup here, see below... +lots Perhaps you may want the utility name as a keyword. I'll also note that doing it this way does not preclude using the ISession adapter in future. But, using the ISession adapter now means never being able to get rid of it again. > If you don't use the ZODB at all, you could still set up local sites (I'm > not sure how hard this is, but it *should* be possible; I've done it as a work around to override a global utility forced upon me. It's not too difficult. > > We do still need to setup parent pointers though for grok.url and > > IAbsoluteURL to work. It looks fairly easy to add location information > > to the containers themselves: > > (note that grok.url uses IAbsoluteURL, so we just care about IAbsoluteURL) I'll note that there is a difference between where things are published (IAbsoluteURL) and where you should go to find their canonical representation. One problem I've found with using IAbsoluteURL and location proxies is getting the object directly from the database. In this situation you don't have an LocationProxy and IAbsoluteURL doesn't work. It's pretty nasty when IAbsoluteURL sometimes works and sometimes doesn't for what essentially is the same object. -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On May 28, 2008, at 7:09 PM, Laurence Rowe wrote: Engine configuration is a subset of session configuration. You cannot have a single ScopedSession for a package if you want to have multiple instances of that application. We must work with unbound metadata if we have this goal. That implies that we must use bound sessions, a session associated with a particular engine (actually it could be more complex than this, you can associate particular classes/tables with particular engines within a session). I would agree that "bound" metadata is a little cumbersome in an environment that wants to switch engines within a single process. "Bound" metadata has always given us a lot of headaches because it's just *so* convenient that we just can't get rid of it, but at the same time people are just so tripped up by it, thinking it's necessary to make anything happen (it's not). Binding sessions allows better control of connection/transaction scope (since the Session relates to transactions, MetaData does not), so it's probabably the way to go in an environment that has a lot of things going on. As far as ScopedSession, it's really just a thread local variable. Like any global threadlocal, you can stick whatever you want on it at the start of a request, and tear it down at the end. Then again it's also reasonable that you might want to have individual ScopedSessions for each application instance within a multi-app process, that way the burden of setting up/tearing down is reduced or removed. A single ScopedSession for a multi-app process is like a one-dimensional approach where both current thread and current app are identified by the current "get()" of the registry; a collection of ScopedSessions is more like a two-dimensional approach where the first level of registry (i.e. which ScopedSession do I choose) distinguishes between app instance, and the second level (i.e. what Session is bound to this thread ID) distinguishes between threads. All of that said I think it can work either way but I think the latter approach might have the explicitness you're looking for. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Sat, May 24, 2008 at 09:30:18PM +0100, Laurence Rowe wrote: > Brian Sutherland wrote: >> On Fri, May 23, 2008 at 11:39:39PM +0100, Laurence Rowe wrote: >>> We need to differentiate between the interface for session configuration >>> and session usage from an application. >>> >>> For session usage I think it is fairly simple. We should define an >>> ISessionContext interface such that: >>> >>> class ISessionContext(Interface): >>> def __call__(): >>> "return a session in this context" >> >> +lots >> >> (I was thinking about proposing an interface called ISessionMaker doing >> much the same thing) >> >> I'm not sure what "in this context" means? > > The context of this application instance. Say you have two instances of an > application, one pointed at database A, another at database B. It is > possible to involve both applications in a single request / transaction. > You must be sure that you are working with the correct session. Are the instances you are talking about persistent objects in the ZODB? >>> A future version of collective.lead could implement an ISessionContext. >>> Client code however should have a cleaner interface, a plain ISession. >>> This is accessed through a lookup on the context, translated into a >>> simple adapter: >>> >>> def session_adapter(context): >>> session_context = queryUtility(ISessionContext, context=context, >> >> Why call queryUtility with the context keyword? >> >>> default=None) >>> if session_context is not None: >>> return session_context() >>> >>> This will allow application code to do something like: >>> >>> session = ISession(context) >>> ob = session.query(MyClass) >> >> This really confuses me. What is the context? Does it have any meaning? >> Or is it just a shorter way to write: >> >> session = getUtility(ISessionContext)() >> >> Does the value of context have an effect on what you get from the >> ISession adaptation? > > Yes, as it translates directly to the getUtility lookup. It ensures that > lookups in /appA go to a local utility defined in /appA and lookups in > /appB go to a local utility defined in /appB. I've been burned by using context as a keyword to getUtility before. When your context doesn't have a proper __parent__ pointer, the default IComponentLookup adapter falls back to the global site manager and ignores the local one. That causes no end of confusion and hard to debug bugs as people will call ISession on objects with and without __parent__ pointers and then wonder why it fails. Perhaps a more robust way is to rely on the site stored while traversing in zope.app.component.hooks? For example: * appA implements ISite and has a local ISessionContext utility * The path appA is traversed over (i.e. the url path is /appA/foo) * queryUtility(ISessionContext) then returns the local utility from appA without the need to specify a "context" (thanks to the magic in zope.app.component.hooks) I think the only situation where you really want to specify a "context" for queryUtility is when you want to access a local utility registered in /appB from a URL path like /appA/foo. That seems pretty rare and almost broken. You could implement a function like: def query_session(name=u''): return queryUtility(ISessionContext, name=name)() That does the right thing in almost all cases (i.e. uses the closest local site), and is much more robust. >> >>> Of course it would be possible to register a ScopedSession globally as >>> such a utility, but more usually a local utility should be registered. >> >> Depends what you're doing. If you are running without a ZODB, you have >> mostly just global utilities. >> >> It would be a pity if zope.sqlalchemy started to depend on the ZODB. > > Wihout ZODB and zope.app.component it seems unlikely that you would be able > to register a local utility, the idea of ISessionContext is so that you > might be able to register a ScopedSession as a global utility too. I very much like the idea of ISessionContext! I'm just not sure about how you suggest using it in client code. -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Fri, May 23, 2008 at 11:39:39PM +0100, Laurence Rowe wrote: > We need to differentiate between the interface for session configuration > and session usage from an application. > > For session usage I think it is fairly simple. We should define an > ISessionContext interface such that: > > class ISessionContext(Interface): > def __call__(): > "return a session in this context" +lots (I was thinking about proposing an interface called ISessionMaker doing much the same thing) I'm not sure what "in this context" means? > A future version of collective.lead could implement an ISessionContext. > Client code however should have a cleaner interface, a plain ISession. This > is accessed through a lookup on the context, translated into a simple > adapter: > > def session_adapter(context): > session_context = queryUtility(ISessionContext, context=context, Why call queryUtility with the context keyword? > default=None) > if session_context is not None: > return session_context() > > This will allow application code to do something like: > > session = ISession(context) > ob = session.query(MyClass) This really confuses me. What is the context? Does it have any meaning? Or is it just a shorter way to write: session = getUtility(ISessionContext)() Does the value of context have an effect on what you get from the ISession adaptation? > Of course it would be possible to register a ScopedSession globally as such > a utility, but more usually a local utility should be registered. Depends what you're doing. If you are running without a ZODB, you have mostly just global utilities. It would be a pity if zope.sqlalchemy started to depend on the ZODB. > (I haven't though about the consequences of this in pre-traversal, before > the site and local utilities are set up) > > session.remove() is not important, sessions are closed by the > zope.sqlalchemy datamanager and closed sessions are recyclable. Presumably > the session object would be referred to by a volatile attribute on the > local utility and the session would be GC'd along with the local utility > object itself. > > Table creation is another matter that I don't think too important. Implicit > creation of tables seems wrong, instead tables should only be created > explicitly, by the use clicking a button in the zope web interface (or > automatically on adding an application). An exception to this is sqlalchemy > in memory databases, which must be created on first access. > > Session configuration would be somewhat similar to collective.lead > currently (registering one as a local utility). > > Laurence > > Martijn Faassen wrote: >> Hi there, >> >> Today I had a discussion with Jasper Spaans about how to go about >> improving megrok.rdb, Grok's relational database integration which aims to >> integrate Grok with SQLAlchemy. We started developing megrok.rdb at the >> Grokkerdam sprint a few weeks ago. We reread the discussion surrounding >> zope.sqlalchemy for ideas on how to go about integration and >> configuration. I think these discussions reach wider than just Grok's >> concerns. Note that I'm not proposing we fold any of these ideas into >> zope.sqlalchemy itself, which should remain as free of policy as possible; >> it could become (yet another) extension. >> >> Let me run our current thinking by the list. >> >> What would be nice in Zope applications (and we think would be good for >> Grok) would be per-instance database configuration. That is, we want to be >> able to install multiple instances of the same application and then >> configure each of them with a different database URN and it should all >> work, each talking to their own database. >> >> Michael Bayer's suggestion involves the use of scoped sessions. He >> proposed the following code: >> >> Session = scoped_session() >> >> # start of request >> engine = get_appropriate_engine() >> Session(bind=engine) >> try: >> # do request >> finally: >> Session.remove() >> >> Let's go through the steps. First it makes a scoped session object, it >> then configures it with the right engine at the start of the request (it >> can do this on a per-class level), and then at the end of the request it >> removes the Session again, which results in the actual session being >> closed. >> >> Our get_appropriate_engine() would probably look the engine up as a local >> utility, as Laurence suggested. There is a bit of question about engine >> configuration, though. >> >> If we want to support the use case of looking up the engine URL in a >> persistent datastore (for instance one URL per location), we have a >> question of ordering. We cannot do it too early; at the start of the >> transaction there isn't a ZODB yet to talk to so we can't look up a local >> utility. We can try doing it just in time: >> >> _Session = scoped_session() >> >> def Session(*args, **kw): >> engine = get_appropriate_engine() >> _Session.bind(bind=engi
