Re: [Zope-dev] Re: zope.sqlalchemy dependencies does not respect setup.py dev egg
Hi Laurence. I reverted to 0.2 and I am good to go. I will upgrade to 0.3 when dobbin can use beta0.5.3. Many thanks. Regards, David Laurence Rowe wrote: zope.sqlalchemy specifically requires a SessionExtension hook added in 0.4.7 and 0.5.0b3 to fix a bug that came to light in porting dobbin over to use it: New objects added to a session did not cause a transaction join, so were not committed at the end of the transaction unless the database was accessed. SQLAlchemy 0.4.7 or 0.5beta3 now required. Other than that nothing really changed since 0.2, you should be able to just use that. Laurence David Pratt wrote: Hi. I have been working with z3c.dobbin 0.4.1 which uses zope.sqlalchemy and z3c.saconfig. The last usable state was 4 days ago when changes were made to release of zope.sqlalchemy where dependencies were changed without changing the version. Revision 88953 was made to zope.sqlalchemy but shows the same package version as 88952 (where SA dependencies were changed to no longer include beta0.5.2). beta0.5.2 was the requirement for z3c.dobbin and it will not work with beta0.5.3. Malthe is currently making changes to dobbin in the interim so it will eventually work again. I thought by checking out revision 88952 and using it as a dev egg in my buildout, I could get back to work. Is this a bug in buildout since dev egg should take precedence regardless? I guess there were some problems with how this was handled. If trunk had been marked as 0.3dev it might have made the difference because both were 0.3 regardless of the change of state and dependencies. When I run the buildout the dev egg's requires.txt is below regardess of what is in setup.py. Doesn't matter what I do my build fails. Any suggestions? Many thanks. Regards, David requires.txt setuptools SQLAlchemy=0.4.7,!=0.5.0beta1,!=0.5.0beta2 transaction zope.interface [test] zope.testing docutils setup.py install_requires=[ 'setuptools', 'SQLAlchemy==0.5.0beta2', # or =0.5b3 'transaction', 'zope.interface', ], ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Fri, May 30, 2008 at 01:55:44PM -0400, Benji York wrote: On Fri, May 30, 2008 at 1:40 PM, Brian Sutherland [EMAIL PROTECTED] wrote: I've just decided to jettison IAbsoluteURL and make a new interface ICanonicalURL. The adapters for ICanonicalURL are available anywhere without specially wrapping the object in a location proxy. IAbsoluteURL doesn't neccesarily require a location proxy, it's just that the default implementations require objects to implement ILocation. No need to define your own interface, just override the IAbsoluteURL implementation(s) you don't like to behave differently (or better yet, don't let them get registered in the first place). One thing I don't like about the default implementation is that if you override IAbsoluteURL for a container, it has no effect for contained objects. I suppose there's a good reason for that -- efficiency -- but it often makes custom IAbsoluteURL adapters pointless. LocationProxy helps then. Marius Gedminas -- Photons have energy, and trying to cram too many into too small of a space can cause a black hole to form, which is, needless to say, not a desirable trait for an optical computer. -- http://scottaaronson.com/blog/?p=261#comment-13693 signature.asc Description: Digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On May 28, 2008, at 7:09 PM, Laurence Rowe wrote: Engine configuration is a subset of session configuration. You cannot have a single ScopedSession for a package if you want to have multiple instances of that application. We must work with unbound metadata if we have this goal. That implies that we must use bound sessions, a session associated with a particular engine (actually it could be more complex than this, you can associate particular classes/tables with particular engines within a session). chiming in, although I haven't read the whole thread I would agree that bound metadata is a little cumbersome in an environment that wants to switch engines within a single process. Bound metadata has always given us a lot of headaches because it's just *so* convenient that we just can't get rid of it, but at the same time people are just so tripped up by it, thinking it's necessary to make anything happen (it's not). Binding sessions allows better control of connection/transaction scope (since the Session relates to transactions, MetaData does not), so it's probabably the way to go in an environment that has a lot of things going on. As far as ScopedSession, it's really just a thread local variable. Like any global threadlocal, you can stick whatever you want on it at the start of a request, and tear it down at the end. Then again it's also reasonable that you might want to have individual ScopedSessions for each application instance within a multi-app process, that way the burden of setting up/tearing down is reduced or removed. A single ScopedSession for a multi-app process is like a one-dimensional approach where both current thread and current app are identified by the current get() of the registry; a collection of ScopedSessions is more like a two-dimensional approach where the first level of registry (i.e. which ScopedSession do I choose) distinguishes between app instance, and the second level (i.e. what Session is bound to this thread ID) distinguishes between threads. All of that said I think it can work either way but I think the latter approach might have the explicitness you're looking for. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Thu, May 29, 2008 at 07:10:16PM +0200, Martijn Faassen wrote: [snip] I'm not sure whether it would be a good idea to wrap this in a session property, or just register it as an adapter. The only other object that would need access to the session (either as a property or through adaption) would be the application instance root object. Something like: @adapter(MyApp) @provider(ISession) def root_session(context): return context._sessioncontext() This looks to me like it could be a simple function that looks up a local utility instead: def session(): return component.getUtility(ISessionContext)() We get the right context from the site that way. I don't see the point in trying to re-implement local utilities with adapters while zope.component already does something quite similar for you. That said, I still have hope we can used ScopedSession and forgo a utility lookup here, see below... +lots Perhaps you may want the utility name as a keyword. I'll also note that doing it this way does not preclude using the ISession adapter in future. But, using the ISession adapter now means never being able to get rid of it again. If you don't use the ZODB at all, you could still set up local sites (I'm not sure how hard this is, but it *should* be possible; I've done it as a work around to override a global utility forced upon me. It's not too difficult. We do still need to setup parent pointers though for grok.url and IAbsoluteURL to work. It looks fairly easy to add location information to the containers themselves: (note that grok.url uses IAbsoluteURL, so we just care about IAbsoluteURL) I'll note that there is a difference between where things are published (IAbsoluteURL) and where you should go to find their canonical representation. One problem I've found with using IAbsoluteURL and location proxies is getting the object directly from the database. In this situation you don't have an LocationProxy and IAbsoluteURL doesn't work. It's pretty nasty when IAbsoluteURL sometimes works and sometimes doesn't for what essentially is the same object. -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Fri, May 30, 2008 at 03:49:46PM +0200, Martijn Faassen wrote: Brian Sutherland wrote: [snip] I'll note that there is a difference between where things are published (IAbsoluteURL) and where you should go to find their canonical representation. One problem I've found with using IAbsoluteURL and location proxies is getting the object directly from the database. In this situation you don't have an LocationProxy and IAbsoluteURL doesn't work. It's pretty nasty when IAbsoluteURL sometimes works and sometimes doesn't for what essentially is the same object. Agreed, we should ideally always get our objects from the db already properly wrapped in a location proxy (or providing ILocation directly). I think we can do this with containers based on relations; those containers are under our control after all. I've just decided to jettison IAbsoluteURL and make a new interface ICanonicalURL. The adapters for ICanonicalURL are available anywhere without specially wrapping the object in a location proxy. Is a direct one on one relation possible in SQLAlchemy? I.e. an object foo that has an attribute 'bar' that is another ORM-ed object. If so we also need to take care of that case somehow. This was a problem with sqlos containers as well. It gets nasty quickly. Say you have 2 containers, of people and addresses. Also people can have an address. So you can get to an address in 2 different ways: /adresses/address1 /people/person1/address1 What should the url for address1 be? I'd vote for /adresses/address1. But due to the LocationProxy, you'll get different results for different IAbsoluteURL calls depending on how you traversed to the object. This is the it should work like ZODB-backed objects pattern I've been trying to follow as a design guideline. Since for contained objects I can always get the URL, it should work that way for RDB-based contained objects too. I'm not sure it's worthwhile following the it should work like ZODB-backed objects pattern. Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Fri, May 30, 2008 at 1:40 PM, Brian Sutherland [EMAIL PROTECTED] wrote: I've just decided to jettison IAbsoluteURL and make a new interface ICanonicalURL. The adapters for ICanonicalURL are available anywhere without specially wrapping the object in a location proxy. IAbsoluteURL doesn't neccesarily require a location proxy, it's just that the default implementations require objects to implement ILocation. No need to define your own interface, just override the IAbsoluteURL implementation(s) you don't like to behave differently (or better yet, don't let them get registered in the first place). -- Benji York Senior Software Engineer Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Fri, May 30, 2008 at 2:35 PM, Martijn Faassen [EMAIL PROTECTED] wrote: I guess one difference here is that it seems more common in the RDB scenario to have multiple ways to reach the same object, and even if you just publish one, you can get to the actual object in multiple ways. In this case it'd be nice to convince IAbsoluteURL to somehow know which URL option is the best, or alternatively, give the object somehow its 'canonical' location (parent, etc) even if you don't get it that way. I have an app that allows the same object to be reachable via several URLs. When we wanted one of the URLs to be preferred (none are really cononical in my case), then we did just as you suggest above and added a way for the IAbsoluteURL adapter(s) to know which URL was preferred. It worked out quite well. -- Benji York Senior Software Engineer Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On May 30, 2008, at 9:44 AM, Martijn Faassen wrote: Hey, Michael Bayer wrote: [snip discussion on BoundMetadata I don't comprehend yet but probably should :)] As far as ScopedSession, it's really just a thread local variable. Like any global threadlocal, you can stick whatever you want on it at the start of a request, and tear it down at the end. Is it really needed to set things up at the start and the request and tear things down on the end, or can a session be retained between requests? I.e. is the intent to recreate a Session class each time a request is issued? in the 0.4 series, the Session is sticky in that whatever is in it retains the state that was last loaded - 0.4 does not go back to the database to re-fetch the state of an object unless its explicitly told to do so. When an HTTP request ends, assuming all objects in the session are marked as clean, they're weakly referenced and will fall out of scope assuming those objects were only referenced by the request. So in that sense, you can just leave the Session alone and it'll work just fine for the next request...but any pending changes in the Session that weren't flushed for some reason would get carried over, as well as anything that might be strongly referenced elsewhere. So we always recommended in 0.4 to at least issue a session.clear() at the end of a request to just empty it out (but you can still reuse that session). Other options included scopedsession.remove() which tears the whole session down, the advantage there being that the new request could configure the next session differently (as in, binding it to something else). in 0.5, Session has been changed to be less reluctant to go and re- fetch data (which is contrary to the particular background I came from, but since then I've learned to see a broader scope of use cases). In 0.5, after a commit() or rollback(), the Session still may be holding on to objects, but their state is expired such that it will all be re-loaded upon access, and all pending and deleted states are reverted. So 0.5's Session, when configured in the default way, makes it very hard to get at stale state, so in that sense you can just re-use a session wihtout worrying much about what may have been left over. Then again it's also reasonable that you might want to have individual ScopedSessions for each application instance within a multi-app process, that way the burden of setting up/tearing down is reduced or removed. This indicates to me is possible to retain a Session object between two requests? I.e. it's a worthwhile goal to reduce the amount of Session configuration going on, right? Its not a strong argument either way to reuse a session or just make a new one on each request. In 0.4, making a brand new session on each request does have a cleaner feel to it since theres no chance of any state left hanging around. Its not an expensive operation. A single ScopedSession for a multi-app process is like a one- dimensional approach where both current thread and current app are identified by the current get() of the registry; a collection of ScopedSessions is more like a two-dimensional approach where the first level of registry (i.e. which ScopedSession do I choose) distinguishes between app instance, and the second level (i.e. what Session is bound to this thread ID) distinguishes between threads. All of that said I think it can work either way but I think the latter approach might have the explicitness you're looking for. I'm trying to understand why you think so. I am looking for a way to let developers use SQLAlchemy in a straightforward way. They should be able to import 'Session', instantiate session in their app, and everything works as expected. The framework takes care of making you get the appropriately configured Session. they can in fact do this without any issue.The question is, when someone writes the call s = Session() three times within one request, do you want each s to all reference the *same* set of objects ? that was the issue scoped_session() was meant to solve. Configuration is easy since you just configure a Session callable for that application. But you guys have this issue of multiple applications in the same process at play, which each talk to a different database. So thats where a decision has to be made how to deal with that complexity, either configure engine on one scoped_session per request, or configure multiple scoped_sessions, one per engine. The latter approach seems easier to me since you don't actually have to do anything on a per-request basis assuming 0.5 usage. Neither of these make any difference to the end user who sees the exact same usage pattern. (I'd still absolutely like to avoid passing in context explicitly each time you need a session; it puts unnecessary burden on the developer and
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Sat, May 24, 2008 at 09:30:18PM +0100, Laurence Rowe wrote: Brian Sutherland wrote: On Fri, May 23, 2008 at 11:39:39PM +0100, Laurence Rowe wrote: We need to differentiate between the interface for session configuration and session usage from an application. For session usage I think it is fairly simple. We should define an ISessionContext interface such that: class ISessionContext(Interface): def __call__(): return a session in this context +lots (I was thinking about proposing an interface called ISessionMaker doing much the same thing) I'm not sure what in this context means? The context of this application instance. Say you have two instances of an application, one pointed at database A, another at database B. It is possible to involve both applications in a single request / transaction. You must be sure that you are working with the correct session. Are the instances you are talking about persistent objects in the ZODB? A future version of collective.lead could implement an ISessionContext. Client code however should have a cleaner interface, a plain ISession. This is accessed through a lookup on the context, translated into a simple adapter: def session_adapter(context): session_context = queryUtility(ISessionContext, context=context, Why call queryUtility with the context keyword? default=None) if session_context is not None: return session_context() This will allow application code to do something like: session = ISession(context) ob = session.query(MyClass) This really confuses me. What is the context? Does it have any meaning? Or is it just a shorter way to write: session = getUtility(ISessionContext)() Does the value of context have an effect on what you get from the ISession adaptation? Yes, as it translates directly to the getUtility lookup. It ensures that lookups in /appA go to a local utility defined in /appA and lookups in /appB go to a local utility defined in /appB. I've been burned by using context as a keyword to getUtility before. When your context doesn't have a proper __parent__ pointer, the default IComponentLookup adapter falls back to the global site manager and ignores the local one. That causes no end of confusion and hard to debug bugs as people will call ISession on objects with and without __parent__ pointers and then wonder why it fails. Perhaps a more robust way is to rely on the site stored while traversing in zope.app.component.hooks? For example: * appA implements ISite and has a local ISessionContext utility * The path appA is traversed over (i.e. the url path is /appA/foo) * queryUtility(ISessionContext) then returns the local utility from appA without the need to specify a context (thanks to the magic in zope.app.component.hooks) I think the only situation where you really want to specify a context for queryUtility is when you want to access a local utility registered in /appB from a URL path like /appA/foo. That seems pretty rare and almost broken. You could implement a function like: def query_session(name=u''): return queryUtility(ISessionContext, name=name)() That does the right thing in almost all cases (i.e. uses the closest local site), and is much more robust. Of course it would be possible to register a ScopedSession globally as such a utility, but more usually a local utility should be registered. Depends what you're doing. If you are running without a ZODB, you have mostly just global utilities. It would be a pity if zope.sqlalchemy started to depend on the ZODB. Wihout ZODB and zope.app.component it seems unlikely that you would be able to register a local utility, the idea of ISessionContext is so that you might be able to register a ScopedSession as a global utility too. I very much like the idea of ISessionContext! I'm just not sure about how you suggest using it in client code. -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On Fri, May 23, 2008 at 11:39:39PM +0100, Laurence Rowe wrote: We need to differentiate between the interface for session configuration and session usage from an application. For session usage I think it is fairly simple. We should define an ISessionContext interface such that: class ISessionContext(Interface): def __call__(): return a session in this context +lots (I was thinking about proposing an interface called ISessionMaker doing much the same thing) I'm not sure what in this context means? A future version of collective.lead could implement an ISessionContext. Client code however should have a cleaner interface, a plain ISession. This is accessed through a lookup on the context, translated into a simple adapter: def session_adapter(context): session_context = queryUtility(ISessionContext, context=context, Why call queryUtility with the context keyword? default=None) if session_context is not None: return session_context() This will allow application code to do something like: session = ISession(context) ob = session.query(MyClass) This really confuses me. What is the context? Does it have any meaning? Or is it just a shorter way to write: session = getUtility(ISessionContext)() Does the value of context have an effect on what you get from the ISession adaptation? Of course it would be possible to register a ScopedSession globally as such a utility, but more usually a local utility should be registered. Depends what you're doing. If you are running without a ZODB, you have mostly just global utilities. It would be a pity if zope.sqlalchemy started to depend on the ZODB. (I haven't though about the consequences of this in pre-traversal, before the site and local utilities are set up) session.remove() is not important, sessions are closed by the zope.sqlalchemy datamanager and closed sessions are recyclable. Presumably the session object would be referred to by a volatile attribute on the local utility and the session would be GC'd along with the local utility object itself. Table creation is another matter that I don't think too important. Implicit creation of tables seems wrong, instead tables should only be created explicitly, by the use clicking a button in the zope web interface (or automatically on adding an application). An exception to this is sqlalchemy in memory databases, which must be created on first access. Session configuration would be somewhat similar to collective.lead currently (registering one as a local utility). Laurence Martijn Faassen wrote: Hi there, Today I had a discussion with Jasper Spaans about how to go about improving megrok.rdb, Grok's relational database integration which aims to integrate Grok with SQLAlchemy. We started developing megrok.rdb at the Grokkerdam sprint a few weeks ago. We reread the discussion surrounding zope.sqlalchemy for ideas on how to go about integration and configuration. I think these discussions reach wider than just Grok's concerns. Note that I'm not proposing we fold any of these ideas into zope.sqlalchemy itself, which should remain as free of policy as possible; it could become (yet another) extension. Let me run our current thinking by the list. What would be nice in Zope applications (and we think would be good for Grok) would be per-instance database configuration. That is, we want to be able to install multiple instances of the same application and then configure each of them with a different database URN and it should all work, each talking to their own database. Michael Bayer's suggestion involves the use of scoped sessions. He proposed the following code: Session = scoped_session() # start of request engine = get_appropriate_engine() Session(bind=engine) try: # do request finally: Session.remove() Let's go through the steps. First it makes a scoped session object, it then configures it with the right engine at the start of the request (it can do this on a per-class level), and then at the end of the request it removes the Session again, which results in the actual session being closed. Our get_appropriate_engine() would probably look the engine up as a local utility, as Laurence suggested. There is a bit of question about engine configuration, though. If we want to support the use case of looking up the engine URL in a persistent datastore (for instance one URL per location), we have a question of ordering. We cannot do it too early; at the start of the transaction there isn't a ZODB yet to talk to so we can't look up a local utility. We can try doing it just in time: _Session = scoped_session() def Session(*args, **kw): engine = get_appropriate_engine() _Session.bind(bind=engine) return _Session(*args, **kw) Here get_appropriate_engine() could do a component.getUtility() and look up the engine for us, possibly in an
Re: [Zope-dev] Re: zope.sqlalchemy
--On 9. Mai 2008 11:20:31 +0100 Laurence Rowe [EMAIL PROTECTED] wrote: You need an svn checkout of SQLAlchemy until 0.4.6 is released. Laurence Great, that worked and made all tests pass. Andreas pgpEracgwI4qR.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy
On May 7, 2008, at 7:08 AM, Martijn Faassen wrote: Hi there (especially Christian), I think we can work with explicits saves. In many cases the user won't have to worry about it anyway as the container object will do it for them (besides making the relation), or this 'query container' we spoke of will do it for them (but just the 'save' bit). One point is that the scoped session approach itself doesn't work very well for using multiple databases in the same app. We could consider passing the session along in the containers during object graph wakling (or traversal) so an app can easily traverse into multiple databases. I'm not sure whether we can make the ORM do this for us though; does it initialize the mapping with a session? SQLAlchemy's Session does support multiple engine binds itself, which most easily can be associated with particular mapped classes (i.e. vertical partitioning), so that a single session (or a scoped_session) can read and write data to the appropriate tables transparently (although things like joins across multiple databases will raise errors). Theres a horizontally-partitioning version of Session as well which obviously has a lot more caveats. Using multiple sessions, one per DB is a valid approach as well. I'm not sure if Grok has other things going on when mulitple DBs are in use but SA's multi-bind capability is something to be aware of. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy
On May 7, 2008, at 1:29 PM, Laurence Rowe wrote: I'm thinking more about having the same classes mapped to different databases at different points in the application. Imagine a departmental address book app. Intstances of the departmental address book are created for each department, each with a different databases: http://addressbook/sales - postgres:///sales http://addressbook/engineering - postgres:///engineering The way I imagine this working is to have a proxy engine object that looks up the real engine through a local utility. Each application would be a `site` and capable of local utility registrations. /sales would have Engine('postgres:///sales') registered and /engineering Engine('postgres:///engineering'). Only a single ScopedSession would be required. This would be bound to proxy that performs the utility lookup. So when in the /sales context the proxy would point to the sales engine and when in the / engineering context to the engineering engine. The limitation of this approach is that it would not be possible to mix objects from /sales and objects from /engineering into the same transaction. So really we need a session per application instance. Perhaps this can be achieved through a custom scoping function: def scopefunc(): return thread.get_ident(), id(zope.component.getSiteManager()) If you want to mix tables (and optionally engines) for the *same* class, we actually have a feature for that too. Its sort of a feature I've wanted to remove but Jason keeps arguing that its worthy. It's called entity_name and it allows multiple primary mappers to be created for a single class. The entity_name has to be specified when you add the element to the session (yet another reason explicit add() is a good thing). This feature is taken directly from the Hibernate feature of the same name. The limitation with entity_name which needs some more fixing in 0.5 is that only one mapper gets to define the attribute instrumentation for the entity. If you are storing the same class in three different tables (across three different databases or just one), the attributes defined on the class need to be compatible with all three. This is reasonable since a class can only have one descriptor per attribute name. Querying is also slightly challenging since the descriptors need to be qualified for a particular mapper (i.e. you cant just say query.filter(Address.id==5)...which id are we talking about ?) The reason I'm not totally keen on this feature is that it seems to be a very exotic way of getting around making simple subclasses, and I've yet to see the use case for it where simple subclasses don't work, except for cosmetic reasons which I have a hard time swallowing (even if the reasons are cosmetic, you can still create subclasses that are all named the same). So I will ask you, why can't your application simply have a SalesAddress and an EngineeringAddress class ? You could even produce them transparently using a custom __new__() method, i.e. class Address(object): def __new__(cls, *args, **kwargs): if my_scoped_thing.context == 'sales': return object.__new__(SalesAddress) elif my_scoped_thing.context == 'engineering': return object.__new__(EngineeringAddress) this seems extremely straightforward to me as each object, once instantiated is now bound for a specific destination. It doesnt seem like youd want the *same* Address to be stored in one and then the other in a different instance (that seems *extremely* complex for no good reason). Isnt explicit better than implicit ? ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy
On May 7, 2008, at 2:29 PM, Laurence Rowe wrote: When the generic address book application is built you don't know what the departments will be called or indeed how many departments there are. An address book is not be a great example, but I know of intranet portal sites where this is a requirement. You want to delegate control to each department so you give each department their own instance of the portal. You only want to maintain one code base though, and you don't want to change it every time someone adds another departmental portal. I'd like to be able to create an add form that has fields for application name and database url. This probably seems like an odd requirement -- why not just run multiple processes with different configurations -- but it's the way zope has traditionally worked. A single process can serve multiple instances of the same application (or `site`). When you get up to running tens of sites, the memory footprint of Zope2 and Plone (before the object cache) becomes significant. If you are running different instances each connected to a different engine within one process you wouldn't need any awareness of engines at the object level (therefore no entity_name) and also no engine proxying - you should have separate Session instances for each process managed by scoped_session(), which was designed to handle this.Multiple apps on one codebase within one process was an original requirement of Pylons as well, though nobody has ever used it. The easiest way to do it is to set up the engine at the request level: Session = scoped_session() # start of request engine = get_appropriate_engine() Session(bind=engine) try: # do request finally: Session.remove() If that isnt granular enough, then you use a custom scope func which maintains Session per-process-per-thread. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy
On May 6, 2008, at 12:14 PM, Laurence Rowe wrote: Martijn Faassen wrote: One thing I understood from Christian Theune is that with scoped sessions, explicit session.save() is not always necessary. Since I see it being used here, could you perhaps comment on this? Registering a mapper with Session.mapper would work with this extension, but I'm reluctant to recommend it for two reasons: I don't know how it works with the declarative plugin; and it necessarily limits mapped classes to a single Session and therefor a single engine. In a zope context I think it's quite likely that you could have the same classes mapped to different databases (i.e. two instances of a single application). hi there - a little background on the save_on_init option of Session.mapper. This behavior has its roots way back in SQLAlchemy 0.1, when there was no Session or Query or anything like that, and objects, when instantiated, went directly to a thread-local registry automatically. When SQLA 0.2 came out, we introduced all the additional constructs like Session and such which are familiar today, but extensions were provided which, when enabled, would re-enable the 0.1 behavior of everything threadlocal/automatic in a similar way. Ultimately thats where Session.mapper comes from. Like all typing-savers, save on init by then was used by dozens of Pylons users who swore by it and would scream and yell at any hint of removing this already legacy feature. At the same time, new users who were using Pylons tutorials (and therefore save_on_init, without really knowing it) in conjunction with creating their own Session objects were baffled by error messages like object X is already in session Y. By the time 0.4 came out, we had started automating Session a lot more, adding autoflush capability to it. This feature immediately had issues with save_on_init for this reason: class MyClass(object): def __init__(self): self.some_variable = session.query(Foobar).filter(xyz).one() Where above, the query(Foobar) would fire off autoflush, MyClass would be flushed, and then an IntegrityError would be raised since MyClass would be missing some necessary state. Changing save_on_init to fire off *after* __init__ was a possibility there but then other things could break. So I've already not liked save_on_init for a couple of years due to its inherent intrusiveness, and because SA historically does not like being in the business of providing framework features (though we have decided to stay in that arena to some degree with declarative and scoped_session). The Session.mapper feature is stressed a whole lot less in the 0.4 docs, and as I work on the 0.5 docs this week I'm feeling very much like I'm going to remove it from the main documentation altogether. We''re consolidating the save/update/save_or_update names into just add() and add_all(), so explicitly adding items to a Session should be a more pleasant experience which I wouldn't want anyone to miss. The aspect of Session.mapper which is somewhat reduntant vs. declarative is that they both want to add an automatic __init__(**kwargs) method which assigns all given keyword values to the instance.They are not incompatible because Session.mapper only adds an __init__ if none is available already. The final feature of Session.mapper which is more reasonable is the query attribute. This feature allows you to say: MyClass.query as an equivalent for session.query(MyClass). For that specific attribute, instead of using Session.mapper, its functionality has been exported into its own descriptor-producing method, like so: class MyBaseClass(object): query = Session.query_property() So this is a way to get that one aspect without buying into the Session.mapper thing. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy
Hi, On Tue, May 06, 2008 at 12:53:05PM -0400, Michael Bayer wrote: On May 6, 2008, at 12:14 PM, Laurence Rowe wrote: Martijn Faassen wrote: One thing I understood from Christian Theune is that with scoped sessions, explicit session.save() is not always necessary. Since I see it being used here, could you perhaps comment on this? Registering a mapper with Session.mapper would work with this extension, but I'm reluctant to recommend it for two reasons: I don't know how it works with the declarative plugin; and it necessarily limits mapped classes to a single Session and therefor a single engine. In a zope context I think it's quite likely that you could have the same classes mapped to different databases (i.e. two instances of a single application). hi there - a little background on the save_on_init option of Session.mapper. This behavior has its roots way back in SQLAlchemy 0.1, when there was no Session or Query or anything like that, and objects, when instantiated, went directly to a thread-local registry automatically. When SQLA 0.2 came out, we introduced all the additional constructs like Session and such which are familiar today, but extensions were provided which, when enabled, would re-enable the 0.1 behavior of everything threadlocal/automatic in a similar way. Ultimately thats where Session.mapper comes from. That's interesting, thanks for the heads-up. For some reason I didn't hit that and was quite happy with save on init (I obviously only use one database at a time ...) but the reasons and concerns given tell me that our core transaction integration shouldn't force this onto people and we might not want to use it for grok at all. In fact at the sprint we discussed the similarities and differences of RDB versus ODB and found that the step of 'add an object to the db' is actually two functions (in one gesture) in the ODB: relate object B to object A and, therefore add it to the same database as object A is in. Whereas (due to the mapping of classes to tables) in RDB we only need to tell which database to go to. Those are similar but differen gestures and I'm not sure we had a definitive result when discussing this. Christian -- gocept gmbh co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy
On May 6, 2008, at 5:24 PM, Martijn Faassen wrote: Hey Michael, Thanks for the input! Michael Bayer wrote: [snip] So I've already not liked save_on_init for a couple of years due to its inherent intrusiveness, and because SA historically does not like being in the business of providing framework features (though we have decided to stay in that arena to some degree with declarative and scoped_session). I'll try to summarize the discussion so I can find out whether I understand it. Basically you're saying you don't think save on instantiation is a good idea generally, and that we should be using session.save(). This is going to be changed to session.add() in the future. What would session.add_all() do? session.add_all() is just: session.add_all([obj1, obj2, obj3, ...]) Also session.save()/update/save_or_update will remain throughout 0.5 at least. This ties into the mapper feature, which also offers other features. The one feature that will remain but in a new shape, without the mapper, is the ability to do MyClass.query. Is that a correct summary? Session.mapper and save_on_init has no plans of going away in 0.5, but I plan to de-emphasize it.The query descriptor function is also available in 0.4. - mike ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )