Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas

2008-06-02 Thread Marius Gedminas
On Fri, May 30, 2008 at 01:55:44PM -0400, Benji York wrote:
 On Fri, May 30, 2008 at 1:40 PM, Brian Sutherland
 [EMAIL PROTECTED] wrote:
  I've just decided to jettison IAbsoluteURL and make a new interface
  ICanonicalURL. The adapters for ICanonicalURL are available anywhere
  without specially wrapping the object in a location proxy.
 
 IAbsoluteURL doesn't neccesarily require a location proxy, it's just that the
 default implementations require objects to implement ILocation.

 No need to define your own interface, just override the IAbsoluteURL
 implementation(s) you don't like to behave differently (or better yet, don't
 let them get registered in the first place).

One thing I don't like about the default implementation is that if you
override IAbsoluteURL for a container, it has no effect for contained
objects.  I suppose there's a good reason for that -- efficiency -- but
it often makes custom IAbsoluteURL adapters pointless.  LocationProxy
helps then.

Marius Gedminas
-- 
Photons have energy, and trying to cram too many into too small of a space can
cause a black hole to form, which is, needless to say, not a desirable trait
for an optical computer.
-- http://scottaaronson.com/blog/?p=261#comment-13693


signature.asc
Description: Digital signature
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas

2008-05-30 Thread Michael Bayer


On May 28, 2008, at 7:09 PM, Laurence Rowe wrote:



Engine configuration is a subset of session configuration. You  
cannot have a single ScopedSession for a package if you want to have  
multiple instances of that application. We must work with unbound  
metadata if we have this goal. That implies that we must use bound  
sessions, a session associated with a particular engine (actually it  
could be more complex than this, you can associate particular  
classes/tables with particular engines within a session).


chiming in, although I haven't read the whole thread

I would agree that bound metadata is a little cumbersome in an  
environment that wants to switch engines within a single process. 
Bound metadata has always given us a lot of headaches because it's  
just *so* convenient that we just can't get rid of it, but at the same  
time people are just so tripped up by it, thinking it's necessary to  
make anything happen (it's not).  Binding sessions allows better  
control of connection/transaction scope (since the Session relates to  
transactions, MetaData does not), so it's probabably the way to go in  
an environment that has a lot of things going on.


As far as ScopedSession, it's really just a thread local variable.   
Like any global threadlocal, you can stick whatever you want on it at  
the start of a request, and tear it down at the end.   Then again it's  
also reasonable that you might want to have individual ScopedSessions  
for each application instance within a multi-app process, that way the  
burden of setting up/tearing down is reduced or removed.   A single  
ScopedSession for a multi-app process is like a one-dimensional  
approach where both current thread and current app are identified by  
the current get() of the registry; a collection of ScopedSessions is  
more like a two-dimensional approach where the first level of registry  
(i.e. which ScopedSession do I choose) distinguishes between app  
instance, and the second level (i.e. what Session is bound to this  
thread ID) distinguishes between threads.


All of that said I think it can work either way but I think the latter  
approach might have the explicitness you're looking for.

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas

2008-05-30 Thread Brian Sutherland
On Thu, May 29, 2008 at 07:10:16PM +0200, Martijn Faassen wrote:
 [snip]
  I'm not sure whether it would be a good idea to wrap this in a session
  property, or just register it as an adapter. The only other object that
  would need access to the session (either as a property or through
  adaption) would be the application instance root object. Something like:
 
  @adapter(MyApp)
  @provider(ISession)
  def root_session(context):
  return context._sessioncontext()

 This looks to me like it could be a simple function that looks up a local 
 utility instead:

 def session():
return component.getUtility(ISessionContext)()
 
 We get the right context from the site that way. I don't see the point in 
 trying to re-implement local utilities with adapters while zope.component 
 already does something quite similar for you. That said, I still have hope 
 we can used ScopedSession and forgo a utility lookup here, see below...

+lots

Perhaps you may want the utility name as a keyword. I'll also note that
doing it this way does not preclude using the ISession adapter in
future. But, using the ISession adapter now means never being able to
get rid of it again.

 If you don't use the ZODB at all, you could still set up local sites (I'm 
 not sure how hard this is, but it *should* be possible; 

I've done it as a work around to override a global utility forced upon
me. It's not too difficult.

  We do still need to setup parent pointers though for grok.url and
  IAbsoluteURL to work. It looks fairly easy to add location information
  to the containers themselves:

 (note that grok.url uses IAbsoluteURL, so we just care about IAbsoluteURL)

I'll note that there is a difference between where things are published
(IAbsoluteURL) and where you should go to find their canonical
representation.

One problem I've found with using IAbsoluteURL and location proxies is
getting the object directly from the database. In this situation you
don't have an LocationProxy and IAbsoluteURL doesn't work.

It's pretty nasty when IAbsoluteURL sometimes works and sometimes
doesn't for what essentially is the same object.

-- 
Brian Sutherland
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas

2008-05-30 Thread Brian Sutherland
On Fri, May 30, 2008 at 03:49:46PM +0200, Martijn Faassen wrote:
 Brian Sutherland wrote:
 [snip]
 I'll note that there is a difference between where things are published
 (IAbsoluteURL) and where you should go to find their canonical
 representation.

 One problem I've found with using IAbsoluteURL and location proxies is
 getting the object directly from the database. In this situation you
 don't have an LocationProxy and IAbsoluteURL doesn't work.

 It's pretty nasty when IAbsoluteURL sometimes works and sometimes
 doesn't for what essentially is the same object.

 Agreed, we should ideally always get our objects from the db already 
 properly wrapped in a location proxy (or providing ILocation directly). I 
 think we can do this with containers based on relations; those containers 
 are under our control after all.

I've just decided to jettison IAbsoluteURL and make a new interface
ICanonicalURL. The adapters for ICanonicalURL are available anywhere
without specially wrapping the object in a location proxy.

 Is a direct one on one relation possible in SQLAlchemy? I.e. an object foo 
 that has an attribute 'bar' that is another ORM-ed object. If so we also 
 need to take care of that case somehow.

This was a problem with sqlos containers as well. It gets nasty quickly.
Say you have 2 containers, of people and addresses. Also people can have
an address.

So you can get to an address in 2 different ways:

/adresses/address1
/people/person1/address1

What should the url for address1 be? I'd vote for /adresses/address1.
But due to the LocationProxy, you'll get different results for different
IAbsoluteURL calls depending on how you traversed to the object.

 This is the it should work like ZODB-backed objects pattern I've been 
 trying to follow as a design guideline. Since for contained objects I can 
 always get the URL, it should work that way for RDB-based contained objects 
 too.

I'm not sure it's worthwhile following the it should work like
ZODB-backed objects pattern.


 Regards,

 Martijn

 ___
 Zope-Dev maillist  -  Zope-Dev@zope.org
 http://mail.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

-- 
Brian Sutherland
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas

2008-05-30 Thread Benji York
On Fri, May 30, 2008 at 1:40 PM, Brian Sutherland
[EMAIL PROTECTED] wrote:
 I've just decided to jettison IAbsoluteURL and make a new interface
 ICanonicalURL. The adapters for ICanonicalURL are available anywhere
 without specially wrapping the object in a location proxy.

IAbsoluteURL doesn't neccesarily require a location proxy, it's just that the
default implementations require objects to implement ILocation.

No need to define your own interface, just override the IAbsoluteURL
implementation(s) you don't like to behave differently (or better yet, don't
let them get registered in the first place).
-- 
Benji York
Senior Software Engineer
Zope Corporation
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas

2008-05-30 Thread Benji York
On Fri, May 30, 2008 at 2:35 PM, Martijn Faassen [EMAIL PROTECTED] wrote:
 I guess one difference here is that it seems more common in the RDB scenario
 to have multiple ways to reach the same object, and even if you just publish
 one, you can get to the actual object in multiple ways.

 In this case it'd be nice to convince IAbsoluteURL to somehow know which URL
 option is the best, or alternatively, give the object somehow its
 'canonical' location (parent, etc) even if you don't get it that way.

I have an app that allows the same object to be reachable via several
URLs.  When we wanted one of the URLs to be preferred (none are really
cononical in my case), then we did just as you suggest above and added
a way for the IAbsoluteURL adapter(s) to know which URL was preferred.
It worked out quite well.
-- 
Benji York
Senior Software Engineer
Zope Corporation
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas

2008-05-30 Thread Michael Bayer


On May 30, 2008, at 9:44 AM, Martijn Faassen wrote:


Hey,

Michael Bayer wrote:

[snip discussion on BoundMetadata I don't comprehend yet but  
probably should :)]


As far as ScopedSession, it's really just a thread local variable.   
Like any global threadlocal, you can stick whatever you want on it  
at the start of a request, and tear it down at the end.


Is it really needed to set things up at the start and the request  
and tear things down on the end, or can a session be retained  
between requests? I.e. is the intent to recreate a Session class  
each time a request is issued?


in the 0.4 series, the Session is sticky in that whatever is in it  
retains the state that was last loaded - 0.4 does not go back to the  
database to re-fetch the state of an object unless its explicitly told  
to do so.   When an HTTP request ends, assuming all objects in the  
session are marked as clean, they're weakly referenced and will fall  
out of scope assuming those objects were only referenced by the  
request.  So in that sense, you can just leave the Session alone and  
it'll work just fine for the next request...but any pending changes in  
the Session that weren't flushed for some reason would get carried  
over, as well as anything that might be strongly referenced  
elsewhere.  So we always recommended in 0.4 to at least issue a  
session.clear() at the end of a request to just empty it out (but you  
can still reuse that session).  Other options included  
scopedsession.remove() which tears the whole session down, the  
advantage there being that the new request could configure the next  
session differently (as in, binding it to something else).


in 0.5, Session has been changed to be less reluctant to go and re- 
fetch data (which is contrary to the particular background I came  
from, but since then I've learned to see a broader scope of use  
cases).   In 0.5, after a commit() or rollback(), the Session still  
may be holding on to objects, but their state is expired such that it  
will all be re-loaded upon access, and all pending and deleted  
states are reverted.  So 0.5's Session, when configured in the default  
way, makes it very hard to get at stale state, so in that sense you  
can just re-use a session wihtout worrying much about what may have  
been left over.



Then again it's also reasonable that you might want to have  
individual ScopedSessions for each application instance within a  
multi-app process, that way the burden of setting up/tearing down  
is reduced or removed.


This indicates to me is possible to retain a Session object between  
two requests? I.e. it's a worthwhile goal to reduce the amount of  
Session configuration going on, right?


Its not a strong argument either way to reuse a session or just make a  
new one on each request.  In 0.4, making a brand new session on each  
request does have a cleaner feel to it since theres no chance of any  
state left hanging around.  Its not an expensive operation.


A single ScopedSession for a multi-app process is like a one- 
dimensional approach where both current thread and current app are  
identified by the current get() of the registry; a collection of  
ScopedSessions is more like a two-dimensional approach where the  
first level of registry (i.e. which ScopedSession do I choose)  
distinguishes between app instance, and the second level (i.e. what  
Session is bound to this thread ID) distinguishes between threads.
All of that said I think it can work either way but I think the  
latter approach might have the explicitness you're looking for.


I'm trying to understand why you think so. I am looking for a way to  
let developers use SQLAlchemy in a straightforward way. They should  
be able to import 'Session', instantiate session in their app, and  
everything works as expected. The framework takes care of making you  
get the appropriately configured Session.


they can in fact do this without any issue.The question is, when  
someone writes the call s = Session() three times within one  
request, do you want each s to all reference the *same* set of  
objects ?  that was the issue scoped_session() was meant to solve.   
Configuration is easy since you just configure a Session callable  
for that application.


But you guys have this issue of multiple applications in the same  
process at play, which each talk to a different database.  So thats  
where a decision has to be made how to deal with that complexity,  
either configure engine on one scoped_session per request, or  
configure multiple scoped_sessions, one per engine.   The latter  
approach seems easier to me since you don't actually have to do  
anything on a per-request basis assuming 0.5 usage.   Neither of these  
make any difference to the end user who sees the exact same usage  
pattern.


(I'd still absolutely like to avoid passing in context explicitly  
each time you need a session; it puts unnecessary burden on the  
developer and 

Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas

2008-05-30 Thread Benji York
On Fri, May 30, 2008 at 3:03 PM, Martijn Faassen [EMAIL PROTECTED] wrote:
 Benji York wrote:
 I have an app that allows the same object to be reachable via several
 URLs.  When we wanted one of the URLs to be preferred (none are really
 cononical in my case), then we did just as you suggest above and added
 a way for the IAbsoluteURL adapter(s) to know which URL was preferred.
 It worked out quite well.

 Cool. Where does the adapter get the information to make a decision from? I
 mean, how does it know to use URL A and not URL B?

Nothing fancy: we adapt the object to (names changed to clarify the
example) IPreferredLocations to get back a list ordered by how preferred
the locations are (from most-preferred to least), we then simply iterate
over the list (it's always short) until we find the first place the
object is visible, and use it.

The location ordering is specified by the user (more or less, there are details
I'm leaving out).
-- 
Benji York
Senior Software Engineer
Zope Corporation
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas

2008-05-25 Thread Brian Sutherland
On Sat, May 24, 2008 at 09:30:18PM +0100, Laurence Rowe wrote:
 Brian Sutherland wrote:
 On Fri, May 23, 2008 at 11:39:39PM +0100, Laurence Rowe wrote:
 We need to differentiate between the interface for session configuration 
 and session usage from an application.

 For session usage I think it is fairly simple. We should define an 
 ISessionContext interface such that:

 class ISessionContext(Interface):
 def __call__():
 return a session in this context

 +lots

 (I was thinking about proposing an interface called ISessionMaker doing
 much the same thing)

 I'm not sure what in this context means?

 The context of this application instance. Say you have two instances of an 
 application, one pointed at database A, another at database B. It is 
 possible to involve both applications in a single request / transaction. 
 You must be sure that you are working with the correct session.

Are the instances you are talking about persistent objects in the ZODB?

 A future version of collective.lead could implement an ISessionContext. 
 Client code however should have a cleaner interface, a plain ISession. 
 This is accessed through a lookup on the context, translated into a 
 simple adapter:

 def session_adapter(context):
 session_context = queryUtility(ISessionContext, context=context, 

 Why call queryUtility with the context keyword?

 default=None)
 if session_context is not None:
 return session_context()

 This will allow application code to do something like:

 session = ISession(context)
 ob = session.query(MyClass)

 This really confuses me. What is the context? Does it have any meaning?
 Or is it just a shorter way to write:

 session = getUtility(ISessionContext)()

 Does the value of context have an effect on what you get from the
 ISession adaptation?

 Yes, as it translates directly to the getUtility lookup. It ensures that 
 lookups in /appA go to a local utility defined in /appA and lookups in 
 /appB go to a local utility defined in /appB.

I've been burned by using context as a keyword to getUtility before.
When your context doesn't have a proper __parent__ pointer, the default
IComponentLookup adapter falls back to the global site manager and
ignores the local one. That causes no end of confusion and hard to debug
bugs as people will call ISession on objects with and without __parent__
pointers and then wonder why it fails.

Perhaps a more robust way is to rely on the site stored while traversing
in zope.app.component.hooks?

For example:
* appA implements ISite and has a local ISessionContext utility
* The path appA is traversed over (i.e. the url path is /appA/foo)
* queryUtility(ISessionContext) then returns the local utility from
  appA without the need to specify a context
  (thanks to the magic in zope.app.component.hooks)

I think the only situation where you really want to specify a context
for queryUtility is when you want to access a local utility registered
in /appB from a URL path like /appA/foo. That seems pretty rare and
almost broken.

You could implement a function like:

def query_session(name=u''):
return queryUtility(ISessionContext, name=name)()

That does the right thing in almost all cases (i.e. uses the closest
local site), and is much more robust.


 Of course it would be possible to register a ScopedSession globally as 
 such a utility, but more usually a local utility should be registered.

 Depends what you're doing. If you are running without a ZODB, you have
 mostly just global utilities.

 It would be a pity if zope.sqlalchemy started to depend on the ZODB.

 Wihout ZODB and zope.app.component it seems unlikely that you would be able 
 to register a local utility, the idea of ISessionContext is so that you 
 might be able to register a ScopedSession as a global utility too.

I very much like the idea of ISessionContext!

I'm just not sure about how you suggest using it in client code.

-- 
Brian Sutherland
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas

2008-05-24 Thread Brian Sutherland
On Fri, May 23, 2008 at 11:39:39PM +0100, Laurence Rowe wrote:
 We need to differentiate between the interface for session configuration 
 and session usage from an application.

 For session usage I think it is fairly simple. We should define an 
 ISessionContext interface such that:

 class ISessionContext(Interface):
 def __call__():
 return a session in this context

+lots

(I was thinking about proposing an interface called ISessionMaker doing
much the same thing)

I'm not sure what in this context means?

 A future version of collective.lead could implement an ISessionContext. 
 Client code however should have a cleaner interface, a plain ISession. This 
 is accessed through a lookup on the context, translated into a simple 
 adapter:

 def session_adapter(context):
 session_context = queryUtility(ISessionContext, context=context, 

Why call queryUtility with the context keyword?

 default=None)
 if session_context is not None:
 return session_context()

 This will allow application code to do something like:

 session = ISession(context)
 ob = session.query(MyClass)

This really confuses me. What is the context? Does it have any meaning?
Or is it just a shorter way to write:

session = getUtility(ISessionContext)()

Does the value of context have an effect on what you get from the
ISession adaptation?

 Of course it would be possible to register a ScopedSession globally as such 
 a utility, but more usually a local utility should be registered.

Depends what you're doing. If you are running without a ZODB, you have
mostly just global utilities.

It would be a pity if zope.sqlalchemy started to depend on the ZODB.

 (I haven't though about the consequences of this in pre-traversal, before 
 the site and local utilities are set up)

 session.remove() is not important, sessions are closed by the 
 zope.sqlalchemy datamanager and closed sessions are recyclable. Presumably 
 the session object would be referred to by a volatile attribute on the 
 local utility and the session would be GC'd along with the local utility 
 object itself.

 Table creation is another matter that I don't think too important. Implicit 
 creation of tables seems wrong, instead tables should only be created 
 explicitly, by the use clicking a button in the zope web interface (or 
 automatically on adding an application). An exception to this is sqlalchemy 
 in memory databases, which must be created on first access.
 
 Session configuration would be somewhat similar to collective.lead 
 currently (registering one as a local utility).

 Laurence

 Martijn Faassen wrote:
 Hi there,

 Today I had a discussion with Jasper Spaans about how to go about 
 improving megrok.rdb, Grok's relational database integration which aims to 
 integrate Grok with SQLAlchemy. We started developing megrok.rdb at the 
 Grokkerdam sprint a few weeks ago. We reread the discussion surrounding 
 zope.sqlalchemy for ideas on how to go about integration and 
 configuration. I think these discussions reach wider than just Grok's 
 concerns. Note that I'm not proposing we fold any of these ideas into 
 zope.sqlalchemy itself, which should remain as free of policy as possible; 
 it could become (yet another) extension.

 Let me run our current thinking by the list.

 What would be nice in Zope applications (and we think would be good for 
 Grok) would be per-instance database configuration. That is, we want to be 
 able to install multiple instances of the same application and then 
 configure each of them with a different database URN and it should all 
 work, each talking to their own database.

 Michael Bayer's suggestion involves the use of scoped sessions. He 
 proposed the following code:

 Session = scoped_session()

 # start of request
 engine = get_appropriate_engine()
 Session(bind=engine)
 try:
 # do request
 finally:
 Session.remove()

 Let's go through the steps. First it makes a scoped session object, it 
 then configures it with the right engine at the start of the request (it 
 can do this on a per-class level), and then at the end of the request it 
 removes the Session again, which results in the actual session being 
 closed.

 Our get_appropriate_engine() would probably look the engine up as a local 
 utility, as Laurence suggested. There is a bit of question about engine 
 configuration, though.

 If we want to support the use case of looking up the engine URL in a 
 persistent datastore (for instance one URL per location), we have a 
 question of ordering. We cannot do it too early; at the start of the 
 transaction there isn't a ZODB yet to talk to so we can't look up a local 
 utility. We can try doing it just in time:

 _Session = scoped_session()

 def Session(*args, **kw):
 engine = get_appropriate_engine()
 _Session.bind(bind=engine)
 return _Session(*args, **kw)

 Here get_appropriate_engine() could do a component.getUtility() and look 
 up the engine for us, possibly in an