[Zope-dev] Re: could zope.sqlalchemy flush before committing?
i should be doing 0.4.7 and 0.5beta3 early this week. both include after_attach(). On Jul 21, 2008, at 4:45 AM, Martijn Faassen wrote: Hey, On Mon, Jul 21, 2008 at 9:55 AM, Malthe Borch [EMAIL PROTECTED] wrote: Martijn Faassen wrote: Thanks for the offer. I think this is up to Laurence to decide, I'd say. I'm aiming my work at the 0.5 series so I'm fine with requiring 0.5. Me too, but I'd be careful to *require* an unreleased version. Well, I was assuming there'd be a (beta) release at least. z3c.saconfig has been requiring a released beta version of 0.5. Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: could zope.sqlalchemy flush before committing?
On Jul 19, 2008, at 6:33 AM, Martijn Faassen wrote: Laurence Rowe wrote: This is now fixed in trunk. For the moment I'm depending on SQLAlchemy trunk for the new after_attach hook until beta3 is released. Maybe it's time to start depending on 0.5? No problem with that from my side, though of course I think this means beta3 should be released first, right? The 0.4 series can handle an after_attach() hook just as easily as 0.5...would this make life easier ? ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: could zope.sqlalchemy flush before committing?
On Jul 19, 2008, at 10:29 AM, Brandon Craig Rhodes wrote: Laurence Rowe [EMAIL PROTECTED] writes: This is now fixed in trunk. For the moment I'm depending on SQLAlchemy trunk for the new after_attach hook until beta3 is released. Could the fix be backported and a new release made? I have to have this particular system ready for production next week, and would love to not have to explain why it was running out of Subversion in production. :-) backported to ... 0.4 ? sure. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: SQLAlchemy integration experiment
On Jun 17, 2008, at 12:41 PM, Laurence Rowe wrote:
What connection pooling is used by default? e.g. with
create_engine('sqlite:///:memory:')
sqlite is a special case, it uses the SingletonThreadPool. This pool
holds onto one connection per thread.This is used in SQLite
because of a sometimes-restriction that a sqlite connection can only
be used in the same thread in which it was created. The pool in
normal use is QueuePool.
I think we are only talking about the difference between using four
pools of one connection versus one pool of four connections (assuming
the standard four threads in zope). I don't see that making a lot of
difference in practice.
in practice, a single pool of four connections means if one of those
connections encounters a connection lost exception, the exception is
raised, and the entire pool is recycled; meaning that only one
exception is raised for the whole application during a database
restart. It also means that the total connections used by the
application for a particular database can be configured/throttled in
one place. To me thats a significant difference.
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] SQLAlchemy (zope.sqlalchemy) integration
On Jun 4, 2008, at 12:02 PM, Hermann Himmelbauer wrote: Hi, Regarding to the discussion some days ago with the SQLAlchemy Zope3 integration, I still have problems with retrieving the session. I currently use a utility for the engine, which seems to work well. However, for retrieving the session, I tried to use the following pattern (many thanks to Michael Bayer, btw.): database module --- SASession = scoped_session(sessionmaker( transactional = True, autoflush = True, extension = ZopeTransactionExtension())) def getSASession(): SASession.remove() engine = getUtility(ISAEngineUtility).getEngine() s = SASession() s.bind = engine return s In my application, I then use getSASession() to retrieve my session. However, what I think is not that beautiful is the s.bind = engine part. Are there any suggestions how to improve this? FTR, my suggestion here is to configure/tear down sessions upon request boundaries, as described in http://www.sqlalchemy.org/docs/04/session.html#unitofwork_contextual_lifespan . ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On May 28, 2008, at 7:09 PM, Laurence Rowe wrote: Engine configuration is a subset of session configuration. You cannot have a single ScopedSession for a package if you want to have multiple instances of that application. We must work with unbound metadata if we have this goal. That implies that we must use bound sessions, a session associated with a particular engine (actually it could be more complex than this, you can associate particular classes/tables with particular engines within a session). chiming in, although I haven't read the whole thread I would agree that bound metadata is a little cumbersome in an environment that wants to switch engines within a single process. Bound metadata has always given us a lot of headaches because it's just *so* convenient that we just can't get rid of it, but at the same time people are just so tripped up by it, thinking it's necessary to make anything happen (it's not). Binding sessions allows better control of connection/transaction scope (since the Session relates to transactions, MetaData does not), so it's probabably the way to go in an environment that has a lot of things going on. As far as ScopedSession, it's really just a thread local variable. Like any global threadlocal, you can stick whatever you want on it at the start of a request, and tear it down at the end. Then again it's also reasonable that you might want to have individual ScopedSessions for each application instance within a multi-app process, that way the burden of setting up/tearing down is reduced or removed. A single ScopedSession for a multi-app process is like a one-dimensional approach where both current thread and current app are identified by the current get() of the registry; a collection of ScopedSessions is more like a two-dimensional approach where the first level of registry (i.e. which ScopedSession do I choose) distinguishes between app instance, and the second level (i.e. what Session is bound to this thread ID) distinguishes between threads. All of that said I think it can work either way but I think the latter approach might have the explicitness you're looking for. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy, integration ideas
On May 30, 2008, at 9:44 AM, Martijn Faassen wrote: Hey, Michael Bayer wrote: [snip discussion on BoundMetadata I don't comprehend yet but probably should :)] As far as ScopedSession, it's really just a thread local variable. Like any global threadlocal, you can stick whatever you want on it at the start of a request, and tear it down at the end. Is it really needed to set things up at the start and the request and tear things down on the end, or can a session be retained between requests? I.e. is the intent to recreate a Session class each time a request is issued? in the 0.4 series, the Session is sticky in that whatever is in it retains the state that was last loaded - 0.4 does not go back to the database to re-fetch the state of an object unless its explicitly told to do so. When an HTTP request ends, assuming all objects in the session are marked as clean, they're weakly referenced and will fall out of scope assuming those objects were only referenced by the request. So in that sense, you can just leave the Session alone and it'll work just fine for the next request...but any pending changes in the Session that weren't flushed for some reason would get carried over, as well as anything that might be strongly referenced elsewhere. So we always recommended in 0.4 to at least issue a session.clear() at the end of a request to just empty it out (but you can still reuse that session). Other options included scopedsession.remove() which tears the whole session down, the advantage there being that the new request could configure the next session differently (as in, binding it to something else). in 0.5, Session has been changed to be less reluctant to go and re- fetch data (which is contrary to the particular background I came from, but since then I've learned to see a broader scope of use cases). In 0.5, after a commit() or rollback(), the Session still may be holding on to objects, but their state is expired such that it will all be re-loaded upon access, and all pending and deleted states are reverted. So 0.5's Session, when configured in the default way, makes it very hard to get at stale state, so in that sense you can just re-use a session wihtout worrying much about what may have been left over. Then again it's also reasonable that you might want to have individual ScopedSessions for each application instance within a multi-app process, that way the burden of setting up/tearing down is reduced or removed. This indicates to me is possible to retain a Session object between two requests? I.e. it's a worthwhile goal to reduce the amount of Session configuration going on, right? Its not a strong argument either way to reuse a session or just make a new one on each request. In 0.4, making a brand new session on each request does have a cleaner feel to it since theres no chance of any state left hanging around. Its not an expensive operation. A single ScopedSession for a multi-app process is like a one- dimensional approach where both current thread and current app are identified by the current get() of the registry; a collection of ScopedSessions is more like a two-dimensional approach where the first level of registry (i.e. which ScopedSession do I choose) distinguishes between app instance, and the second level (i.e. what Session is bound to this thread ID) distinguishes between threads. All of that said I think it can work either way but I think the latter approach might have the explicitness you're looking for. I'm trying to understand why you think so. I am looking for a way to let developers use SQLAlchemy in a straightforward way. They should be able to import 'Session', instantiate session in their app, and everything works as expected. The framework takes care of making you get the appropriately configured Session. they can in fact do this without any issue.The question is, when someone writes the call s = Session() three times within one request, do you want each s to all reference the *same* set of objects ? that was the issue scoped_session() was meant to solve. Configuration is easy since you just configure a Session callable for that application. But you guys have this issue of multiple applications in the same process at play, which each talk to a different database. So thats where a decision has to be made how to deal with that complexity, either configure engine on one scoped_session per request, or configure multiple scoped_sessions, one per engine. The latter approach seems easier to me since you don't actually have to do anything on a per-request basis assuming 0.5 usage. Neither of these make any difference to the end user who sees the exact same usage pattern. (I'd still absolutely like to avoid passing in context explicitly each time you need a session; it puts unnecessary burden on the developer
Re: [Zope-dev] Re: zope.sqlalchemy
On May 7, 2008, at 7:08 AM, Martijn Faassen wrote: Hi there (especially Christian), I think we can work with explicits saves. In many cases the user won't have to worry about it anyway as the container object will do it for them (besides making the relation), or this 'query container' we spoke of will do it for them (but just the 'save' bit). One point is that the scoped session approach itself doesn't work very well for using multiple databases in the same app. We could consider passing the session along in the containers during object graph wakling (or traversal) so an app can easily traverse into multiple databases. I'm not sure whether we can make the ORM do this for us though; does it initialize the mapping with a session? SQLAlchemy's Session does support multiple engine binds itself, which most easily can be associated with particular mapped classes (i.e. vertical partitioning), so that a single session (or a scoped_session) can read and write data to the appropriate tables transparently (although things like joins across multiple databases will raise errors). Theres a horizontally-partitioning version of Session as well which obviously has a lot more caveats. Using multiple sessions, one per DB is a valid approach as well. I'm not sure if Grok has other things going on when mulitple DBs are in use but SA's multi-bind capability is something to be aware of. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy
On May 7, 2008, at 1:29 PM, Laurence Rowe wrote:
I'm thinking more about having the same classes mapped to different
databases at different points in the application. Imagine a
departmental address book app. Intstances of the departmental
address book are created for each department, each with a different
databases:
http://addressbook/sales - postgres:///sales
http://addressbook/engineering - postgres:///engineering
The way I imagine this working is to have a proxy engine object that
looks up the real engine through a local utility. Each application
would be a `site` and capable of local utility registrations. /sales
would have Engine('postgres:///sales') registered and /engineering
Engine('postgres:///engineering').
Only a single ScopedSession would be required. This would be bound
to proxy that performs the utility lookup. So when in the /sales
context the proxy would point to the sales engine and when in the /
engineering context to the engineering engine.
The limitation of this approach is that it would not be possible to
mix objects from /sales and objects from /engineering into the same
transaction. So really we need a session per application instance.
Perhaps this can be achieved through a custom scoping function:
def scopefunc():
return thread.get_ident(), id(zope.component.getSiteManager())
If you want to mix tables (and optionally engines) for the *same*
class, we actually have a feature for that too. Its sort of a feature
I've wanted to remove but Jason keeps arguing that its worthy. It's
called entity_name and it allows multiple primary mappers to be
created for a single class. The entity_name has to be specified when
you add the element to the session (yet another reason explicit add()
is a good thing). This feature is taken directly from the Hibernate
feature of the same name.
The limitation with entity_name which needs some more fixing in 0.5 is
that only one mapper gets to define the attribute instrumentation for
the entity. If you are storing the same class in three different
tables (across three different databases or just one), the attributes
defined on the class need to be compatible with all three. This is
reasonable since a class can only have one descriptor per attribute
name. Querying is also slightly challenging since the descriptors
need to be qualified for a particular mapper (i.e. you cant just say
query.filter(Address.id==5)...which id are we talking about ?)
The reason I'm not totally keen on this feature is that it seems to be
a very exotic way of getting around making simple subclasses, and I've
yet to see the use case for it where simple subclasses don't work,
except for cosmetic reasons which I have a hard time swallowing
(even if the reasons are cosmetic, you can still create subclasses
that are all named the same).
So I will ask you, why can't your application simply have a
SalesAddress and an EngineeringAddress class ? You could even
produce them transparently using a custom __new__() method, i.e.
class Address(object):
def __new__(cls, *args, **kwargs):
if my_scoped_thing.context == 'sales':
return object.__new__(SalesAddress)
elif my_scoped_thing.context == 'engineering':
return object.__new__(EngineeringAddress)
this seems extremely straightforward to me as each object, once
instantiated is now bound for a specific destination. It doesnt seem
like youd want the *same* Address to be stored in one and then the
other in a different instance (that seems *extremely* complex for no
good reason). Isnt explicit better than implicit ?
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy
On May 7, 2008, at 2:29 PM, Laurence Rowe wrote: When the generic address book application is built you don't know what the departments will be called or indeed how many departments there are. An address book is not be a great example, but I know of intranet portal sites where this is a requirement. You want to delegate control to each department so you give each department their own instance of the portal. You only want to maintain one code base though, and you don't want to change it every time someone adds another departmental portal. I'd like to be able to create an add form that has fields for application name and database url. This probably seems like an odd requirement -- why not just run multiple processes with different configurations -- but it's the way zope has traditionally worked. A single process can serve multiple instances of the same application (or `site`). When you get up to running tens of sites, the memory footprint of Zope2 and Plone (before the object cache) becomes significant. If you are running different instances each connected to a different engine within one process you wouldn't need any awareness of engines at the object level (therefore no entity_name) and also no engine proxying - you should have separate Session instances for each process managed by scoped_session(), which was designed to handle this.Multiple apps on one codebase within one process was an original requirement of Pylons as well, though nobody has ever used it. The easiest way to do it is to set up the engine at the request level: Session = scoped_session() # start of request engine = get_appropriate_engine() Session(bind=engine) try: # do request finally: Session.remove() If that isnt granular enough, then you use a custom scope func which maintains Session per-process-per-thread. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy
On May 6, 2008, at 12:14 PM, Laurence Rowe wrote: Martijn Faassen wrote: One thing I understood from Christian Theune is that with scoped sessions, explicit session.save() is not always necessary. Since I see it being used here, could you perhaps comment on this? Registering a mapper with Session.mapper would work with this extension, but I'm reluctant to recommend it for two reasons: I don't know how it works with the declarative plugin; and it necessarily limits mapped classes to a single Session and therefor a single engine. In a zope context I think it's quite likely that you could have the same classes mapped to different databases (i.e. two instances of a single application). hi there - a little background on the save_on_init option of Session.mapper. This behavior has its roots way back in SQLAlchemy 0.1, when there was no Session or Query or anything like that, and objects, when instantiated, went directly to a thread-local registry automatically. When SQLA 0.2 came out, we introduced all the additional constructs like Session and such which are familiar today, but extensions were provided which, when enabled, would re-enable the 0.1 behavior of everything threadlocal/automatic in a similar way. Ultimately thats where Session.mapper comes from. Like all typing-savers, save on init by then was used by dozens of Pylons users who swore by it and would scream and yell at any hint of removing this already legacy feature. At the same time, new users who were using Pylons tutorials (and therefore save_on_init, without really knowing it) in conjunction with creating their own Session objects were baffled by error messages like object X is already in session Y. By the time 0.4 came out, we had started automating Session a lot more, adding autoflush capability to it. This feature immediately had issues with save_on_init for this reason: class MyClass(object): def __init__(self): self.some_variable = session.query(Foobar).filter(xyz).one() Where above, the query(Foobar) would fire off autoflush, MyClass would be flushed, and then an IntegrityError would be raised since MyClass would be missing some necessary state. Changing save_on_init to fire off *after* __init__ was a possibility there but then other things could break. So I've already not liked save_on_init for a couple of years due to its inherent intrusiveness, and because SA historically does not like being in the business of providing framework features (though we have decided to stay in that arena to some degree with declarative and scoped_session). The Session.mapper feature is stressed a whole lot less in the 0.4 docs, and as I work on the 0.5 docs this week I'm feeling very much like I'm going to remove it from the main documentation altogether. We''re consolidating the save/update/save_or_update names into just add() and add_all(), so explicitly adding items to a Session should be a more pleasant experience which I wouldn't want anyone to miss. The aspect of Session.mapper which is somewhat reduntant vs. declarative is that they both want to add an automatic __init__(**kwargs) method which assigns all given keyword values to the instance.They are not incompatible because Session.mapper only adds an __init__ if none is available already. The final feature of Session.mapper which is more reasonable is the query attribute. This feature allows you to say: MyClass.query as an equivalent for session.query(MyClass). For that specific attribute, instead of using Session.mapper, its functionality has been exported into its own descriptor-producing method, like so: class MyBaseClass(object): query = Session.query_property() So this is a way to get that one aspect without buying into the Session.mapper thing. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zope.sqlalchemy
On May 6, 2008, at 5:24 PM, Martijn Faassen wrote: Hey Michael, Thanks for the input! Michael Bayer wrote: [snip] So I've already not liked save_on_init for a couple of years due to its inherent intrusiveness, and because SA historically does not like being in the business of providing framework features (though we have decided to stay in that arena to some degree with declarative and scoped_session). I'll try to summarize the discussion so I can find out whether I understand it. Basically you're saying you don't think save on instantiation is a good idea generally, and that we should be using session.save(). This is going to be changed to session.add() in the future. What would session.add_all() do? session.add_all() is just: session.add_all([obj1, obj2, obj3, ...]) Also session.save()/update/save_or_update will remain throughout 0.5 at least. This ties into the mapper feature, which also offers other features. The one feature that will remain but in a new shape, without the mapper, is the ability to do MyClass.query. Is that a correct summary? Session.mapper and save_on_init has no plans of going away in 0.5, but I plan to de-emphasize it.The query descriptor function is also available in 0.4. - mike ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
