[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Martijn Faassen wrote: Andreas Jung wrote: --On 10. April 2008 19:10:49 +0200 Brian Sutherland [EMAIL PROTECTED] wrote: Just adding my few requirements: - Integration into the component architecture in such a way that I can specify the db connection parameters in ZCML and that database reflection still works. I want to be able to get a handle on SQLAlchemy things by calling getUtility. Please *NO* database specific configurations within ZCML. We're running applications in up three or four different environments and I don't want to maintain instance specific configurations within ZCML. Either we pass such informations through environment variables or the database connections parameters are read from a configuration within the instance configuration. As long as it can also be specified in the code, that way people can build UIs to set up the connection. Perhaps that's what you mean by a configuration within the instance configuration. The way I use collective.lead in my book is to have it look up the database settings in a local utility. That utility is editable via a control panel page in Plone. I suspect that it'd be quite easy to do something similar where the settings were deferred to a global utility configured with ZCML or looked up in a config file of some sort. Once the config is available, it's all about calling getUtility() to get hold of a database connection/session. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
On Fri, Apr 11, 2008 at 05:07:56AM +0200, Andreas Jung wrote: --On 10. April 2008 21:29:43 +0200 Brian Sutherland [EMAIL PROTECTED] wrote: What defines if something is the instance configuration? Isn't ZCML just another format of configuration file? I agree it's evil to hardcode database connections in the application as they must be under the control of the sysadmin. But that doesn't preclude them from being expressed in zcml. The point is: we don't need yet another configuration language for the configuration of a site. ZCML should be used for the purpose it has been intended for: for the configuration of the components of your site (easily spoken). This kind of configuration is static if you setup a site for project X. But settings like ports, paths etc. often differ between dev, test and production environments. We already have ZConfig as standard cofniguration language for instance specific configuration aspects. In addition: we often do use environment variables and .ini files for additional configurations. Introducing another configuration language is stupid. Especially because system administrators or co-workers know little or nothing about ZCML. This is an unnecessary introduction of complexity. Setting an environment variable with a DSN is much easier then having to fiddle around with ZCML and XML...totally unnecessary overheadkeep it simple ...using ZCML in this configuration context is only for XML masochists. Unfortunately, it doesn't seem an option, right now, to avoid using ZCML to setup database connections. So I'm not sure how you want to implement your ideas? In general, I think if you want your configuration expressed in ZConfig and available as a utility, you have a problem. I'm not sure of: * How do I register a utility in ZConfig? * How will the above registration work properly in functional tests (especially with tearing down and setting up layers)? -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
--On 11. April 2008 10:09:16 +0200 Brian Sutherland [EMAIL PROTECTED] wrote: On Fri, Apr 11, 2008 at 05:07:56AM +0200, Andreas Jung wrote: --On 10. April 2008 21:29:43 +0200 Brian Sutherland [EMAIL PROTECTED] wrote: What defines if something is the instance configuration? Isn't ZCML just another format of configuration file? I agree it's evil to hardcode database connections in the application as they must be under the control of the sysadmin. But that doesn't preclude them from being expressed in zcml. The point is: we don't need yet another configuration language for the configuration of a site. ZCML should be used for the purpose it has been intended for: for the configuration of the components of your site (easily spoken). This kind of configuration is static if you setup a site for project X. But settings like ports, paths etc. often differ between dev, test and production environments. We already have ZConfig as standard cofniguration language for instance specific configuration aspects. In addition: we often do use environment variables and .ini files for additional configurations. Introducing another configuration language is stupid. Especially because system administrators or co-workers know little or nothing about ZCML. This is an unnecessary introduction of complexity. Setting an environment variable with a DSN is much easier then having to fiddle around with ZCML and XML...totally unnecessary overheadkeep it simple ...using ZCML in this configuration context is only for XML masochists. Unfortunately, it doesn't seem an option, right now, to avoid using ZCML to setup database connections. So I'm not sure how you want to implement your ideas? I have no need for ideas to be implemented. We're using SA within in several large projects based on z3c.sqlalchemy. And nothing beats the simplicity of of an .ini file or an environment variable. In general, I think if you want your configuration expressed in ZConfig and available as a utility, you have a problem. I'm not sure of: * How do I register a utility in ZConfig? z3c.sqlalchemy provides a method for registering a SA connection as a named utility. This code is usually called during the startup phase of Zope within the related initialize() methds of our products. The DSN are acquired from well-known environment variables...this just works - easy, simple, reliable. * How will the above registration work properly in functional tests (especially with tearing down and setting up layers)? Functional tests in our company are tests done by humans. Why? Our applications are very complex with lots of dependencies. Not speaking of lots of involved databases, big storages.. Andreas pgp98vhOQjdEm.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
--On 11. April 2008 08:36:43 +0100 Martin Aspeli [EMAIL PROTECTED] wrote: Martijn Faassen wrote: Andreas Jung wrote: --On 10. April 2008 19:10:49 +0200 Brian Sutherland [EMAIL PROTECTED] wrote: Just adding my few requirements: - Integration into the component architecture in such a way that I can specify the db connection parameters in ZCML and that database reflection still works. I want to be able to get a handle on SQLAlchemy things by calling getUtility. Please *NO* database specific configurations within ZCML. We're running applications in up three or four different environments and I don't want to maintain instance specific configurations within ZCML. Either we pass such informations through environment variables or the database connections parameters are read from a configuration within the instance configuration. As long as it can also be specified in the code, that way people can build UIs to set up the connection. Perhaps that's what you mean by a configuration within the instance configuration. The way I use collective.lead in my book is to have it look up the database settings in a local utility. That utility is editable via a control panel page in Plone. I suspect that it'd be quite easy to do something similar where the settings were deferred to a global utility configured with ZCML or looked up in a config file of some sort. Once the config is available, it's all about calling getUtility() to get hold of a database connection/session. Possibly a sane and clean solution but way of over-designed. I am interested in simple and robust solutions that work and not in a solution that adds more and more unnecessary layers just the sake of having them :-) Andreas pgpZS98rSgYnq.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Andreas Jung lists at zopyx.com writes: The way I use collective.lead in my book is to have it look up the database settings in a local utility. That utility is editable via a control panel page in Plone. I suspect that it'd be quite easy to do something similar where the settings were deferred to a global utility configured with ZCML or looked up in a config file of some sort. Once the config is available, it's all about calling getUtility() to get hold of a database connection/session. Possibly a sane and clean solution but way of over-designed. I am interested in simple and robust solutions that work and not in a solution that adds more and more unnecessary layers just the sake of having them collective.lead doesn't enforce this style at all. If you want TTW configurable connections, though, it's a good pattern to use. All collective.lead does is: - Set up database connections and engines properly - Hook into transaction machinery so you don't need to do your own transaction management - Let you look up a database session/connection as a utility You need to register that utility, of course. Lead has a base class that means you just fill in a few template methods (database url, and some callbacks for configuring tables and orm classes). If you want some kind of config-file based or ZCML based version, you'd just construct an appropriate class and register it as a utility on startup. Lead doesn't want to enforce one particular connection parameter configuration model or config file format. That should be the job of a higher level package that is concerned only with getting these parameters, not with the database setup itself. However, you probably still need some imperative code to set up the tables and mappers. Martin ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Brian Sutherland wrote: [snip] Unfortunately, it doesn't seem an option, right now, to avoid using ZCML to setup database connections. So I'm not sure how you want to implement your ideas? collective.lead can do it without ZCML; you set up the right connection by implementing a 'url' property on a (global) utility. This to me seems to be flexible, as the code could get the connection from anywhere. Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Andreas Jung wrote: [snip] Possibly a sane and clean solution but way of over-designed. I am interested in simple and robust solutions that work and not in a solution that adds more and more unnecessary layers just the sake of having them :-) It's a 1 liner to set up the connection in this story if you want to hard-code it in Python, and by implementing it as a property, you can get the connection string from any place you want, including environment variables or whatever else your Python code can get to. This cuts the whole debate about where the connection string should come from short, as you can code whatever you want yourself. This has an example of the utility setup: http://faassen.n--tree.net/blog/view/weblog/2008/04/08/0 Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Brian Sutherland schreef: Unfortunately, it doesn't seem an option, right now, to avoid using ZCML to setup database connections. So I'm not sure how you want to implement your ideas? Something we did for an LDAP connection that needed the same kind of server-dependent config: put an ldapconfig.py file in zope's etc/ dir and load that if available from the code that configures ldap. Not the best of solutions, probably, but it worked. What I wanted to say: this *is* a way to avoid using zcml! Reinout -- Reinout van Rees Blog: http://vanrees.org/weblog/ [EMAIL PROTECTED] Work: http://zestsoftware.nl/ http://vanrees.org Video: http://reinout.blip.tv/ ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
On 11.04.2008, at 10:09, Brian Sutherland wrote: Unfortunately, it doesn't seem an option, right now, to avoid using ZCML to setup database connections. So I'm not sure how you want to implement your ideas? +1 k, i didn't read the whole thread, but we (lovely systems) use zope.storm but i think its the same story somehow and we do all instance specific configuration in zcml - also database uris. we have a base.zcml config for all instances and include this it and do all instance specific configuration in an instance specific zcml file. this file is then used with the site-definition otpion in zope.conf. i can see no reason why we should avoid zcml except in environments where there is no zcml at all. although it is a few lines of code to get db uris from somewhere else, and this could also be application code regards, bernd smime.p7s Description: S/MIME cryptographic signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Bernd Dorn wrote: [whether configuration should be in ZCML] although it is a few lines of code to get db uris from somewhere else, and this could also be application code. I'm not against ZCML, as long as those lines of code to get the db URI from application code also exist. A good API to configure things without ZCML is necessary (that the ZCML can then turn around and use - this is a good way to create ZCML directives anyway). A nice thing about collective.lead is that it doesn't introduce custom ZCML directives, so in Grok, using it means there's no ZCML at all and no need to write any grokkers for it either. If it did introduce custom directives to do the same thing, that's fine, as long the original way isn't damaged by it. Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
On Friday 11 April 2008, Bernd Dorn wrote: although it is a few lines of code to get db uris from somewhere else, and this could also be application code if you use paster, you can stick options into your pater ini script. We are currently doing this for a setting in our app. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. Zope Stephan Richter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
On Fri, Apr 11, 2008 at 01:21:45PM +0200, Andreas Jung wrote: Unfortunately, it doesn't seem an option, right now, to avoid using ZCML to setup database connections. So I'm not sure how you want to implement your ideas? I have no need for ideas to be implemented. We're using SA within in several large projects based on z3c.sqlalchemy. And nothing beats the simplicity of of an .ini file or an environment variable. I see. I suppose you could also write a zcml declaration to pluck out the database config from an environment variable and register a utility. In general, I think if you want your configuration expressed in ZConfig and available as a utility, you have a problem. I'm not sure of: * How do I register a utility in ZConfig? z3c.sqlalchemy provides a method for registering a SA connection as a named utility. This code is usually called during the startup phase of Zope within the related initialize() methds of our products. The DSN are acquired from well-known environment variables...this just works - easy, simple, reliable. * How will the above registration work properly in functional tests (especially with tearing down and setting up layers)? Functional tests in our company are tests done by humans. Why? Our applications are very complex with lots of dependencies. Not speaking of lots of involved databases, big storages.. I try do what I can... which is relatively minimal. At least some components are thoroughly tested with the parts of the db schema they depend on. -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Martin Aspeli wrote: [snip] It's never particularly pleasant. I think we need to make sure we make this as easy and pleasant as possible though. This means at the very least some document that tells you what to do. :) Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Martin Aspeli wrote: Laurence Rowe wrote: Should one phase commit be set as the default to make it easier to work with sqlite (and mssql)? Probably yes. Ideally we'd guess based on the URL scheme but allow it to be set explicitly, IMHO. Single phase would be the fallback, I guess. I don't like the idea of guessing too much, but it could be done in the __init__. There might be good reasons to choose one phase commit even for a two phase commit supporting database (i.e. when you only are talking to a single database, it is more efficient). Given that a single database is the common case I am tempted to say just make 1PC the default. Should the default be for sessions to start out `active` or `dirty`? If they start out `dirty` then existing 1.0 code should work as before. What are the implications of having them start out dirty? Unnecessary commits to the database for read only transactions (i.e. the current situation) Laurence ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Martin Aspeli wrote: Martijn Faassen wrote: Martijn Pieters wrote: On Tue, Apr 8, 2008 at 11:54 PM, Martijn Faassen [EMAIL PROTECTED] wrote: All of these are in various states of brokenness. z3c.zalchemy doesn't work with SQLAlchemy trunk. collective.lead works with it, but only if you check out a particular branch, and not with sqlite. Quite possibly z3c.sqlalchemy has a release that actually works. One out of three is not bad... :) We are going into production with collective.lead and are committed to releasing the 1.0 branch as 1.0. sqlite works just great for us, we use it to run unit tests and for developers that just need to adjust the styling and such. The production environment will run against Oracle. The elro-tpc branch (which, I was told, is the future) doesn't appear to work with sqlite out of the box. I think this is a temporary bug, as Laurence pointed out, caused by SQLite not supporting two-phase commit. It should be fixed before release, by adding a one-phase fallback. I'm curious how to do functional tests with collective.lead - I'd like to have some real easy way to get the database set up and tore down down in my tests. Are you doing this? I'm doing a lot of it in Java. :) The typical pattern is to use a test setup that does database clean-down to return it to a steady state. Test data is kept in a declarative file (e.g. with an XML syntax that maps to DB structure) and used to initialise the database before each run. Doing nested transactions and rollback would be nicer, but probably doesn't work in all cases since app code may well do explicit commits. Note that sqlite does not support nested transactions or savepoints. It's never particularly pleasant. Martin Laurence ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
On Wed, Apr 09, 2008 at 05:47:06PM +0200, Andreas Jung wrote: --On 9. April 2008 14:15:38 +0100 Laurence Rowe [EMAIL PROTECTED] wrote: @everyone: If we can all agree to use the same basic session and transaction management then we should probably push for it to be included as a sqlalchemy extension module. I would be happy with such a solution. As said, I am not into the datamanager-building business and more interested in application-integration issues. The basic requirements I have for our applications: - having the same session within one ZODB transaction (per thread) - need to get hold of the underlying database connection for a given session Just adding my few requirements: - Integration into transactions. - Integration into the component architecture in such a way that I can specify the db connection parameters in ZCML and that database reflection still works. I want to be able to get a handle on SQLAlchemy things by calling getUtility. - Have a connection available as IZopeDatabaseAdapter utility (for BBB) -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
--On 10. April 2008 19:10:49 +0200 Brian Sutherland [EMAIL PROTECTED] wrote: Just adding my few requirements: - Integration into the component architecture in such a way that I can specify the db connection parameters in ZCML and that database reflection still works. I want to be able to get a handle on SQLAlchemy things by calling getUtility. Please *NO* database specific configurations within ZCML. We're running applications in up three or four different environments and I don't want to maintain instance specific configurations within ZCML. Either we pass such informations through environment variables or the database connections parameters are read from a configuration within the instance configuration. Andreas pgpPVek73cZst.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
On Thu, Apr 10, 2008 at 07:31:54PM +0200, Andreas Jung wrote: --On 10. April 2008 19:10:49 +0200 Brian Sutherland [EMAIL PROTECTED] wrote: Just adding my few requirements: - Integration into the component architecture in such a way that I can specify the db connection parameters in ZCML and that database reflection still works. I want to be able to get a handle on SQLAlchemy things by calling getUtility. Please *NO* database specific configurations within ZCML. We're running applications in up three or four different environments and I don't want to maintain instance specific configurations within ZCML. Either we pass such informations through environment variables or the database connections parameters are read from a configuration within the instance configuration. What defines if something is the instance configuration? Isn't ZCML just another format of configuration file? I agree it's evil to hardcode database connections in the application as they must be under the control of the sysadmin. But that doesn't preclude them from being expressed in zcml. I *do* like the idea of using one and only one file format to configure applications. We also run the in a few different environments, the /etc/myserver/site.zcml is different on a per instance basis and looks like this: configure xmlns=http://namespaces.zope.org/zope; xmlns:rdb=http://namespaces.zope.org/rdb; include package=mymodule file=configure.zcml / rdb:provideConnection name=admin component=psycopg2da.adapter.Psycopg2Adapter dsn=dbi://prototype / /configure But, I'm diverging, my point is that it would be nice to be able to have SQLAlchemy connections available as IZopeDatabaseAdapter utilities to allow a migration path. Whether they are defined in or out of ZCML, I just don't care. -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
On Thu, Apr 10, 2008 at 09:29:43PM +0200, Brian Sutherland wrote: On Thu, Apr 10, 2008 at 07:31:54PM +0200, Andreas Jung wrote: Please *NO* database specific configurations within ZCML. We're running applications in up three or four different environments and I don't want to maintain instance specific configurations within ZCML. Either we pass such informations through environment variables or the database connections parameters are read from a configuration within the instance configuration. I must admit though, that being able to set an environment variable and have an IZopeDatabaseAdapter utility registered from that would make setting up functional tests much nicer. Currently, because of layers, all my functional tests run against a magical database test :((( -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
On Thu, Apr 10, 2008 at 1:31 PM, Andreas Jung [EMAIL PROTECTED] wrote: --On 10. April 2008 19:10:49 +0200 Brian Sutherland [EMAIL PROTECTED] wrote: Just adding my few requirements: - Integration into the component architecture in such a way that I can specify the db connection parameters in ZCML and that database reflection still works. I want to be able to get a handle on SQLAlchemy things by calling getUtility. Please *NO* database specific configurations within ZCML. We're running applications in up three or four different environments and I don't want to maintain instance specific configurations within ZCML. Either we pass such informations through environment variables or the database connections parameters are read from a configuration within the instance configuration. i still don't see the harm having zcml db conf in a base package, if you don't have to use it to setup your particular deployment's connections. -kapil ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Andreas Jung wrote: --On 10. April 2008 19:10:49 +0200 Brian Sutherland [EMAIL PROTECTED] wrote: Just adding my few requirements: - Integration into the component architecture in such a way that I can specify the db connection parameters in ZCML and that database reflection still works. I want to be able to get a handle on SQLAlchemy things by calling getUtility. Please *NO* database specific configurations within ZCML. We're running applications in up three or four different environments and I don't want to maintain instance specific configurations within ZCML. Either we pass such informations through environment variables or the database connections parameters are read from a configuration within the instance configuration. As long as it can also be specified in the code, that way people can build UIs to set up the connection. Perhaps that's what you mean by a configuration within the instance configuration. Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
--On 10. April 2008 21:29:43 +0200 Brian Sutherland [EMAIL PROTECTED] wrote: On Thu, Apr 10, 2008 at 07:31:54PM +0200, Andreas Jung wrote: --On 10. April 2008 19:10:49 +0200 Brian Sutherland [EMAIL PROTECTED] wrote: Just adding my few requirements: - Integration into the component architecture in such a way that I can specify the db connection parameters in ZCML and that database reflection still works. I want to be able to get a handle on SQLAlchemy things by calling getUtility. Please *NO* database specific configurations within ZCML. We're running applications in up three or four different environments and I don't want to maintain instance specific configurations within ZCML. Either we pass such informations through environment variables or the database connections parameters are read from a configuration within the instance configuration. What defines if something is the instance configuration? Isn't ZCML just another format of configuration file? I agree it's evil to hardcode database connections in the application as they must be under the control of the sysadmin. But that doesn't preclude them from being expressed in zcml. The point is: we don't need yet another configuration language for the configuration of a site. ZCML should be used for the purpose it has been intended for: for the configuration of the components of your site (easily spoken). This kind of configuration is static if you setup a site for project X. But settings like ports, paths etc. often differ between dev, test and production environments. We already have ZConfig as standard cofniguration language for instance specific configuration aspects. In addition: we often do use environment variables and .ini files for additional configurations. Introducing another configuration language is stupid. Especially because system administrators or co-workers know little or nothing about ZCML. This is an unnecessary introduction of complexity. Setting an environment variable with a DSN is much easier then having to fiddle around with ZCML and XML...totally unnecessary overheadkeep it simple ...using ZCML in this configuration context is only for XML masochists. Andreas pgpyigxtKwGmK.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Martijn Faassen wrote: Hi there, [I originally picked this up on a thread on zope3-users, but this deserves its own thread here] There are at least three approaches to SQLAlchemy integration with Zope: * z3c.zalchemy (Christian Theune) * z3c.sqlalchemy (Andreas Jung) * collective.lead (Laurence Rowe) All of these are in various states of brokenness. z3c.zalchemy doesn't work with SQLAlchemy trunk. collective.lead works with it, but only if you check out a particular branch, and not with sqlite. Quite possibly z3c.sqlalchemy has a release that actually works. One out of three is not bad... :) Then there's also mentions about WSGI-based integration, and I think in Plone, Alchemist probably also does its own integration... There must be a reason for this proliferation of approaches. What is it? We all get along, don't we? I know that the various packages are taking code and approaches from each other too. Can't we work together more and at least come up with *one* package that works? Perhaps factor out some low-level commonality than then all share? Criticize one of the other packages until you're satisfied, and then retire your own package perhaps? I know the various packages add on their own approaches to configuration and might offer higher level container approaches. Those could be in different packages, sharing a foundation. In the end, I hope we will end up with just *one* integration layer, that is released, that works with Zope 2 and Zope 3 and a recent release of SQLAlchemy, that is documented, and that people know about. We can then offer packages on top of this that offer extra features. I'm all for one integration layer, even though I spurred one of the above (collective.lead). I would ask that we be very conservative, though. collective.lead wants to provide: - automatic transaction integration with ZODB transactions - an easy way to register your database connections - an easy way to look those up as utilities and get an ORM session Everything else (e.g. Zope ORM integration, configuration abstractions, etc) should be in some other package. collective.lead has a reasonable degree of traction in the Plone universe, at least, but should work just fine with plain Zope 3. We have other things that work on top it (mercury, rope, probably others). The branch that works with trunk also has some carefully worked out TPC support, which is difficult to get right, though I'm disappointed that we seem to have broken sqlite integration. Hopefully it's an easy fix, though. I'm CC'ing Laurence to ask: - can we get a release soon? - can we fix the sqlite integration? Cheers, Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
The transaction manager here: http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collective/lead/tx.py Has support for TPC and savepoints. It is tested and works. The only issue with sqlite is that the default in the branch is to use two-phase commit and sqlite does not support this. Apart from that the module is completely self contained. The public interfaces to it is the two functions at the bottom, join_transaction and dirty_session. The idea of the dirty_session stuff is to prevent unnecessary commits when data is only read. Session writes trigger the 'dirtying' automatically with a SessionExtension: http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collective/lead/database.py The downside of this approach is that the when you interact with the connection directly to modify data you must call dirty_session yourself. If this is a problem for you then you may set up the transaction to be in the dirty state initially. Thread based session management is used (scoped_session). There is fairly extensive testing of this and the transaction support here: http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collective/lead/tests.py I think it would be worth factoring out the threaded session support as well as the transactions. @Christian: I'm not quite sure what you mean by: - Database triggers as events I don't see how you could reasonably get a database trigger on the db server to instantiate a zope event on the db client. Perhaps the database could be persuaded to make a request to a URL on a trigger, but this seems a bit slow and maybe not a great solution. - Making SA 0.4 work with security proxies I don't know anything about security proxies in zope 3, but surely this is just a matter of what the base class is / the container implementation? I don't see why an SQL Alchemy mapped object would be any different to any other object. @Martin: Should one phase commit be set as the default to make it easier to work with sqlite (and mssql)? Probably yes. Should the default be for sessions to start out `active` or `dirty`? If they start out `dirty` then existing 1.0 code should work as before. It'll probably be next week before I manage to get this tidied up and merged back to trunk. @everyone: If we can all agree to use the same basic session and transaction management then we should probably push for it to be included as a sqlalchemy extension module. Laurence Martin Aspeli wrote: Martijn Faassen wrote: Hi there, [I originally picked this up on a thread on zope3-users, but this deserves its own thread here] There are at least three approaches to SQLAlchemy integration with Zope: * z3c.zalchemy (Christian Theune) * z3c.sqlalchemy (Andreas Jung) * collective.lead (Laurence Rowe) All of these are in various states of brokenness. z3c.zalchemy doesn't work with SQLAlchemy trunk. collective.lead works with it, but only if you check out a particular branch, and not with sqlite. Quite possibly z3c.sqlalchemy has a release that actually works. One out of three is not bad... :) Then there's also mentions about WSGI-based integration, and I think in Plone, Alchemist probably also does its own integration... There must be a reason for this proliferation of approaches. What is it? We all get along, don't we? I know that the various packages are taking code and approaches from each other too. Can't we work together more and at least come up with *one* package that works? Perhaps factor out some low-level commonality than then all share? Criticize one of the other packages until you're satisfied, and then retire your own package perhaps? I know the various packages add on their own approaches to configuration and might offer higher level container approaches. Those could be in different packages, sharing a foundation. In the end, I hope we will end up with just *one* integration layer, that is released, that works with Zope 2 and Zope 3 and a recent release of SQLAlchemy, that is documented, and that people know about. We can then offer packages on top of this that offer extra features. I'm all for one integration layer, even though I spurred one of the above (collective.lead). I would ask that we be very conservative, though. collective.lead wants to provide: - automatic transaction integration with ZODB transactions - an easy way to register your database connections - an easy way to look those up as utilities and get an ORM session Everything else (e.g. Zope ORM integration, configuration abstractions, etc) should be in some other package. collective.lead has a reasonable degree of traction in the Plone universe, at least, but should work just fine with plain Zope 3. We have other things that work on top it (mercury, rope, probably others). The branch that works with trunk also has some carefully worked out TPC support, which is difficult to get right,
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Martijn Pieters wrote: On Tue, Apr 8, 2008 at 11:54 PM, Martijn Faassen [EMAIL PROTECTED] wrote: All of these are in various states of brokenness. z3c.zalchemy doesn't work with SQLAlchemy trunk. collective.lead works with it, but only if you check out a particular branch, and not with sqlite. Quite possibly z3c.sqlalchemy has a release that actually works. One out of three is not bad... :) We are going into production with collective.lead and are committed to releasing the 1.0 branch as 1.0. sqlite works just great for us, we use it to run unit tests and for developers that just need to adjust the styling and such. The production environment will run against Oracle. The elro-tpc branch (which, I was told, is the future) doesn't appear to work with sqlite out of the box. I'm curious how to do functional tests with collective.lead - I'd like to have some real easy way to get the database set up and tore down down in my tests. Are you doing this? Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
--On 9. April 2008 14:15:38 +0100 Laurence Rowe [EMAIL PROTECTED] wrote: @everyone: If we can all agree to use the same basic session and transaction management then we should probably push for it to be included as a sqlalchemy extension module. I would be happy with such a solution. As said, I am not into the datamanager-building business and more interested in application-integration issues. The basic requirements I have for our applications: - having the same session within one ZODB transaction (per thread) - need to get hold of the underlying database connection for a given session Andreas pgpFEZVYm6t1t.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
On Wed, Apr 9, 2008 at 9:15 AM, Laurence Rowe [EMAIL PROTECTED] wrote: The transaction manager here: http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collective/lead/tx.py Has support for TPC and savepoints. It is tested and works. The only issue with sqlite is that the default in the branch is to use two-phase commit and sqlite does not support this. Apart from that the module is completely self contained. The public interfaces to it is the two functions at the bottom, join_transaction and dirty_session. The idea of the dirty_session stuff is to prevent unnecessary commits when data is only read. Session writes trigger the 'dirtying' automatically with a SessionExtension: http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collective/lead/database.py The downside of this approach is that the when you interact with the connection directly to modify data you must call dirty_session yourself. If this is a problem for you then you may set up the transaction to be in the dirty state initially. Thread based session management is used (scoped_session). There is fairly extensive testing of this and the transaction support here: http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collective/lead/tests.py I think it would be worth factoring out the threaded session support as well as the transactions. +1, the collective.lead transaction support looks nice. @Christian: I'm not quite sure what you mean by: - Database triggers as events I don't see how you could reasonably get a database trigger on the db server to instantiate a zope event on the db client. Perhaps the database could be persuaded to make a request to a URL on a trigger, but this seems a bit slow and maybe not a great solution. - Making SA 0.4 work with security proxies I don't know anything about security proxies in zope 3, but surely this is just a matter of what the base class is / the container implementation? I don't see why an SQL Alchemy mapped object would be any different to any other object. its more to do with the fact that proxies are restrictive by default, and the sa api likes to monkey attributes on instances. christian kicked off some nice work with the sa developers at pycon to get this behavior configurable in sa ( instrumentation branch) which obviate the need to remove the proxies before calling into sa apis on an object. @Martin: Should one phase commit be set as the default to make it easier to work with sqlite (and mssql)? Probably yes. Should the default be for sessions to start out `active` or `dirty`? If they start out `dirty` then existing 1.0 code should work as before. It'll probably be next week before I manage to get this tidied up and merged back to trunk. @everyone: If we can all agree to use the same basic session and transaction management then we should probably push for it to be included as a sqlalchemy extension module. sounds good, or failing that (due to additional dependencies), a common base package. cheers, kapil ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Laurence Rowe wrote: Should one phase commit be set as the default to make it easier to work with sqlite (and mssql)? Probably yes. Ideally we'd guess based on the URL scheme but allow it to be set explicitly, IMHO. Single phase would be the fallback, I guess. Should the default be for sessions to start out `active` or `dirty`? If they start out `dirty` then existing 1.0 code should work as before. What are the implications of having them start out dirty? It'll probably be next week before I manage to get this tidied up and merged back to trunk. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Martijn Faassen wrote: Martijn Pieters wrote: On Tue, Apr 8, 2008 at 11:54 PM, Martijn Faassen [EMAIL PROTECTED] wrote: All of these are in various states of brokenness. z3c.zalchemy doesn't work with SQLAlchemy trunk. collective.lead works with it, but only if you check out a particular branch, and not with sqlite. Quite possibly z3c.sqlalchemy has a release that actually works. One out of three is not bad... :) We are going into production with collective.lead and are committed to releasing the 1.0 branch as 1.0. sqlite works just great for us, we use it to run unit tests and for developers that just need to adjust the styling and such. The production environment will run against Oracle. The elro-tpc branch (which, I was told, is the future) doesn't appear to work with sqlite out of the box. I think this is a temporary bug, as Laurence pointed out, caused by SQLite not supporting two-phase commit. It should be fixed before release, by adding a one-phase fallback. I'm curious how to do functional tests with collective.lead - I'd like to have some real easy way to get the database set up and tore down down in my tests. Are you doing this? I'm doing a lot of it in Java. :) The typical pattern is to use a test setup that does database clean-down to return it to a steady state. Test data is kept in a declarative file (e.g. with an XML syntax that maps to DB structure) and used to initialise the database before each run. Doing nested transactions and rollback would be nicer, but probably doesn't work in all cases since app code may well do explicit commits. It's never particularly pleasant. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: straighting out the SQLAlchemy integration mess
Andreas Jung wrote: --On 9. April 2008 14:15:38 +0100 Laurence Rowe [EMAIL PROTECTED] wrote: @everyone: If we can all agree to use the same basic session and transaction management then we should probably push for it to be included as a sqlalchemy extension module. I would be happy with such a solution. As said, I am not into the datamanager-building business and more interested in application-integration issues. The basic requirements I have for our applications: - having the same session within one ZODB transaction (per thread) - need to get hold of the underlying database connection for a given session I think that's very much the business collective.lead wants to be in. :) Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess
--On 10. April 2008 01:57:12 +0100 Martin Aspeli [EMAIL PROTECTED] wrote: Andreas Jung wrote: --On 9. April 2008 14:15:38 +0100 Laurence Rowe [EMAIL PROTECTED] wrote: @everyone: If we can all agree to use the same basic session and transaction management then we should probably push for it to be included as a sqlalchemy extension module. I would be happy with such a solution. As said, I am not into the datamanager-building business and more interested in application-integration issues. The basic requirements I have for our applications: - having the same session within one ZODB transaction (per thread) - need to get hold of the underlying database connection for a given session I think that's very much the business collective.lead wants to be in. :) You know what was first :-) Even with the collective.lead and z3c.zalchemy datamanager...still two instead of one :-) -aj pgpBydBlV3U6n.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )