[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-11 Thread Martin Aspeli

Martijn Faassen wrote:

Andreas Jung wrote:
--On 10. April 2008 19:10:49 +0200 Brian Sutherland 
[EMAIL PROTECTED] wrote:

Just adding my few requirements:
- Integration into the component architecture in such a way that I
  can specify the db connection parameters in ZCML and that database
  reflection still works. I want to be able to get a handle on
SQLAlchemy   things by calling getUtility.
Please *NO* database specific configurations within ZCML. We're running 
applications in up three or four different environments and I don't want 
to maintain instance specific configurations within ZCML. Either we pass
such informations through environment variables or the database 
connections parameters are read from a configuration within the instance 
configuration.


As long as it can also be specified in the code, that way people can 
build UIs to set up the connection. Perhaps that's what you mean by a 
configuration within the instance configuration.


The way I use collective.lead in my book is to have it look up the 
database settings in a local utility. That utility is editable via a 
control panel page in Plone. I suspect that it'd be quite easy to do 
something similar where the settings were deferred to a global utility 
configured with ZCML or looked up in a config file of some sort.


Once the config is available, it's all about calling getUtility() to get 
hold of a database connection/session.


Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-11 Thread Brian Sutherland
On Fri, Apr 11, 2008 at 05:07:56AM +0200, Andreas Jung wrote:
 --On 10. April 2008 21:29:43 +0200 Brian Sutherland 
 [EMAIL PROTECTED] wrote:
 What defines if something is the instance configuration?

 Isn't ZCML just another format of configuration file? I agree it's evil
 to hardcode database connections in the application as they must be
 under the control of the sysadmin. But that doesn't preclude them from
 being expressed in zcml.

 The point is: we don't need yet another configuration language for the 
 configuration of a site. ZCML should be used for the purpose it has been 
 intended for: for the configuration of the components of your site (easily 
 spoken). This kind of configuration is static if you setup a site for 
 project X. But settings like ports, paths etc. often differ between dev, 
 test and production environments.
 
 We already have ZConfig as standard cofniguration language for instance 
 specific configuration aspects. In addition: we often do use environment 
 variables and .ini files for additional configurations. Introducing another 
 configuration language is stupid. Especially because system administrators 
 or co-workers know little or nothing about ZCML. This is an unnecessary 
 introduction of complexity. Setting an environment variable with a DSN is 
 much easier then having to fiddle around with ZCML and XML...totally 
 unnecessary overheadkeep it simple ...using ZCML in this configuration 
 context is only for XML masochists.

Unfortunately, it doesn't seem an option, right now, to avoid using ZCML
to setup database connections. So I'm not sure how you want to implement
your ideas?

In general, I think if you want your configuration expressed in ZConfig
and available as a utility, you have a problem.

I'm not sure of:
* How do I register a utility in ZConfig?
* How will the above registration work properly in functional tests
  (especially with tearing down and setting up layers)?

-- 
Brian Sutherland
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-11 Thread Andreas Jung



--On 11. April 2008 10:09:16 +0200 Brian Sutherland 
[EMAIL PROTECTED] wrote:



On Fri, Apr 11, 2008 at 05:07:56AM +0200, Andreas Jung wrote:

--On 10. April 2008 21:29:43 +0200 Brian Sutherland
[EMAIL PROTECTED] wrote:

What defines if something is the instance configuration?

Isn't ZCML just another format of configuration file? I agree it's evil
to hardcode database connections in the application as they must be
under the control of the sysadmin. But that doesn't preclude them from
being expressed in zcml.


The point is: we don't need yet another configuration language for the
configuration of a site. ZCML should be used for the purpose it has been
intended for: for the configuration of the components of your site
(easily  spoken). This kind of configuration is static if you setup a
site for  project X. But settings like ports, paths etc. often differ
between dev,  test and production environments.

We already have ZConfig as standard cofniguration language for instance
specific configuration aspects. In addition: we often do use environment
variables and .ini files for additional configurations. Introducing
another  configuration language is stupid. Especially because system
administrators  or co-workers know little or nothing about ZCML. This is
an unnecessary  introduction of complexity. Setting an environment
variable with a DSN is  much easier then having to fiddle around with
ZCML and XML...totally  unnecessary overheadkeep it simple ...using
ZCML in this configuration  context is only for XML masochists.


Unfortunately, it doesn't seem an option, right now, to avoid using ZCML
to setup database connections. So I'm not sure how you want to implement
your ideas?



I have no need for ideas to be implemented. We're using SA within in 
several large projects based on z3c.sqlalchemy. And nothing beats the 
simplicity

of of an .ini file or an environment variable.


In general, I think if you want your configuration expressed in ZConfig
and available as a utility, you have a problem.

I'm not sure of:
* How do I register a utility in ZConfig?


z3c.sqlalchemy provides a method for registering a SA connection as a named 
utility. This code is usually called during the startup phase of Zope 
within the related initialize() methds of our products. The DSN are 
acquired from well-known environment variables...this just works - easy, 
simple, reliable.



* How will the above registration work properly in functional tests
  (especially with tearing down and setting up layers)?


Functional tests in our company are tests done by humans. Why? Our 
applications are very complex with lots of dependencies. Not speaking of 
lots of involved databases, big storages..


Andreas



pgp98vhOQjdEm.pgp
Description: PGP signature
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-11 Thread Andreas Jung



--On 11. April 2008 08:36:43 +0100 Martin Aspeli [EMAIL PROTECTED] wrote:


Martijn Faassen wrote:

Andreas Jung wrote:

--On 10. April 2008 19:10:49 +0200 Brian Sutherland
[EMAIL PROTECTED] wrote:

Just adding my few requirements:
- Integration into the component architecture in such a way that I
  can specify the db connection parameters in ZCML and that
  database reflection still works. I want to be able to get a
  handle on
SQLAlchemy   things by calling getUtility.

Please *NO* database specific configurations within ZCML. We're running
applications in up three or four different environments and I don't
want  to maintain instance specific configurations within ZCML. Either
we pass such informations through environment variables or the database
connections parameters are read from a configuration within the
instance  configuration.


As long as it can also be specified in the code, that way people can
build UIs to set up the connection. Perhaps that's what you mean by a
configuration within the instance configuration.


The way I use collective.lead in my book is to have it look up the
database settings in a local utility. That utility is editable via a
control panel page in Plone. I suspect that it'd be quite easy to do
something similar where the settings were deferred to a global utility
configured with ZCML or looked up in a config file of some sort.

Once the config is available, it's all about calling getUtility() to get
hold of a database connection/session.



Possibly a sane and clean solution but way of over-designed. I am 
interested in simple and robust solutions that work and not in a solution 
that adds more and more unnecessary layers just the sake of having them :-)


Andreas

pgpZS98rSgYnq.pgp
Description: PGP signature
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-11 Thread Martin Aspeli
Andreas Jung lists at zopyx.com writes:

  The way I use collective.lead in my book is to have it look up the
  database settings in a local utility. That utility is editable via a
  control panel page in Plone. I suspect that it'd be quite easy to do
  something similar where the settings were deferred to a global utility
  configured with ZCML or looked up in a config file of some sort.
 
  Once the config is available, it's all about calling getUtility() to get
  hold of a database connection/session.
 
 Possibly a sane and clean solution but way of over-designed. I am 
 interested in simple and robust solutions that work and not in a solution 
 that adds more and more unnecessary layers just the sake of having them 

collective.lead doesn't enforce this style at all. If you want TTW configurable
connections, though, it's a good pattern to use.

All collective.lead does is:

 - Set up database connections and engines properly
 - Hook into transaction machinery so you don't need to do your own transaction
management
 - Let you look up a database session/connection as a utility

You need to register that utility, of course. Lead has a base class that means
you just fill in a few template methods (database url, and some callbacks for
configuring tables and orm classes).

If you want some kind of config-file based or ZCML based version, you'd just
construct an appropriate class and register it as a utility on startup. Lead
doesn't want to enforce one particular connection parameter configuration model
or config file format. That should be the job of a higher level package that is
concerned only with getting these parameters, not with the database setup 
itself.

However, you probably still need some imperative code to set up the tables and
mappers.

Martin

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-11 Thread Martijn Faassen

Brian Sutherland wrote:
[snip]

Unfortunately, it doesn't seem an option, right now, to avoid using ZCML
to setup database connections. So I'm not sure how you want to implement
your ideas?


collective.lead can do it without ZCML; you set up the right connection 
by implementing a 'url' property on a (global) utility. This to me seems 
to be flexible, as the code could get the connection from anywhere.


Regards,

Martijn

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-11 Thread Martijn Faassen

Andreas Jung wrote:
[snip]
Possibly a sane and clean solution but way of over-designed. I am 
interested in simple and robust solutions that work and not in a 
solution that adds more and more unnecessary layers just the sake of 
having them :-)


It's a 1 liner to set up the connection in this story if you want to 
hard-code it in Python, and by implementing it as a property, you can 
get the connection string from any place you want, including environment 
variables or whatever else your Python code can get to. This cuts the 
whole debate about where the connection string should come from short, 
as you can code whatever you want yourself.


This has an example of the utility setup:

http://faassen.n--tree.net/blog/view/weblog/2008/04/08/0

Regards,

Martijn

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-11 Thread Reinout van Rees

Brian Sutherland schreef:


Unfortunately, it doesn't seem an option, right now, to avoid using ZCML
to setup database connections. So I'm not sure how you want to implement
your ideas?


Something we did for an LDAP connection that needed the same kind of 
server-dependent config: put an ldapconfig.py file in zope's etc/ dir 
and load that if available from the code that configures ldap.



Not the best of solutions, probably, but it worked. What I wanted to 
say: this *is* a way to avoid using zcml!


Reinout

--
Reinout van Rees  Blog: http://vanrees.org/weblog/
[EMAIL PROTECTED]  Work: http://zestsoftware.nl/
http://vanrees.org  Video: http://reinout.blip.tv/

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-11 Thread Bernd Dorn


On 11.04.2008, at 10:09, Brian Sutherland wrote:
Unfortunately, it doesn't seem an option, right now, to avoid using  
ZCML
to setup database connections. So I'm not sure how you want to  
implement

your ideas?


+1

k, i didn't read the whole thread, but we (lovely systems) use  
zope.storm but i think its the same story somehow and we do all  
instance specific configuration in zcml - also database uris.
we have a base.zcml config for all instances and include this it and  
do all instance specific configuration in an instance specific zcml  
file. this file is then used with the site-definition otpion in  
zope.conf. i can see no reason why we should avoid zcml except in  
environments where there is no zcml at all.


although it is a few lines of code to get db uris from somewhere else,  
and this could also be application code


regards, bernd





smime.p7s
Description: S/MIME cryptographic signature
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-11 Thread Martijn Faassen

Bernd Dorn wrote:
[whether configuration should be in ZCML]
although it is a few lines of code to get db uris from somewhere else, 
and this could also be application code.


I'm not against ZCML, as long as those lines of code to get the db URI 
from application code also exist. A good API to configure things without 
ZCML is necessary (that the ZCML can then turn around and use - this is 
a good way to create ZCML directives anyway).


A nice thing about collective.lead is that it doesn't introduce custom 
ZCML directives, so in Grok, using it means there's no ZCML at all and 
no need to write any grokkers for it either. If it did introduce custom 
directives to do the same thing, that's fine, as long the original way 
isn't damaged by it.


Regards,

Martijn

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-11 Thread Stephan Richter
On Friday 11 April 2008, Bernd Dorn wrote:
 although it is a few lines of code to get db uris from somewhere else,  
 and this could also be application code

if you use paster, you can stick options into your pater ini script. We are 
currently doing this for a setting in our app.

Regards,
Stephan
-- 
Stephan Richter
Web Software Design, Development and Training
Google me. Zope Stephan Richter
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-11 Thread Brian Sutherland
On Fri, Apr 11, 2008 at 01:21:45PM +0200, Andreas Jung wrote:
 Unfortunately, it doesn't seem an option, right now, to avoid using ZCML
 to setup database connections. So I'm not sure how you want to implement
 your ideas?


 I have no need for ideas to be implemented. We're using SA within in 
 several large projects based on z3c.sqlalchemy. And nothing beats the 
 simplicity
 of of an .ini file or an environment variable.

I see. I suppose you could also write a zcml declaration to pluck out
the database config from an environment variable and register a utility.

 In general, I think if you want your configuration expressed in ZConfig
 and available as a utility, you have a problem.

 I'm not sure of:
 * How do I register a utility in ZConfig?

 z3c.sqlalchemy provides a method for registering a SA connection as a named 
 utility. This code is usually called during the startup phase of Zope 
 within the related initialize() methds of our products. The DSN are 
 acquired from well-known environment variables...this just works - easy, 
 simple, reliable.

 * How will the above registration work properly in functional tests
   (especially with tearing down and setting up layers)?

 Functional tests in our company are tests done by humans. Why? Our 
 applications are very complex with lots of dependencies. Not speaking of 
 lots of involved databases, big storages..

I try do what I can... which is relatively minimal.

At least some components are thoroughly tested with the parts of the db
schema they depend on. 

-- 
Brian Sutherland
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-10 Thread Martijn Faassen

Martin Aspeli wrote:
[snip]

It's never particularly pleasant.


I think we need to make sure we make this as easy and pleasant as 
possible though. This means at the very least some document that tells 
you what to do. :)


Regards,

Martijn

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-10 Thread Laurence Rowe

Martin Aspeli wrote:

Laurence Rowe wrote:

Should one phase commit be set as the default to make it easier to 
work with sqlite (and mssql)? Probably yes.


Ideally we'd guess based on the URL scheme but allow it to be set 
explicitly, IMHO. Single phase would be the fallback, I guess.


I don't like the idea of guessing too much, but it could be done in the 
__init__. There might be good reasons to choose one phase commit even 
for a two phase commit supporting database (i.e. when you only are 
talking to a single database, it is more efficient).


Given that a single database is the common case I am tempted to say just 
make 1PC the default.


Should the default be for sessions to start out `active` or `dirty`? 
If they start out `dirty` then existing 1.0 code should work as before.


What are the implications of having them start out dirty?


Unnecessary commits to the database for read only transactions (i.e. the 
current situation)


Laurence

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-10 Thread Laurence Rowe

Martin Aspeli wrote:

Martijn Faassen wrote:

Martijn Pieters wrote:
On Tue, Apr 8, 2008 at 11:54 PM, Martijn Faassen 
[EMAIL PROTECTED] wrote:
 All of these are in various states of brokenness. z3c.zalchemy 
doesn't work
with SQLAlchemy trunk. collective.lead works with it, but only if 
you check
out a particular branch, and not with sqlite. Quite possibly 
z3c.sqlalchemy

has a release that actually works. One out of three is not bad... :)

We are going into production with collective.lead and are committed to
releasing the 1.0 branch as 1.0. sqlite works just great for us, we
use it to run unit tests and for developers that just need to adjust
the styling and such. The production environment will run against
Oracle.


The elro-tpc branch (which, I was told, is the future) doesn't appear 
to work with sqlite out of the box.


I think this is a temporary bug, as Laurence pointed out, caused by 
SQLite not supporting two-phase commit. It should be fixed before 
release, by adding a one-phase fallback.


I'm curious how to do functional tests with collective.lead - I'd like 
to have some real easy way to get the database set up and tore down 
down in my tests. Are you doing this?


I'm doing a lot of it in Java. :) The typical pattern is to use a test 
setup that does database clean-down to return it to a steady state. Test 
data is kept in a declarative file (e.g. with an XML syntax that maps to 
DB structure) and used to initialise the database before each run.


Doing nested transactions and rollback would be nicer, but probably 
doesn't work in all cases since app code may well do explicit commits.


Note that sqlite does not support nested transactions or savepoints.



It's never particularly pleasant.

Martin



Laurence

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-10 Thread Brian Sutherland
On Wed, Apr 09, 2008 at 05:47:06PM +0200, Andreas Jung wrote:


 --On 9. April 2008 14:15:38 +0100 Laurence Rowe [EMAIL PROTECTED] wrote:

 @everyone:

 If we can all agree to use the same basic session and transaction
 management then we should probably push for it to be included as a
 sqlalchemy extension module.

 I would be happy with such a solution. As said, I am not into the 
 datamanager-building business and more interested in 
 application-integration issues. The basic requirements I have for our 
 applications:

 - having the same session within one ZODB transaction (per thread)
 - need to get hold of the underlying database connection for a given
   session

Just adding my few requirements:

- Integration into transactions.

- Integration into the component architecture in such a way that I
  can specify the db connection parameters in ZCML and that database
  reflection still works. I want to be able to get a handle on SQLAlchemy
  things by calling getUtility.

- Have a connection available as IZopeDatabaseAdapter utility (for
  BBB)

-- 
Brian Sutherland
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-10 Thread Andreas Jung



--On 10. April 2008 19:10:49 +0200 Brian Sutherland 
[EMAIL PROTECTED] wrote:





Just adding my few requirements:
- Integration into the component architecture in such a way that I
  can specify the db connection parameters in ZCML and that database
  reflection still works. I want to be able to get a handle on
SQLAlchemy   things by calling getUtility.


Please *NO* database specific configurations within ZCML. We're running 
applications in up three or four different environments and I don't want to 
maintain instance specific configurations within ZCML. Either we pass
such informations through environment variables or the database connections 
parameters are read from a configuration within the instance configuration.


Andreas

pgpPVek73cZst.pgp
Description: PGP signature
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-10 Thread Brian Sutherland
On Thu, Apr 10, 2008 at 07:31:54PM +0200, Andreas Jung wrote:


 --On 10. April 2008 19:10:49 +0200 Brian Sutherland 
 [EMAIL PROTECTED] wrote:



 Just adding my few requirements:
 - Integration into the component architecture in such a way that I
   can specify the db connection parameters in ZCML and that database
   reflection still works. I want to be able to get a handle on
 SQLAlchemy   things by calling getUtility.

 Please *NO* database specific configurations within ZCML. We're running 
 applications in up three or four different environments and I don't want to 
 maintain instance specific configurations within ZCML. Either we pass
 such informations through environment variables or the database connections 
 parameters are read from a configuration within the instance configuration.

What defines if something is the instance configuration?

Isn't ZCML just another format of configuration file? I agree it's evil
to hardcode database connections in the application as they must be
under the control of the sysadmin. But that doesn't preclude them from
being expressed in zcml.

I *do* like the idea of using one and only one file format to configure
applications.

We also run the in a few different environments, the
/etc/myserver/site.zcml is different on a per instance basis and looks
like this:

configure xmlns=http://namespaces.zope.org/zope;
   xmlns:rdb=http://namespaces.zope.org/rdb;

  include package=mymodule file=configure.zcml /

  rdb:provideConnection
  name=admin
  component=psycopg2da.adapter.Psycopg2Adapter
  dsn=dbi://prototype /

/configure

But, I'm diverging, my point is that it would be nice to be able to have
SQLAlchemy connections available as IZopeDatabaseAdapter utilities to
allow a migration path.  Whether they are defined in or out of ZCML, I
just don't care.

-- 
Brian Sutherland
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-10 Thread Brian Sutherland
On Thu, Apr 10, 2008 at 09:29:43PM +0200, Brian Sutherland wrote:
 On Thu, Apr 10, 2008 at 07:31:54PM +0200, Andreas Jung wrote:
  Please *NO* database specific configurations within ZCML. We're running 
  applications in up three or four different environments and I don't want to 
  maintain instance specific configurations within ZCML. Either we pass
  such informations through environment variables or the database connections 
  parameters are read from a configuration within the instance configuration.

I must admit though, that being able to set an environment variable and
have an IZopeDatabaseAdapter utility registered from that would make
setting up functional tests much nicer.

Currently, because of layers, all my functional tests run against a
magical database test :(((

-- 
Brian Sutherland
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-10 Thread Kapil Thangavelu
On Thu, Apr 10, 2008 at 1:31 PM, Andreas Jung [EMAIL PROTECTED] wrote:



 --On 10. April 2008 19:10:49 +0200 Brian Sutherland 
 [EMAIL PROTECTED] wrote:



  Just adding my few requirements:
 - Integration into the component architecture in such a way that I
   can specify the db connection parameters in ZCML and that database
   reflection still works. I want to be able to get a handle on
  SQLAlchemy   things by calling getUtility.
 

 Please *NO* database specific configurations within ZCML. We're running
 applications in up three or four different environments and I don't want to
 maintain instance specific configurations within ZCML. Either we pass
 such informations through environment variables or the database
 connections parameters are read from a configuration within the instance
 configuration.



i still don't see the harm having zcml db conf  in a base package, if you
don't have to use it to setup your particular deployment's connections.

-kapil
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-10 Thread Martijn Faassen

Andreas Jung wrote:
--On 10. April 2008 19:10:49 +0200 Brian Sutherland 
[EMAIL PROTECTED] wrote:

Just adding my few requirements:
- Integration into the component architecture in such a way that I
  can specify the db connection parameters in ZCML and that database
  reflection still works. I want to be able to get a handle on
SQLAlchemy   things by calling getUtility.


Please *NO* database specific configurations within ZCML. We're running 
applications in up three or four different environments and I don't want 
to maintain instance specific configurations within ZCML. Either we pass
such informations through environment variables or the database 
connections parameters are read from a configuration within the instance 
configuration.


As long as it can also be specified in the code, that way people can 
build UIs to set up the connection. Perhaps that's what you mean by a 
configuration within the instance configuration.


Regards,

Martijn

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-10 Thread Andreas Jung



--On 10. April 2008 21:29:43 +0200 Brian Sutherland 
[EMAIL PROTECTED] wrote:



On Thu, Apr 10, 2008 at 07:31:54PM +0200, Andreas Jung wrote:



--On 10. April 2008 19:10:49 +0200 Brian Sutherland
[EMAIL PROTECTED] wrote:




Just adding my few requirements:
- Integration into the component architecture in such a way that I
  can specify the db connection parameters in ZCML and that database
  reflection still works. I want to be able to get a handle on
SQLAlchemy   things by calling getUtility.


Please *NO* database specific configurations within ZCML. We're running
applications in up three or four different environments and I don't want
to  maintain instance specific configurations within ZCML. Either we pass
such informations through environment variables or the database
connections  parameters are read from a configuration within the
instance configuration.


What defines if something is the instance configuration?

Isn't ZCML just another format of configuration file? I agree it's evil
to hardcode database connections in the application as they must be
under the control of the sysadmin. But that doesn't preclude them from
being expressed in zcml.


The point is: we don't need yet another configuration language for the 
configuration of a site. ZCML should be used for the purpose it has been 
intended for: for the configuration of the components of your site (easily 
spoken). This kind of configuration is static if you setup a site for 
project X. But settings like ports, paths etc. often differ between dev, 
test and production environments.


We already have ZConfig as standard cofniguration language for instance 
specific configuration aspects. In addition: we often do use environment 
variables and .ini files for additional configurations. Introducing another 
configuration language is stupid. Especially because system administrators 
or co-workers know little or nothing about ZCML. This is an unnecessary 
introduction of complexity. Setting an environment variable with a DSN is 
much easier then having to fiddle around with ZCML and XML...totally 
unnecessary overheadkeep it simple ...using ZCML in this configuration 
context is only for XML masochists.


Andreas

pgpyigxtKwGmK.pgp
Description: PGP signature
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-09 Thread Martin Aspeli

Martijn Faassen wrote:

Hi there,

[I originally picked this up on a thread on zope3-users, but this 
deserves its own thread here]


There are at least three approaches to SQLAlchemy integration with Zope:

* z3c.zalchemy (Christian Theune)

* z3c.sqlalchemy (Andreas Jung)

* collective.lead (Laurence Rowe)

All of these are in various states of brokenness. z3c.zalchemy doesn't 
work with SQLAlchemy trunk. collective.lead works with it, but only if 
you check out a particular branch, and not with sqlite. Quite possibly 
z3c.sqlalchemy has a release that actually works. One out of three is 
not bad... :)


Then there's also mentions about WSGI-based integration, and I think in 
Plone, Alchemist probably also does its own integration...


There must be a reason for this proliferation of approaches. What is it? 
  We all get along, don't we? I know that the various packages are 
taking code and approaches from each other too.


Can't we work together more and at least come up with *one* package that 
works? Perhaps factor out some low-level commonality than then all 
share? Criticize one of the other packages until you're satisfied, and 
then retire your own package perhaps? I know the various packages add on 
their own approaches to configuration and might offer higher level 
container approaches. Those could be in different packages, sharing a 
foundation.


In the end, I hope we will end up with just *one* integration layer, 
that is released, that works with Zope 2 and Zope 3 and a recent release 
of SQLAlchemy, that is documented, and that people know about. We can 
then offer packages on top of this that offer extra features.


I'm all for one integration layer, even though I spurred one of the 
above (collective.lead). I would ask that we be very conservative, 
though. collective.lead wants to provide:


 - automatic transaction integration with ZODB transactions
 - an easy way to register your database connections
 - an easy way to look those up as utilities and get an ORM session

Everything else (e.g. Zope ORM integration, configuration abstractions, 
etc) should be in some other package.


collective.lead has a reasonable degree of traction in the Plone 
universe, at least, but should work just fine with plain Zope 3. We have 
other things that work on top it (mercury, rope, probably others).


The branch that works with trunk also has some carefully worked out TPC 
support, which is difficult to get right, though I'm disappointed that 
we seem to have broken sqlite integration. Hopefully it's an easy fix, 
though.


I'm CC'ing Laurence to ask:

 - can we get a release soon?
 - can we fix the sqlite integration?

Cheers,
Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-09 Thread Laurence Rowe

The transaction manager here:

http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collective/lead/tx.py

Has support for TPC and savepoints. It is tested and works. The only 
issue with sqlite is that the default in the branch is to use two-phase 
commit and sqlite does not support this. Apart from that the module is 
completely self contained.


The public interfaces to it is the two functions at the bottom, 
join_transaction and dirty_session. The idea of the dirty_session stuff 
is to prevent unnecessary commits when data is only read. Session writes 
 trigger the 'dirtying' automatically with a SessionExtension:


http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collective/lead/database.py

The downside of this approach is that the when you interact with the 
connection directly to modify data you must call dirty_session yourself. 
If this is a problem for you then you may set up the transaction to be 
in the dirty state initially.


Thread based session management is used (scoped_session). There is 
fairly extensive testing of this and the transaction support here:


http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collective/lead/tests.py

I think it would be worth factoring out the threaded session support as 
well as the transactions.


@Christian:

I'm not quite sure what you mean by:

- Database triggers as events

I don't see how you could reasonably get a database trigger on the db 
server to instantiate a zope event on the db client. Perhaps the 
database could be persuaded to make a request to a URL on a trigger, but 
this seems a bit slow and maybe not a great solution.


- Making SA 0.4 work with security proxies

I don't know anything about security proxies in zope 3, but surely this 
is just a matter of what the base class is / the container 
implementation? I don't see why an SQL Alchemy mapped object would be 
any different to any other object.


@Martin:

Should one phase commit be set as the default to make it easier to work 
with sqlite (and mssql)? Probably yes.


Should the default be for sessions to start out `active` or `dirty`? If 
they start out `dirty` then existing 1.0 code should work as before.


It'll probably be next week before I manage to get this tidied up and 
merged back to trunk.


@everyone:

If we can all agree to use the same basic session and transaction 
management then we should probably push for it to be included as a 
sqlalchemy extension module.



Laurence

Martin Aspeli wrote:

Martijn Faassen wrote:

Hi there,

[I originally picked this up on a thread on zope3-users, but this 
deserves its own thread here]


There are at least three approaches to SQLAlchemy integration with Zope:

* z3c.zalchemy (Christian Theune)

* z3c.sqlalchemy (Andreas Jung)

* collective.lead (Laurence Rowe)

All of these are in various states of brokenness. z3c.zalchemy doesn't 
work with SQLAlchemy trunk. collective.lead works with it, but only if 
you check out a particular branch, and not with sqlite. Quite possibly 
z3c.sqlalchemy has a release that actually works. One out of three is 
not bad... :)


Then there's also mentions about WSGI-based integration, and I think 
in Plone, Alchemist probably also does its own integration...


There must be a reason for this proliferation of approaches. What is 
it?   We all get along, don't we? I know that the various packages are 
taking code and approaches from each other too.


Can't we work together more and at least come up with *one* package 
that works? Perhaps factor out some low-level commonality than then 
all share? Criticize one of the other packages until you're satisfied, 
and then retire your own package perhaps? I know the various packages 
add on their own approaches to configuration and might offer higher 
level container approaches. Those could be in different packages, 
sharing a foundation.


In the end, I hope we will end up with just *one* integration layer, 
that is released, that works with Zope 2 and Zope 3 and a recent 
release of SQLAlchemy, that is documented, and that people know about. 
We can then offer packages on top of this that offer extra features.


I'm all for one integration layer, even though I spurred one of the 
above (collective.lead). I would ask that we be very conservative, 
though. collective.lead wants to provide:


 - automatic transaction integration with ZODB transactions
 - an easy way to register your database connections
 - an easy way to look those up as utilities and get an ORM session

Everything else (e.g. Zope ORM integration, configuration abstractions, 
etc) should be in some other package.


collective.lead has a reasonable degree of traction in the Plone 
universe, at least, but should work just fine with plain Zope 3. We have 
other things that work on top it (mercury, rope, probably others).


The branch that works with trunk also has some carefully worked out TPC 
support, which is difficult to get right, 

[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-09 Thread Martijn Faassen

Martijn Pieters wrote:

On Tue, Apr 8, 2008 at 11:54 PM, Martijn Faassen [EMAIL PROTECTED] wrote:

 All of these are in various states of brokenness. z3c.zalchemy doesn't work
with SQLAlchemy trunk. collective.lead works with it, but only if you check
out a particular branch, and not with sqlite. Quite possibly z3c.sqlalchemy
has a release that actually works. One out of three is not bad... :)


We are going into production with collective.lead and are committed to
releasing the 1.0 branch as 1.0. sqlite works just great for us, we
use it to run unit tests and for developers that just need to adjust
the styling and such. The production environment will run against
Oracle.


The elro-tpc branch (which, I was told, is the future) doesn't appear to 
work with sqlite out of the box.


I'm curious how to do functional tests with collective.lead - I'd like 
to have some real easy way to get the database set up and tore down down 
in my tests. Are you doing this?


Regards,

Martijn

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-09 Thread Andreas Jung



--On 9. April 2008 14:15:38 +0100 Laurence Rowe [EMAIL PROTECTED] wrote:


@everyone:

If we can all agree to use the same basic session and transaction
management then we should probably push for it to be included as a
sqlalchemy extension module.


I would be happy with such a solution. As said, I am not into the 
datamanager-building business and more interested in 
application-integration issues. The basic requirements I have for our 
applications:


- having the same session within one ZODB transaction (per thread)
- need to get hold of the underlying database connection for a given
  session

Andreas

pgpFEZVYm6t1t.pgp
Description: PGP signature
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-09 Thread Kapil Thangavelu
On Wed, Apr 9, 2008 at 9:15 AM, Laurence Rowe [EMAIL PROTECTED] wrote:

 The transaction manager here:


 http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collective/lead/tx.py

 Has support for TPC and savepoints. It is tested and works. The only issue
 with sqlite is that the default in the branch is to use two-phase commit and
 sqlite does not support this. Apart from that the module is completely self
 contained.


 The public interfaces to it is the two functions at the bottom,
 join_transaction and dirty_session. The idea of the dirty_session stuff is
 to prevent unnecessary commits when data is only read. Session writes
  trigger the 'dirtying' automatically with a SessionExtension:



 http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collective/lead/database.py

 The downside of this approach is that the when you interact with the
 connection directly to modify data you must call dirty_session yourself. If
 this is a problem for you then you may set up the transaction to be in the
 dirty state initially.

 Thread based session management is used (scoped_session). There is fairly
 extensive testing of this and the transaction support here:


 http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collective/lead/tests.py

 I think it would be worth factoring out the threaded session support as
 well as the transactions.


+1, the collective.lead transaction support looks nice.




 @Christian:

 I'm not quite sure what you mean by:

 - Database triggers as events

 I don't see how you could reasonably get a database trigger on the db
 server to instantiate a zope event on the db client. Perhaps the database
 could be persuaded to make a request to a URL on a trigger, but this seems a
 bit slow and maybe not a great solution.

 - Making SA 0.4 work with security proxies

 I don't know anything about security proxies in zope 3, but surely this is
 just a matter of what the base class is / the container implementation? I
 don't see why an SQL Alchemy mapped object would be any different to any
 other object.


its more to do with the fact that proxies are restrictive by default, and
the sa api likes to monkey attributes on instances. christian kicked off
some nice work with the sa developers at pycon to get this behavior
configurable in sa ( instrumentation branch) which obviate the need to
remove the proxies before calling into sa apis on an object.




 @Martin:

 Should one phase commit be set as the default to make it easier to work
 with sqlite (and mssql)? Probably yes.

 Should the default be for sessions to start out `active` or `dirty`? If
 they start out `dirty` then existing 1.0 code should work as before.

 It'll probably be next week before I manage to get this tidied up and
 merged back to trunk.

 @everyone:

 If we can all agree to use the same basic session and transaction
 management then we should probably push for it to be included as a
 sqlalchemy extension module.


sounds good, or failing that (due to additional dependencies), a common base
package.

cheers,

kapil
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-09 Thread Martin Aspeli

Laurence Rowe wrote:

Should one phase commit be set as the default to make it easier to work 
with sqlite (and mssql)? Probably yes.


Ideally we'd guess based on the URL scheme but allow it to be set 
explicitly, IMHO. Single phase would be the fallback, I guess.


Should the default be for sessions to start out `active` or `dirty`? If 
they start out `dirty` then existing 1.0 code should work as before.


What are the implications of having them start out dirty?

It'll probably be next week before I manage to get this tidied up and 
merged back to trunk.


Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-09 Thread Martin Aspeli

Martijn Faassen wrote:

Martijn Pieters wrote:

On Tue, Apr 8, 2008 at 11:54 PM, Martijn Faassen [EMAIL PROTECTED] wrote:

 All of these are in various states of brokenness. z3c.zalchemy doesn't work
with SQLAlchemy trunk. collective.lead works with it, but only if you check
out a particular branch, and not with sqlite. Quite possibly z3c.sqlalchemy
has a release that actually works. One out of three is not bad... :)

We are going into production with collective.lead and are committed to
releasing the 1.0 branch as 1.0. sqlite works just great for us, we
use it to run unit tests and for developers that just need to adjust
the styling and such. The production environment will run against
Oracle.


The elro-tpc branch (which, I was told, is the future) doesn't appear to 
work with sqlite out of the box.


I think this is a temporary bug, as Laurence pointed out, caused by 
SQLite not supporting two-phase commit. It should be fixed before 
release, by adding a one-phase fallback.


I'm curious how to do functional tests with collective.lead - I'd like 
to have some real easy way to get the database set up and tore down down 
in my tests. Are you doing this?


I'm doing a lot of it in Java. :) The typical pattern is to use a test 
setup that does database clean-down to return it to a steady state. Test 
data is kept in a declarative file (e.g. with an XML syntax that maps to 
DB structure) and used to initialise the database before each run.


Doing nested transactions and rollback would be nicer, but probably 
doesn't work in all cases since app code may well do explicit commits.


It's never particularly pleasant.

Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-09 Thread Martin Aspeli

Andreas Jung wrote:


--On 9. April 2008 14:15:38 +0100 Laurence Rowe [EMAIL PROTECTED] wrote:


@everyone:

If we can all agree to use the same basic session and transaction
management then we should probably push for it to be included as a
sqlalchemy extension module.


I would be happy with such a solution. As said, I am not into the 
datamanager-building business and more interested in 
application-integration issues. The basic requirements I have for our 
applications:


 - having the same session within one ZODB transaction (per thread)
 - need to get hold of the underlying database connection for a given
   session


I think that's very much the business collective.lead wants to be in. :)

Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: straighting out the SQLAlchemy integration mess

2008-04-09 Thread Andreas Jung



--On 10. April 2008 01:57:12 +0100 Martin Aspeli [EMAIL PROTECTED] wrote:


Andreas Jung wrote:


--On 9. April 2008 14:15:38 +0100 Laurence Rowe [EMAIL PROTECTED] wrote:


@everyone:

If we can all agree to use the same basic session and transaction
management then we should probably push for it to be included as a
sqlalchemy extension module.


I would be happy with such a solution. As said, I am not into the
datamanager-building business and more interested in
application-integration issues. The basic requirements I have for our
applications:

 - having the same session within one ZODB transaction (per thread)
 - need to get hold of the underlying database connection for a given
   session


I think that's very much the business collective.lead wants to be in. :)



You know what was first :-) Even with the collective.lead and z3c.zalchemy 
datamanager...still two instead of one :-)


-aj

pgpBydBlV3U6n.pgp
Description: PGP signature
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )