Re: Turbogears now based on Pylons!

[Mike Orr]

On 6/27/07, Michael Bayer [EMAIL PROTECTED] wrote:
  Of course the engine must be initialized before any tables can be
  defined or any work done.

 since ive seen people get the wrong idea about this, let me
 reiterate:  *you can define all your tables and mappers with no
 database connection defined whatsoever*,

Not if you're using autoloading or want to be able to switch
autoloading on or off for any table without having to modify the rest
of the application.  To me that's important, especially when you have
to start changing the schema of an existing table (adding a field,
changing its type, etc), or if the schema is modified externally to
the application.

  SAContext initializes the default engine in
  the constructor.  Is there a need to push that back to some later
  stage?  Which stage?  Or can Ian's ideal wrapper simply delay creating
  the 'sac' until it has to?

 easy enough to make uri optional in SAContext (the way it is for
 everything else in SA) and just allow binding it later via
 add_engine() or whatever.   SAContext is designed the way it is right
 now because Pylons defines the default engine in the .ini file; theres
 no need for a delayed connection in the usual case.

I would probably want this optional URI in a subclass rather than in
SAContext itself.  The reason 'uri' is a required argument is to
guarantee that the default engine is initialized at all times.  When
we designed SAContext we (Mike  I) agreed that the bound metadata
strategy was the most straightforward and adequate for most user apps:
the 80/20 rule.  Hiding the engine (connectable) as much as possible
in the metadata, while still making it accessible when you really need
it.  Now we're adding one unbound metadata strategy after another.
That's fine as log as it doesn't detract from SAContex's primary
commitment to its primary userbase.  I'd say a mandatory URI and bound
metadata is part of its primary commitment for an easy-to-use front
end: a normal Python TypeError for You forgot the 'uri' argument is
about as straightforward as one can get.  Or is that changing too?
Are you having second thoughts about using bound metadata by default?

-- 
Mike Orr [EMAIL PROTECTED]

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Mike Orr]

On 6/27/07, Uwe C. Schroeder [EMAIL PROTECTED] wrote:
  if you
  really want COMMIT for every SELECT, i'd favor it being enabled via an
  option passed to create_engine().

 Not every select, every transaction that didn't roll back.
 I just think the default of rollback on every transaction is wrong - a
 rollback should occur when there is a problem, not when the transaction was
 fine. But that may just be me.

But the framework doesn't know whether there's a problem, or whether
some uncommitted changes exist (maybe you called a function that
changed the database but didn't commit).  It certainly doesn't know
whether it should commit or roll back these changes if they do
exist.   But it DOES know that stray uncommitted changes should not
leak into the next web request or all hell will break loose.  So it
has to commit or roll back.  Rolling back is the prudent thing because
if the function really wanted to save those changes permanently, well,
it should've committed them.

  Beyond that I think the model
  implemented as stored procedure style of development is much in the
  minority these days, particularly within the lightweight/open-source
  development community.   just that the stored procedure changes the
  semantics of SELECT to be a write operation reminds me of the
  RESTful sin of using GET to post data.

 Probably because a lot of people can't figure out how to use stored procedures
 and triggers, since the lightweight/open-source programming is often done
 on a database that has very limited support for both :-)

 Personally I'm not a big fan of handling database integrity outside the
 database. Take a simple portal as example, where every new user who signs up
 gets a couple of mailboxes by default (inbox, outbox,drafts,trash). I could
 handle that in the db structure itself, but that would make the db model
 overly complicated for such a simple thing. The proper way is to handle that
 in a trigger, or I can handle it in a stored procedure that just makes sure
 the mailboxes are created. In my case I chose a stored procedure because it's
 actually a select on the user's profile, but the stored procedure also does
 certain sanity checks and inserts/updates stuff according to its findings.

 Handling the same in the application code is IMHO the least desirable
 solution, simply because I can test the trigger/stored procedure once and I
 know everything works until I change the database structure. Application code
 is meddled around with much more, so the chance of introducing a bug is
 higher on that end.
 You could argue that using SA with stored procedures defeats the portability
 approach. I agree that you can't easily move a Oracle or PostgreSQL or DB2
 based application with stored procedures to a different database, but how
 many people do that? If you build a generic application the argument is
 valid, but if you build something for a specific purpose, the database will
 never be changed for the life of the application (because generally special
 purpose applications are funded by someone and those people usually don't
 invest another couple grand just to make it work with some other database).

I'd argue the opposite, that people don't use stored procedures and
triggers because that ties them to a certain database, and that's the
opposite of why they chose SQLAlchemy in the first place.  Not only
that, you have to learn a second language that's limited to that
database, with its own quirks and limitations, and the language is
much less convenient/featureful than Python (it looks like a fossil
from 20 years ago), which is why we're using Python in the first
place.  I would be more likely to change my database than change my
application, or at least just as likely.  It's not that big a deal to
copy everything into a new database and adjust the program for it --
provided you don't have huge databases or stored procedures.  The
huge database part is outside your control, but the stored procedure
part is.  Tying myself to one database product gives me bad memories
of the vendor lock-in problems that used to be more prevelant in the
past, but are thankfully becoming rarer now that interoperability has
gotten better.

I actually have a practical example of this.  I may have to switch one
application from MySQL to PostgreSQL in order to access its PostGIS
geographical functions, which I hear are more advanced than MySQL's.
So if we decide we need this functionality we'll have to switch.
Without stored procedures I can just change my DBURI string and voila,
just write the SQL expressions.  With stored procedures I'd have to
port all those to the new database.

Having said all this, I have had to break down and start using views
because it made performance acceptable in one case (a mapped object on
a SELECT was just too slow).  Someday I may have to write stored
procedures for that reason.  But so far I haven't needed to.

-- 
Mike Orr [EMAIL PROTECTED]

Re: Turbogears now based on Pylons!

[Uwe C. Schroeder]

On Wednesday 27 June 2007, Mike Orr wrote:
 On 6/27/07, Uwe C. Schroeder [EMAIL PROTECTED] wrote:
   if you
   really want COMMIT for every SELECT, i'd favor it being enabled via an
   option passed to create_engine().
 
  Not every select, every transaction that didn't roll back.
  I just think the default of rollback on every transaction is wrong - a
  rollback should occur when there is a problem, not when the transaction
  was fine. But that may just be me.

 But the framework doesn't know whether there's a problem, or whether
 some uncommitted changes exist (maybe you called a function that
 changed the database but didn't commit).  It certainly doesn't know
 whether it should commit or roll back these changes if they do
 exist.   But it DOES know that stray uncommitted changes should not
 leak into the next web request or all hell will break loose.  So it
 has to commit or roll back.  Rolling back is the prudent thing because
 if the function really wanted to save those changes permanently, well,
 it should've committed them.

If it could, yes. Right now it looks to me like SA automatically rolls every 
select back without a way to tell it not to (and no, explicit transaction 
didn't do the trick, but then I'm not an expert on SA, so there may be a 
better way)

   Beyond that I think the model
   implemented as stored procedure style of development is much in the
   minority these days, particularly within the lightweight/open-source
   development community.   just that the stored procedure changes the
   semantics of SELECT to be a write operation reminds me of the
   RESTful sin of using GET to post data.
 
  Probably because a lot of people can't figure out how to use stored
  procedures and triggers, since the lightweight/open-source programming
  is often done on a database that has very limited support for both :-)
 
  Personally I'm not a big fan of handling database integrity outside the
  database. Take a simple portal as example, where every new user who signs
  up gets a couple of mailboxes by default (inbox, outbox,drafts,trash). I
  could handle that in the db structure itself, but that would make the db
  model overly complicated for such a simple thing. The proper way is to
  handle that in a trigger, or I can handle it in a stored procedure that
  just makes sure the mailboxes are created. In my case I chose a stored
  procedure because it's actually a select on the user's profile, but the
  stored procedure also does certain sanity checks and inserts/updates
  stuff according to its findings.
 
  Handling the same in the application code is IMHO the least desirable
  solution, simply because I can test the trigger/stored procedure once and
  I know everything works until I change the database structure.
  Application code is meddled around with much more, so the chance of
  introducing a bug is higher on that end.
  You could argue that using SA with stored procedures defeats the
  portability approach. I agree that you can't easily move a Oracle or
  PostgreSQL or DB2 based application with stored procedures to a different
  database, but how many people do that? If you build a generic application
  the argument is valid, but if you build something for a specific purpose,
  the database will never be changed for the life of the application
  (because generally special purpose applications are funded by someone
  and those people usually don't invest another couple grand just to make
  it work with some other database).

 I'd argue the opposite, that people don't use stored procedures and
 triggers because that ties them to a certain database, and that's the
 opposite of why they chose SQLAlchemy in the first place.  Not only
 that, you have to learn a second language that's limited to that
 database, with its own quirks and limitations, and the language is
 much less convenient/featureful than Python (it looks like a fossil
 from 20 years ago), which is why we're using Python in the first
 place.  I would be more likely to change my database than change my
 application, or at least just as likely.  It's not that big a deal to
 copy everything into a new database and adjust the program for it --
 provided you don't have huge databases or stored procedures.  The
 huge database part is outside your control, but the stored procedure
 part is.  Tying myself to one database product gives me bad memories
 of the vendor lock-in problems that used to be more prevelant in the
 past, but are thankfully becoming rarer now that interoperability has
 gotten better.

Interoperability has gotten better? A bit probably, but not all that much :-(
On the other hand, if you're as likely to change your DB model as you are to 
modify your application, your app and db model are either very simple, or the 
db model wasn't designed all that well. My current application has 513 tables 
and been in production for about 5 years. The application running on top of 
it was changed numerous times, the DB 

Re: Turbogears now based on Pylons!

[Wichert Akkerman]

Previously Michael Bayer wrote:
 On Jun 27, 6:42 pm, Ian Bicking [EMAIL PROTECTED] wrote:
  The way I see this working is something like (vaguely):
 
  def transaction_middleware(app):
   def wrapper(environ, start_response):
   manager = TransactionManager()
   environ['transaction.manager'] = manager
   try:
   app_iter = app(environ, start_response)
   except:
   manager.rollback()
   raise
   else:
   manager.commit()
 
  The manager is basically a container of *actual* transactions, and
  calling rollback or commit on it gets passed on to all the transactions
  in the manager.
 
 this is fine.  we're *really* starting to imitate J2EE in some ways.
 but its not a bad thing.

For what it's worth: Zope has a similar structure. The Zope publisher
starts a transaction when it receives a request and commits it when the
request processing has finished, unless an exception has been thrown or
someone did an explicit rollback. Rollback tends to be very rare so
it seems that it is not very useful to optimize for that case. The Zope
machinery allows you to add your own commit/rollback hooks, which is
used to tie SA into Zope.

Wichert.

-- 
Wichert Akkerman [EMAIL PROTECTED]It is simple to make things.
http://www.wiggy.net/   It is hard to make things simple.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Uwe C. Schroeder]

On Thursday 28 June 2007, Mike Orr wrote:

 And this.  If you have a 50-column table, by gosh SQLAlchemy is going
 to select all 50 fields in full `TableName`.columnName AS columnName
 notation, which makes the query hard to read in the log.

M - a 50 column table? My old CS teacher will wiggle in his grave like a 
nightcrawler. What happened to normalizing tables? :-)
But then - I've done pretty awful things myself :-)

Uwe

--
Open Source Solutions 4U, LLC   1618 Kelly St
Phone:  +1 707 568 3056 Santa Rosa, CA 95401
Cell:   +1 650 302 2405 United States
Fax:+1 707 568 6416

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Jonathan LaCour]

Michael Bayer wrote:

 im not exactly sure why Jonathan thinks that the url forces one
 to define my connection up front. if youre saying, i want to
 distribute the application without an .ini file thats fine...but when
 the thing is actually *used*, there will be an .ini file or other
 configuration defined when things run. if you want to define your
 model independently of connection, just use an empty MetaData.  Set
 up the SAContext to bind the engine or individual connections to the
 session instead...or give it your MetaData for it to bind when youre
 ready. theres really nothing to it.

I just didn't like having my model get its URI directly from the Pylons
config, for the reasons I stated previously.  However, you just solved
my issue :) I had no idea that I was able to do this with SAContext, and
effectively use it just as I want to.

Thanks for the explanation Mike!

--
Jonathan LaCour
http://cleverdevil.org



--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

'Form' Authentication Issues with AuthKit and Turbogears

[Ryan Petrello]

I'm currently making an attempt to wrap a Turbogears app in a simple
authentication layer using AuthKit, but I am running into a problem
that's got me stumped.  Here's a snippet from my start-project.py:

def valid(environ, username, password):
return (username=='ryan' and password=='secret')

wrappedApp = middleware(
wsgiApp,
method='form',
cookie_secret='BigSecret',
catch='*',
users_valid=valid
)

# Start the wsgi server
server = CherryPyWSGIServer(('127.0.0.1', 8123), wrappedApp)
server.start()

As you can see, the authentication I've got here mimics one of the
most basic examples in the AuthKit manual, and attempts to perform
simple form authentication.  As I start the server up, everything
appears to work correctly.  I can open my browser and navigate to a
page and am presented with the default Please Log In! form.  If I
enter details (correct or incorrect) and submit the form however, I
get nothing.  Checking the console, I find that an error has occurred,
and this error is causing the rest of the script to stop execution,
resulting in a timeout.  Here's the traceback that I receive in the
console:

Traceback (most recent call last):
  File c:\python24\lib\site-packages\CherryPy-2.2.1-py2.4.egg\cherrypy
\_cpwsgiserver.py, line 201, in run
request.start_response)
  File c:\documents and settings\rpetrello\desktop\authkit\authkit
\authenticate\auth_tkt.py, line 285, in __call__
return self.app(environ, cookie_setting_start_response)
  File c:\documents and settings\rpetrello\desktop\authkit\authkit
\authenticate\multi.py, line 52, in __call__
result = check()
  File c:\documents and settings\rpetrello\desktop\authkit\authkit
\authenticate\multi.py, line 41, in check
return binding(environ, start_response)
  File c:\documents and settings\rpetrello\desktop\authkit\authkit
\authenticate\form.py, line 47, in __call__
formvars = parse_formvars(environ, include_get_vars=False)
  File C:\Python24\lib\site-packages\paste-1.3-py2.4.egg\paste
\request.py, line 174, in parse_formvars
keep_blank_values=1)
  File C:\Python24\lib\cgi.py, line 528, in __init__
self.read_urlencoded()
  File C:\Python24\lib\cgi.py, line 633, in read_urlencoded
qs = self.fp.read(self.length)
  File C:\Python24\lib\socket.py, line 295, in read
data = self._sock.recv(recv_size)
timeout: timed out

I've attempted other forms of authentication provided by AuthKit (such
as basic HTTP and digest) and they work perfectly fine.
The AuthKit that I have is the latest development source from the SVN
trunk at authkit.org.
The Turbogears that I'm running is version 1.0.2.2.

Any help or insight is much appreciated!


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Jonathan LaCour]

Michael Bayer wrote:

 just to clarify, however this works, it should be *really easy* for
 individual controller methods to be marked as transactional or not.
 I have written decorators like these already, in the style of:

 class MyController(...):
  def index_page(self):
  .

  @transactional
  def post_message(self):
  .

  def display_message(self):
  .

 so id want that kind of thing to be readily available, i suppose it
 would communicate with the WSGI middleware on a per-request basis.

The only issue I can see with this is that having the transaction begin
in the controller method itself might be too late if you have other
things going on in middleware or elsewhere and something goes wrong
before you even get to the controller method -- say in object dispatch,
the new TG controller's lookup method, or even some middleware that
stores sessions in a database for example.

Transactions should start at the earliest possible point, IMO, so that
you can handle all of these cases.

 but another thing i like, is that of the response to the request being
 delivered *after* the transaction commits. i have raised this issue
 on the pylons list before but i dont think i impressed anyone. if you
 render your response template, then your transaction fails, you have
 the potential for that response template to still be delivered with
 the success! message. anyway, might be nice for the middleware to
 address this if possible.

I am totally with you here, as this is the other side of the coin that
I brought up above.  Open transactions shouldn't be committed until the
latest possible point.

I spent some dark days working on J2EE applications, and while there
isn't a whole lot that is worth emulating in J2EE, the transaction
handling is something that I actually liked.

 also i think this is all outside of the scope of SAContext.  SAContext
 should remain as a facade to SA-specific elements; it can accept
 strategy objects which control its internal workings, so if need be
 particular strategies can be delivered in order to function with the
 transaction manager, without changing the external view of SAContext.

Yes, I think agree here too, as long as there is a way to hook SAContext
into whatever ends up getting decided in this (really long) thread.  I
played a bit more with SAContext after these discussions, and it is
indeed very nice.  Good work Mike and Mike.


--
Jonathan LaCour
http://cleverdevil.org


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Michael Bayer]

On Jun 28, 1:40 am, Uwe C. Schroeder [EMAIL PROTECTED] wrote:
 On Wednesday 27 June 2007, Michael Bayer wrote:

  this issue can be worked around by using explicit transactions.

 actually no, it can't. Maybe I don't get it right, but the only way for me to
 get a commit was actually to modify Connection._autocommit in
 sqlalchemy.engine.base.

either TG is getting in the way, or youre not getting it right.  if
anyone ever needs to modify the internals of SA to get something to
work, i would *highly* prefer if they could email the ML or post a
trac ticket with their issue so that it may be handled properly.


 Obviously SA thinks there is no transaction in TG,
 so it just wraps one around it.

if TG actually has a transaction going on, theyd certainly have to
configure SA to be aware of it (most likely via SessionTransaction).
if not, then yes things arent going to work at all (though still, an
explicit SA transaction should work all by itself).

 I agree, and that is certainly DB dependent. Personally I can't imagine that
 an automatically issued rollback for every select transaction is in any way
 more overhead than issuing a commit. Not wrapping a select in a transaction
 will definitely be the least overhead.

we dont issue a rollback for every select transaction.  we issue a
rollback when a connection is returned to the pool.  you can check out
a connection explicitly and perform any number of selects on it
without any rollbacks or commits.

because the rollback is at the connection-pool checkin level, it
should be more apparent how inappropriate it would be to issue a
*commit* every time a connection is returned to the pool, an operation
that knows nothing about what just happened with that connection.  the
rollback is to release database locks.

im thinking that it might be time to allow an option in SA that just
turns the DBAPI's autocommit flag on.  that way you can just blame
the DBAPI for whatever issues arise.

its not always possible to not wrap a select in a transaction.
oracle for example *always* has a transaction going on, so everything
is in a transaction in all cases.

  that a stored-procedure-oriented application is far more efficient
  is *extremely* debatable and database-dependent as well.

 I doubt it's *extremely* debatable.

its extremely debatable:

http://www.google.com/search?q=stored+procedures+vs

 Just issue 100 inserts from inside a
 stored procedure (or 100 updates) and do the same discretely with any kind of
 db interface. In case of the interface every statement has to be parsed by
 the db, whereas in a stored procedure the statement is already compiled of
 sorts (at least Oracle and PostgreSQL do that).

the debate over SP's is about a larger issue than is an SP faster
than 5 separate INSERT statements.  SP's are of course much better
for micro-benchmarks like that.  its their impact on application
development and architecture where the debate comes in (read some of
the googled articles).

I am certainly not anti-SP, ive done pure SP applications before (on
projects where the DBAs controlled the SPs)...I just dont want to
start hardwiring SQLAlchemy to expect that sort of application.   I
think 80/20 as applied to SELECT is that 80% of SELECTs are for read
operations and a COMMIT is inappropriate.

  if you
  really want COMMIT for every SELECT, i'd favor it being enabled via an
  option passed to create_engine().

 Not every select, every transaction that didn't roll back.
 I just think the default of rollback on every transaction is wrong - a
 rollback should occur when there is a problem, not when the transaction was
 fine. But that may just be me.

this is the use case:

c1 = pool.connect()
row = c1.cursor().execute(select * from sometable).fetchone()
pool.return_connection(c1)

c2 = pool.connect()  # (returns a connection that is not c1)
c2.cursor().execute(drop sometable)  # -- deadlock

if DBAPI supported a release_locks() method, we'd be using that.

 Probably because a lot of people can't figure out how to use stored procedures
 and triggers, since the lightweight/open-source programming is often done
 on a database that has very limited support for both :-)

keep in mind youre including the vast Hibernate community, including
its creators, etc.  im not sure if the im to dum to use stored
procedures argument can fully explain why the SP-architecture remains
a minority use case.  i think the overall inconvenience of it, the
clunky old languages you have to use on DBs like Oracle and SQL
Server, as well as the harsh resistance it puts up to so-called agile
development methods are better reasons.

 Personally I'm not a big fan of handling database integrity outside the
 database.
  continue SP arguments.

thats great, you can have your preferences..the google link above
should reveal that quite a few people have established their
preferences in this matter.  If you are truly writing an SP-only
application which prevents direct SQL access (that was the 

Re: Turbogears now based on Pylons!

[Michael Bayer]

On Jun 28, 2:09 am, Mike Orr [EMAIL PROTECTED] wrote:

 I would probably want this optional URI in a subclass rather than in
 SAContext itself.  The reason 'uri' is a required argument is to
 guarantee that the default engine is initialized at all times.  When
 we designed SAContext we (Mike  I) agreed that the bound metadata
 strategy was the most straightforward and adequate for most user apps:
 the 80/20 rule.  Hiding the engine (connectable) as much as possible
 in the metadata, while still making it accessible when you really need
 it.  Now we're adding one unbound metadata strategy after another.
 That's fine as log as it doesn't detract from SAContex's primary
 commitment to its primary userbase.  I'd say a mandatory URI and bound
 metadata is part of its primary commitment for an easy-to-use front
 end: a normal Python TypeError for You forgot the 'uri' argument is
 about as straightforward as one can get.  Or is that changing too?
 Are you having second thoughts about using bound metadata by default?


OK just FTR heres why SQLAlchemy is not a framework and why in order
to do things without the advent of a framework you work with all these
little highly granular components, and also why im still antsy
including SAContext in SA.  just trying to establish the simplest
heres your connection object spawns all this debate over how it
should look/work/act/etc.  I would encourage those here to look at
SAContext, see that its a dead-simple piece of code, and just work up
various preferred versions of it to taste.  My goal with it was more
to illustrate *how* you can build these things.


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Michael Bayer]

On Jun 27, 5:56 pm, Ian Bicking [EMAIL PROTECTED] wrote:
 * Way for a library to get the transaction manager.
 * Interface for that transaction manager, maybe copied from Zope.
 * Single convention for how to specify a database connection (ideally
 string-based).
 * Probably a way to get the configuration, so you can lazily get a
 connection based on the configured connection.

just as a point of reference, here is zalchemy's integration of Zope
datamanager:

http://svn.zope.org/z3c.zalchemy/trunk/src/z3c/zalchemy/datamanager.py?rev=77165view=auto

I would favor that the interface supports two-phase semantics like
zope's.  sqlalchemy's flush() model already fits in pretty well with
the two-phase model, and will eventually be building explicit two-
phase hooks into SQLAlchemy's engine and session, with real
implementations for postgres to start with.


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Michael Bayer]

On Jun 28, 2007, at 9:58 AM, Jonathan LaCour wrote:


 Michael Bayer wrote:

 just to clarify, however this works, it should be *really easy* for
 individual controller methods to be marked as transactional or not.
 I have written decorators like these already, in the style of:

 class MyController(...):
  def index_page(self):
  .

  @transactional
  def post_message(self):
  .

  def display_message(self):
  .

 so id want that kind of thing to be readily available, i suppose it
 would communicate with the WSGI middleware on a per-request basis.

 The only issue I can see with this is that having the transaction  
 begin
 in the controller method itself might be too late if you have other
 things going on in middleware or elsewhere and something goes wrong
 before you even get to the controller method -- say in object  
 dispatch,
 the new TG controller's lookup method, or even some middleware that
 stores sessions in a database for example.

the way this idea works is that if some wrapping component knows it  
needs a transaction, it opens one.  then the controller method  
participates in that transaction, regardless of whether or not its  
marked as @transactional.

 but another thing i like, is that of the response to the request  
 being
 delivered *after* the transaction commits. i have raised this issue
 on the pylons list before but i dont think i impressed anyone. if you
 render your response template, then your transaction fails, you have
 the potential for that response template to still be delivered with
 the success! message. anyway, might be nice for the middleware to
 address this if possible.

 I am totally with you here, as this is the other side of the coin that
 I brought up above.  Open transactions shouldn't be committed until  
 the
 latest possible point.

yes.  im talking specifcally about the way Pylons passes off from  
controller to view...right now you call render_response(), which  
typically renders the entire template in a buffer before the  
controller method has finished.  id favor changing the behavior of  
render_response() to creating a deferred render object which Pylons  
explcitly places outside the scope of the controller call (however,  
probably still within the scope of enclosing middleware).


 I spent some dark days working on J2EE applications, and while there
 isn't a whole lot that is worth emulating in J2EE, the transaction
 handling is something that I actually liked.

thats where im getting this from.  theres subtleties to it that you  
cant emulate casually.


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Jonathan LaCour]

Michael Bayer wrote:

 the way this idea works is that if some wrapping component knows
 it needs a transaction, it opens one. then the controller method
 participates in that transaction, regardless of whether or not its
 marked as @transactional.

Of course, that makes complete sense.  As long at the wrapping component
knows about the transaction manager (which, clearly, I think it should).

 I am totally with you here, as this is the other side of the coin
 that I brought up above.  Open transactions shouldn't be committed
 until the latest possible point.

 yes. im talking specifcally about the way Pylons passes off from
 controller to view...right now you call render_response(), which
 typically renders the entire template in a buffer before the
 controller method has finished. id favor changing the behavior of
 render_response() to creating a deferred render object which Pylons
 explcitly places outside the scope of the controller call (however,
 probably still within the scope of enclosing middleware).

Makes sense to me.

 I spent some dark days working on J2EE applications, and while there
 isn't a whole lot that is worth emulating in J2EE, the transaction
 handling is something that I actually liked.

 thats where im getting this from. theres subtleties to it that you
 cant emulate casually.

Yes, the further down the rabbit hole I go, the more clear this is
becoming.  That being said, I think I am sold on the concept that we
have hashed out here.

I need to take some time to look at Zope's transaction manager at some
point, and see how hard it would be to extract it from Zope, and get rid
of any dependencies, so it can easily be packaged up as its own, simple
egg.  Has anyone done this before?  Is anyone interested in taking this
up as a project?

--
Jonathan LaCour
http://cleverdevil.org


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Ian Bicking]

Michael Bayer wrote:
 On Jun 27, 6:42 pm, Ian Bicking [EMAIL PROTECTED] wrote:
 The way I see this working is something like (vaguely):

 def transaction_middleware(app):
  def wrapper(environ, start_response):
  manager = TransactionManager()
  environ['transaction.manager'] = manager
  try:
  app_iter = app(environ, start_response)
  except:
  manager.rollback()
  raise
  else:
  manager.commit()

 The manager is basically a container of *actual* transactions, and
 calling rollback or commit on it gets passed on to all the transactions
 in the manager.
 
 this is fine.  we're *really* starting to imitate J2EE in some ways.
 but its not a bad thing.

At least someone else thought it was a good idea too ;).  It's also all 
very similar to Zope.

 If you don't do anything that needs a transaction (e.g., read-only), you
 shouldn't put your transaction in the manager.
 
 just to clarify, however this works, it should be *really easy* for
 individual controller methods to be marked as transactional or not.
 I have written decorators like these already, in the style of:
 
 class MyController(...):
  def index_page(self):
  .
 
  @transactional
  def post_message(self):
  .
 
  def display_message(self):
  .
 
 so id want that kind of thing to be readily available, i suppose it
 would communicate with the WSGI middleware on a per-request basis.

I think that would be easy enough.  All that is setup early is the 
transaction container; nothing is automatically added to it.  So 
@transactional could just add a transaction to the container (presumably 
everything would otherwise be autocommit, or perhaps automatically 
rolled back?)

Presumably what you really want to do is 
add-this-transaction-if-it-isn't-there-already all over the place.  For 
instance, if you've already started a transaction in middleware, I don't 
think you'll need to start another transaction.  Unless you have 
subtransactions, which I suppose is possible.

 but another thing i like, is that of the response to the request being
 delivered *after* the transaction commits.  i have raised this issue
 on the pylons list before but i dont think i impressed anyone.  if you
 render your response template, then your transaction fails, you have
 the potential for that response template to still be delivered with
 the success! message.  anyway, might be nice for the middleware to
 address this if possible.

Hmm... I believe paste.exceptions should do this already, even if an 
exception is raised after start_response is called, as long as it is 
raised before the app_iter is returned.  If it's after the app_iter, 
then you've somehow escaped the stack, which doesn't much concern me ;)

So as middleware I think it should be fine.  If you want to catch a 
problem with the commit and handle it gracefully, I think you can just 
do your own commit in your code (and the middleware commit will just 
commit what little happens after that).

And of course, what you put in as a transaction is up to you.  You could 
put in some object which uses whatever flags you want to actually pass 
through a commit or else just ignore it.

 also i think this is all outside of the scope of SAContext.  SAContext
 should remain as a facade to SA-specific elements; it can accept
 strategy objects which control its internal workings, so if need be
 particular strategies can be delivered in order to function with the
 transaction manager, without changing the external view of SAContext.

I dunno... I think if we can get this in place, then a whole lot of 
things are going to become simpler.  But there also still has to be 
wrappers like SAContext from this fairly low-level stuff to a particular 
framework.


-- 
Ian Bicking | [EMAIL PROTECTED] | http://blog.ianbicking.org
 | Write code, do good | http://topp.openplans.org/careers

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Michael Bayer]

On Jun 28, 11:46 am, Ian Bicking [EMAIL PROTECTED] wrote:

 So as middleware I think it should be fine.  If you want to catch a
 problem with the commit and handle it gracefully, I think you can just
 do your own commit in your code (and the middleware commit will just
 commit what little happens after that).

 And of course, what you put in as a transaction is up to you.  You could
 put in some object which uses whatever flags you want to actually pass
 through a commit or else just ignore it.

usually, applications want a single big transaction for the whole
request.  if an app wants something that will commit independently of
the enclosing transaction, that can be an option to the decorator:

@transactional(independent=True)
def do_something(self):

the term independent is made up and maybe not so clear...but again,
this is straight out of EJB.  they let you annotate session beans in
an XML file using these terms: required, requiresNew, Supports,
Mandatory, NotSupported, Never.  I think we should consider each
of these cases and determine if we think we need to support that case,
and if so how. (also does Zope express this many variants?)


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Alberto Valverde]

On Jun 28, 2007, at 12:42 AM, Ian Bicking wrote:


 Jonathan LaCour wrote:
 Ian Bicking wrote:

 Personally I'd like to see something a bit different:

 * Way for a library to get the transaction manager.
 * Interface for that transaction manager, maybe copied from Zope.
 * Single convention for how to specify a database connection  
 (ideally
   string-based).
 * Probably a way to get the configuration, so you can lazily get
   a connection based on the configured connection.

 The frameworks would setup the transaction manager and  
 configuration,
 and database frameworks would basically consume this information.

 This would be pretty much ideal, and would satisfy my use cases very
 well.  It would also allow framework authors to build in support  
 for any
 ORM / database layer in an agnostic way, sort of like the Buffet API
 does for templating systems.

 Then, we could also create a simple WSGI middleware for Pylons that
 gives it TurboGears-like automatic transaction start/commit/rollback
 on a per request basis.  Only, we could make it configurable so that
 it didn't happen on read-only operations.  All of this would be ORM /
 database layer agnostic, which would be very nice indeed.

 The way I see this working is something like (vaguely):

 def transaction_middleware(app):
  def wrapper(environ, start_response):
  manager = TransactionManager()
  environ['transaction.manager'] = manager
  try:
  app_iter = app(environ, start_response)
  except:
  manager.rollback()
  raise
  else:
  manager.commit()

I see you've mentioned vaguely and that paste.transaction  
implements it properly... but I just had to say this. An almost exact  
implementation of that pseudocode caused me a very embarrassing bug  
in which transactions were committed after the authorization  
middleware trapped security exceptions and turned them into 403s.  
Luckily it was only an internal app and the bug was noticed in- 
house... :)

Bottom line is that the transaction middleware should rollback any  
transaction if the status code is = 400 to handle case like this  
(and that using existing, well tested code is usually better than  
rolling your own no matter how fun it is :)

Alberto

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Jonathan LaCour]

Alberto Valverde wrote:

 I'd like to add here that I've got use-cases for the transaction
 boundary and session initialization being closer to the server than
 what the __before__ and __after__ hooks provide.

 For example, the authentication function I feed to AuthKit which uses
 a DB connection to retrieve the user and it's associated roles and
 permissions.

 I can think of other cases where other layers of middleware (more
 precisely, Middleware FrameWork Components) might want to access the
 database. Having all these layers plus the controller method (or other
 WSGI which sit below PylonsWSGIApp) wrapped in the same transaction
 sounds like a good idea to me.

Fair enough to all of the above, however all the feedback from Mike and
Ian has made it clear to me that we need a few different things:

1. A standard transaction manager that can be plugged into any
   application, WSGI or not.  It should be similar in scope and API
   to Zope's transaction manager, but free of Zope dependancies.

2. A piece of middleware that uses the transaction manager to  
rollback
   any active transactions in the case of an exception, or commit  
them
   in the other case.

3. Either some middleware or just a few lines inside the  
BaseController
   of the TurboGears template that starts a transaction on every
   request and puts it into the transaction manager.

The first and second item are generally useful, and will allow you to
manage transactions manually if you so desire.  Any general-purpose
middleware out there that needs to use a transaction can simply use the
transaction manager, and all will be well.  TurboGears will have the
same behavior as it has had for a while, and since it happens in the
__call__ of the BaseController, you can tweak it to your heart's content
based upon the URI, HTTP method, or whatever you want.

And, in my opinion, items #1 and #2 should be in Pylons by default, and
the third item should be in the TurboGears template by default.

There is, of course, some extra stuff we could add for nested
transactions, and all of the cases he enumerated in his last email
(required, requiresNew, supports, mandatory, etc), but I think the three
things above will give a good set of building blocks for making those
things easy to deal with.

--
Jonathan LaCour
http://cleverdevil.org

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Ian Bicking]

Alberto Valverde wrote:
 
 On Jun 28, 2007, at 12:42 AM, Ian Bicking wrote:
 
 Jonathan LaCour wrote:
 Ian Bicking wrote:

 Personally I'd like to see something a bit different:

 * Way for a library to get the transaction manager.
 * Interface for that transaction manager, maybe copied from Zope.
 * Single convention for how to specify a database connection  
 (ideally
   string-based).
 * Probably a way to get the configuration, so you can lazily get
   a connection based on the configured connection.

 The frameworks would setup the transaction manager and  
 configuration,
 and database frameworks would basically consume this information.
 This would be pretty much ideal, and would satisfy my use cases very
 well.  It would also allow framework authors to build in support  
 for any
 ORM / database layer in an agnostic way, sort of like the Buffet API
 does for templating systems.

 Then, we could also create a simple WSGI middleware for Pylons that
 gives it TurboGears-like automatic transaction start/commit/rollback
 on a per request basis.  Only, we could make it configurable so that
 it didn't happen on read-only operations.  All of this would be ORM /
 database layer agnostic, which would be very nice indeed.
 The way I see this working is something like (vaguely):

 def transaction_middleware(app):
  def wrapper(environ, start_response):
  manager = TransactionManager()
  environ['transaction.manager'] = manager
  try:
  app_iter = app(environ, start_response)
  except:
  manager.rollback()
  raise
  else:
  manager.commit()
 
 I see you've mentioned vaguely and that paste.transaction  
 implements it properly... but I just had to say this. An almost exact  
 implementation of that pseudocode caused me a very embarrassing bug  
 in which transactions were committed after the authorization  
 middleware trapped security exceptions and turned them into 403s.  
 Luckily it was only an internal app and the bug was noticed in- 
 house... :)

paste.transaction itself is something I should have really removed, as 
it's so incomplete.  But yes, it's the basic pattern, but without a 
clear concept of a TransactionManager.

 Bottom line is that the transaction middleware should rollback any  
 transaction if the status code is = 400 to handle case like this  
 (and that using existing, well tested code is usually better than  
 rolling your own no matter how fun it is :)

Er... I'm not sure about this.  I'm not sure =400 is really a failure. 
  Maybe it is; I just don't feel comfortable making that claim generally.

If you have a security exception that is caught, you could also look for 
the manager and roll it back exactly then, instead of expecting the 
middleware to do it.  The middleware is just the furthest-out boundary 
for transactions, it's not the only place where you can control the 
transactions.

-- 
Ian Bicking | [EMAIL PROTECTED] | http://blog.ianbicking.org
 | Write code, do good | http://topp.openplans.org/careers

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Alberto Valverde]

On Jun 28, 2007, at 7:33 PM, Jonathan LaCour wrote:


 Alberto Valverde wrote:

 I'd like to add here that I've got use-cases for the transaction
 boundary and session initialization being closer to the server than
 what the __before__ and __after__ hooks provide.

 For example, the authentication function I feed to AuthKit which uses
 a DB connection to retrieve the user and it's associated roles and
 permissions.

 I can think of other cases where other layers of middleware (more
 precisely, Middleware FrameWork Components) might want to access the
 database. Having all these layers plus the controller method (or  
 other
 WSGI which sit below PylonsWSGIApp) wrapped in the same transaction
 sounds like a good idea to me.

 Fair enough to all of the above, however all the feedback from Mike  
 and
 Ian has made it clear to me that we need a few different things:

 1. A standard transaction manager that can be plugged into any
application, WSGI or not.  It should be similar in scope and  
 API
to Zope's transaction manager, but free of Zope dependancies.

 2. A piece of middleware that uses the transaction manager to
 rollback
any active transactions in the case of an exception, or commit
 them
in the other case.

(I'm assuming this middleware above the PylonsWSGIApp...)


 3. Either some middleware or just a few lines inside the
 BaseController
of the TurboGears template that starts a transaction on every
request and puts it into the transaction manager.

Hmmm, wouldn't having two transaction boundaries at 2 and 3 be  
redundant? Or maybe both boundaries will use the same manager in a  
similar way paste.registry has only one Registry for the whole stack?  
(albeit different context boundaries every time a RegistryManager is  
stacked)

One problem I see is that this might cause trouble in cases like this:

Request
|
   V
db transaction middleware  - creates session A
   |
   V
authentication middleware - authenticates user and keeps ref to User  
instance (attached to session A)
   |
   V
other middleware and PylonsApp
   |
   V
TG controller  - creates session B inside __call__
   |
   V
authorization library tries to check permissions for mapped Foo  
instance (attached to session B) which has a owner attribute:

foo.owner (sess B) == logged_in_user (sess A)

I haven't actually tried it so I'm not sure how to objects attached  
to different sessions compare as or interact, so this might be a non- 
issue (maybe Mike can shed some light on this?)

Anyway, this might be a non-issue on the whole if the transaction  
middleware does not handle session creation at all.

OTOH, I do see the difficulty of having just one boundary in a single  
TG/Pylons app in a layer of middleware above PylonsApp since I don't  
see a clear way of making parameters passed via a decorator reach the  
middleware *before* the method is actually called (which will be  
needed to signal that a BEGIN should not be issued)

Alberto

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Alberto Valverde]

On Jun 28, 2007, at 7:45 PM, Ian Bicking wrote:


 Alberto Valverde wrote:

 On Jun 28, 2007, at 12:42 AM, Ian Bicking wrote:

 Jonathan LaCour wrote:
 Ian Bicking wrote:

 Personally I'd like to see something a bit different:

 * Way for a library to get the transaction manager.
 * Interface for that transaction manager, maybe copied from Zope.
 * Single convention for how to specify a database connection
 (ideally
   string-based).
 * Probably a way to get the configuration, so you can lazily get
   a connection based on the configured connection.

 The frameworks would setup the transaction manager and
 configuration,
 and database frameworks would basically consume this information.
 This would be pretty much ideal, and would satisfy my use cases  
 very
 well.  It would also allow framework authors to build in support
 for any
 ORM / database layer in an agnostic way, sort of like the Buffet  
 API
 does for templating systems.

 Then, we could also create a simple WSGI middleware for Pylons that
 gives it TurboGears-like automatic transaction start/commit/ 
 rollback
 on a per request basis.  Only, we could make it configurable so  
 that
 it didn't happen on read-only operations.  All of this would be  
 ORM /
 database layer agnostic, which would be very nice indeed.
 The way I see this working is something like (vaguely):

 def transaction_middleware(app):
  def wrapper(environ, start_response):
  manager = TransactionManager()
  environ['transaction.manager'] = manager
  try:
  app_iter = app(environ, start_response)
  except:
  manager.rollback()
  raise
  else:
  manager.commit()

 I see you've mentioned vaguely and that paste.transaction
 implements it properly... but I just had to say this. An almost exact
 implementation of that pseudocode caused me a very embarrassing bug
 in which transactions were committed after the authorization
 middleware trapped security exceptions and turned them into 403s.
 Luckily it was only an internal app and the bug was noticed in-
 house... :)

 paste.transaction itself is something I should have really removed, as
 it's so incomplete.  But yes, it's the basic pattern, but without a
 clear concept of a TransactionManager.

 Bottom line is that the transaction middleware should rollback any
 transaction if the status code is = 400 to handle case like this
 (and that using existing, well tested code is usually better than
 rolling your own no matter how fun it is :)

 Er... I'm not sure about this.  I'm not sure =400 is really a  
 failure.
   Maybe it is; I just don't feel comfortable making that claim  
 generally.

I can understand this... at least we're sure we shouldn't implicitly  
*commit* the transaction

 If you have a security exception that is caught, you could also  
 look for
 the manager and roll it back exactly then, instead of expecting the
 middleware to do it.  The middleware is just the furthest-out boundary
 for transactions, it's not the only place where you can control the
 transactions.

However, this would couple the security-exception-trapping middleware  
with the transaction manager. Right now, the mentioned middleware's  
responsability is very simple: Trap security exceptions and turn them  
into 403s or 401s (depending if there's a user logged in). it doesn't  
even handle authentication (which is handled above to provide a  
REMOTE_USER and/or wsgi_org.user_info and trapping 401s). I can also  
think of other layers which could change status codes, expecting all  
this layers to be aware that a transaction manager is in the stack  
and explicitly rollback any transaction doesn't favor loose-coupling  
from my POV.

I believe that in this scenario the most straightforward way to  
signal that an error (or non OK) condition has ocurred is to through  
the HTTP status code hence the db transaction middleware should take  
it into account. The question of what status codes should be treated  
as errors is still open though...

Alberto

 -- 
 Ian Bicking | [EMAIL PROTECTED] | http://blog.ianbicking.org
  | Write code, do good | http://topp.openplans.org/careers

 


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Uwe C. Schroeder]

On Thursday 28 June 2007, Michael Bayer wrote:
 On Jun 28, 1:40 am, Uwe C. Schroeder [EMAIL PROTECTED] wrote:
  On Wednesday 27 June 2007, Michael Bayer wrote:
   this issue can be worked around by using explicit transactions.
 
  actually no, it can't. Maybe I don't get it right, but the only way for
  me to get a commit was actually to modify Connection._autocommit in
  sqlalchemy.engine.base.

 either TG is getting in the way, or youre not getting it right.  if
 anyone ever needs to modify the internals of SA to get something to
 work, i would *highly* prefer if they could email the ML or post a
 trac ticket with their issue so that it may be handled properly.

I agree. I didn't put this on the ML or trac because I just don't know why it 
behaves like that and frankly I just gave up on it after a couple of hours 
and took the shortcut. 
Obviously TG wraps every request into a transaction and commits/rollback's  
that when the controller method returns the response.
There is a valid transaction object in the tg.sa_transaction variable which 
should allow to end the outer transaction and start a new one on your own.
Although I can do that, the result was always the same - SA doesn't see the 
transaction and runs into the base._autocommit method (which I think it 
shouldn't if the transaction was handled properly before that). Tracking what 
the DB gets, it never sees a rollback/commit when I tell the original 
transaction object in tg.sa_transaction to rollback. The command is just 
ignored.

When it comes to submitting a patch: I do have code changes for SA to handle 
arrays and user defined types for postgresql in the autodetect code. So I 
might roll that up and send it over when I have a minute.


  Obviously SA thinks there is no transaction in TG,
  so it just wraps one around it.

 if TG actually has a transaction going on, theyd certainly have to
 configure SA to be aware of it (most likely via SessionTransaction).
 if not, then yes things arent going to work at all (though still, an
 explicit SA transaction should work all by itself).

The tg.sa_transaction variable contains a SessionTransaction object at that 
time. So they obviously do something to get that.

  I agree, and that is certainly DB dependent. Personally I can't imagine
  that an automatically issued rollback for every select transaction is in
  any way more overhead than issuing a commit. Not wrapping a select in a
  transaction will definitely be the least overhead.

 we dont issue a rollback for every select transaction.  we issue a
 rollback when a connection is returned to the pool.  you can check out
 a connection explicitly and perform any number of selects on it
 without any rollbacks or commits.

 because the rollback is at the connection-pool checkin level, it
 should be more apparent how inappropriate it would be to issue a
 *commit* every time a connection is returned to the pool, an operation
 that knows nothing about what just happened with that connection.  the
 rollback is to release database locks.

 im thinking that it might be time to allow an option in SA that just
 turns the DBAPI's autocommit flag on.  that way you can just blame
 the DBAPI for whatever issues arise.

:-) Yes. I bet the result would be the same in my case. 


 its not always possible to not wrap a select in a transaction.
 oracle for example *always* has a transaction going on, so everything
 is in a transaction in all cases.

   that a stored-procedure-oriented application is far more efficient
   is *extremely* debatable and database-dependent as well.
 
  I doubt it's *extremely* debatable.

 its extremely debatable:

 http://www.google.com/search?q=stored+procedures+vs

  Just issue 100 inserts from inside a
  stored procedure (or 100 updates) and do the same discretely with any
  kind of db interface. In case of the interface every statement has to be
  parsed by the db, whereas in a stored procedure the statement is already
  compiled of sorts (at least Oracle and PostgreSQL do that).

 the debate over SP's is about a larger issue than is an SP faster
 than 5 separate INSERT statements.  SP's are of course much better
 for micro-benchmarks like that.  its their impact on application
 development and architecture where the debate comes in (read some of
 the googled articles).

 I am certainly not anti-SP, ive done pure SP applications before (on
 projects where the DBAs controlled the SPs)...I just dont want to
 start hardwiring SQLAlchemy to expect that sort of application.   I
 think 80/20 as applied to SELECT is that 80% of SELECTs are for read
 operations and a COMMIT is inappropriate.

I agree with your goals and I also agree that a lot of people don't use stored 
procedures and similar db-dependent things. All I'd need is a way to tell SA 
that this one select is something worth committing.

   if you
   really want COMMIT for every SELECT, i'd favor it being enabled via an
   option passed to create_engine().
 
  Not every 

Re: Turbogears now based on Pylons!

[Mike Orr]

On 6/28/07, Jonathan LaCour [EMAIL PROTECTED] wrote:
 3. Either some middleware or just a few lines inside the
 BaseController
of the TurboGears template that starts a transaction on every
request and puts it into the transaction manager.

 And, in my opinion, items #1 and #2 should be in Pylons by default, and
 the third item should be in the TurboGears template by default.

... and available to non-TG Pylons applications.  The @transactional
decorator sounds like a good idea, possibly with a False flag if the
default is transactional.

This is a good point to ask, how will the TG template differ from the
standard Pylons template?  Will there be a base controller, or at what
point will they diverge?  Where will CherryPy-style dispatching be
implemented?  I imagine you will override PylonsApp and/or
PylonsBaseWSGIApp in pylons.wsgiapp, no?

-- 
Mike Orr [EMAIL PROTECTED]

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Alberto Valverde]

On Jun 28, 2007, at 8:52 PM, Mike Orr wrote:


 On 6/28/07, Jonathan LaCour [EMAIL PROTECTED] wrote:
 3. Either some middleware or just a few lines inside the
 BaseController
of the TurboGears template that starts a transaction on every
request and puts it into the transaction manager.

 And, in my opinion, items #1 and #2 should be in Pylons by  
 default, and
 the third item should be in the TurboGears template by default.

 ... and available to non-TG Pylons applications.  The @transactional
 decorator sounds like a good idea, possibly with a False flag if the
 default is transactional.

 This is a good point to ask, how will the TG template differ from the
 standard Pylons template?  Will there be a base controller, or at what
 point will they diverge?  Where will CherryPy-style dispatching be
 implemented?  I imagine you will override PylonsApp and/or
 PylonsBaseWSGIApp in pylons.wsgiapp, no?

The dispatching is currently implemented in a route method in a  
PylonsWSGIController subclass. There's a route defined like:

map.connect('*url', controller='root', action='route')

That handles it url to further dispatch it, cherrypy style. Here's  
all the meat:

http://trac.turbogears.org/browser/trunk/tg/controllers.py#L58

Alberto

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Noah Gift]

On 6/28/07, Alberto Valverde [EMAIL PROTECTED] wrote:



 On Jun 28, 2007, at 8:52 PM, Mike Orr wrote:

 
  On 6/28/07, Jonathan LaCour [EMAIL PROTECTED] wrote:
  3. Either some middleware or just a few lines inside the
  BaseController
 of the TurboGears template that starts a transaction on every
 request and puts it into the transaction manager.
 
  And, in my opinion, items #1 and #2 should be in Pylons by
  default, and
  the third item should be in the TurboGears template by default.
 
  ... and available to non-TG Pylons applications.  The @transactional
  decorator sounds like a good idea, possibly with a False flag if the
  default is transactional.
 
  This is a good point to ask, how will the TG template differ from the
  standard Pylons template?  Will there be a base controller, or at what
  point will they diverge?  Where will CherryPy-style dispatching be
  implemented?  I imagine you will override PylonsApp and/or
  PylonsBaseWSGIApp in pylons.wsgiapp, no?

 The dispatching is currently implemented in a route method in a
 PylonsWSGIController subclass. There's a route defined like:

 map.connect('*url', controller='root', action='route')

 That handles it url to further dispatch it, cherrypy style. Here's
 all the meat:

 http://trac.turbogears.org/browser/trunk/tg/controllers.py#L58


The controller is also slightly different as it is based a bit on Rick
Copeland's q-lookup Traversable system to a degree:

http://code.feestjeproject.com/feestje/browser/trunk/pyatl/lib/generic.py





Alberto

 



-- 
http://www.blog.noahgift.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Jonathan LaCour]

Mike Orr wrote:

 ... and available to non-TG Pylons applications.  The @transactional
 decorator sounds like a good idea, possibly with a False flag if the
 default is transactional.

Agreed.

 This is a good point to ask, how will the TG template differ from
 the standard Pylons template?  Will there be a base controller,
 or at what point will they diverge?  Where will CherryPy-style
 dispatching be implemented?  I imagine you will override PylonsApp
 and/or PylonsBaseWSGIApp in pylons.wsgiapp, no?

The TurboGears template, as of right now, is basically the same as
a Pylons template, only it includes a default route to a special
WSGIController subclass called TurboGearsController that implements
all sorts of nice things, like object-dispatch, content-negotation,
validation, and TG style @expose decoration.  This controller is called
the Root controller and is preconfigured in the routes configuration:

 map.connect('*url', controller='root', action='route')

You can find the controller here:

 http://trac.turbogears.org/browser/trunk/tg/controllers.py

The route method of the controller probably needs to be split up
into some smaller, overridable chunks.  The nice thing about the
approach that we have taken thus far is that you can mix TG-style
object-dispatching controllers and regular Pylons controllers if you
like.

--
Jonathan LaCour
http://cleverdevil.org


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Mike Orr]

On 6/28/07, Jonathan LaCour [EMAIL PROTECTED] wrote:
 The TurboGears template, as of right now, is basically the same as
 a Pylons template, only it includes a default route to a special
 WSGIController subclass called TurboGearsController that implements
 all sorts of nice things, like object-dispatch, content-negotation,
 validation, and TG style @expose decoration.  This controller is called
 the Root controller and is preconfigured in the routes configuration:

  map.connect('*url', controller='root', action='route')

 You can find the controller here:

  http://trac.turbogears.org/browser/trunk/tg/controllers.py

 The route method of the controller probably needs to be split up
 into some smaller, overridable chunks.  The nice thing about the
 approach that we have taken thus far is that you can mix TG-style
 object-dispatching controllers and regular Pylons controllers if you
 like.

So if i want to use this for all URLs under a certain prefix, I would
direct the route to my subclass of TurboGearsController, using action
route (always?).
Line 66 (object_dispatch) returns my TG action method in all its
decorated glory, line 85 calls it and receives a dict, and line 88
calls a special render_response that invokes the template.
render_response is copying things out of the 'c' variable, so
templates would use '${foo}' rather than '${c.foo}' for data values.
Is this right?

I have long thought about making a TG-style dispatcher for Pylons.
It's great that somebody with more expertise has actually done it.

Noah wrote:
 The controller is also slightly different as it is based a bit on Rick 
 Copeland's  q-lookup Traversable system to a degree:

 http://code.feestjeproject.com/feestje/browser/trunk/pyatl/lib/generic.py

This reminds me of Quixote, especially the '._q_lookup' method.  Did
they borrow it from him or did he borrow it from them?

-- 
Mike Orr [EMAIL PROTECTED]

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Jonathan LaCour]

Mike Orr wrote:

 So if i want to use this for all URLs under a certain prefix, I would
 direct the route to my subclass of TurboGearsController, using action
 route (always?).

Well, thats the idea.  It needs lots of love and testing, though.

 Line 66 (object_dispatch) returns my TG action method in all its
 decorated glory, line 85 calls it and receives a dict, and line
 88 calls a special render_response that invokes the template.
 render_response is copying things out of the 'c' variable, so
 templates would use '${foo}' rather than '${c.foo}' for data values.
 Is this right?

Yes.  Most of us TurboGears folks are not big fans of single
character globals that represent stacked object proxies :) The
TurboGearsController will work like TurboGears 1.0 controllers in that
methods return dicts and those dicts are passed into the namespace of
the template.

 I have long thought about making a TG-style dispatcher for Pylons.
 It's great that somebody with more expertise has actually done it.

Well, most of the credit goes to Rick Copeland, since he hacked most of
the code for the new controller.

 This reminds me of Quixote, especially the '._q_lookup' method.  Did
 they borrow it from him or did he borrow it from them?

You'll have to ask Rick where he came up with the idea, all I know is
that I find it to be extremely useful!

--
Jonathan LaCour
http://cleverdevil.org


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Noah Gift]

On 6/28/07, Jonathan LaCour [EMAIL PROTECTED] wrote:


 Mike Orr wrote:

  So if i want to use this for all URLs under a certain prefix, I would
  direct the route to my subclass of TurboGearsController, using action
  route (always?).

 Well, thats the idea.  It needs lots of love and testing, though.

  Line 66 (object_dispatch) returns my TG action method in all its
  decorated glory, line 85 calls it and receives a dict, and line
  88 calls a special render_response that invokes the template.
  render_response is copying things out of the 'c' variable, so
  templates would use '${foo}' rather than '${c.foo}' for data values.
  Is this right?

 Yes.  Most of us TurboGears folks are not big fans of single
 character globals that represent stacked object proxies :) The
 TurboGearsController will work like TurboGears 1.0 controllers in that
 methods return dicts and those dicts are passed into the namespace of
 the template.

  I have long thought about making a TG-style dispatcher for Pylons.
  It's great that somebody with more expertise has actually done it.

 Well, most of the credit goes to Rick Copeland, since he hacked most of
 the code for the new controller.

  This reminds me of Quixote, especially the '._q_lookup' method.  Did
  they borrow it from him or did he borrow it from them?

 You'll have to ask Rick where he came up with the idea, all I know is
 that I find it to be extremely useful!


Yes, Rick is a big fan of Quixote... :)  I am sure he will get in on this
thread at some point...


--
 Jonathan LaCour
 http://cleverdevil.org


 



-- 
http://www.blog.noahgift.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Michael Bayer]

On Jun 28, 2:32 pm, Uwe C. Schroeder [EMAIL PROTECTED] wrote:

 If I could figure out how, I probably would. The code from the docs doesn't
 work, because TG does something around that. I guess one of the major
 drawbacks of TG is it's documentation (or lack thereof) when it comes to
 using SA together with it. I guess that will change once TG uses SA by
 default.

my random guess is that they create a SessionTransaction, then they
forget about it.  if you then go and do some straight SQL like
select(foo).execute(), that goes against the engine, but that engine
needs to be set up explicitly within the SessionTransaction:

http://www.sqlalchemy.org/docs/unitofwork.html#unitofwork_transaction_sql

alternatively, you can pull out the connection straight from the
session and use that, via session.connection() - this also registers
the engine with the current transaction.

when using just the ORM for Query/flush(), the Session does this for
you when it first uses an engine while the transaction is present...so
it would seem like it just works, when really it only just works
for ORM logic.


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Michael Bayer]

On Jun 28, 2:01 pm, Alberto Valverde [EMAIL PROTECTED] wrote:
 One problem I see is that this might cause trouble in cases like this:

 Request
 |
V
 db transaction middleware  - creates session A
|
V
 authentication middleware - authenticates user and keeps ref to User
 instance (attached to session A)
|
V
 other middleware and PylonsApp
|
V
 TG controller  - creates session B inside __call__
|
V
 authorization library tries to check permissions for mapped Foo
 instance (attached to session B) which has a owner attribute:

 foo.owner (sess B) == logged_in_user (sess A)

you generally should not share objects between sessions, things will
break.  for example, the right way to use the user from session a into
session b would be :

foo.owner = sessionb.merge(user_from_sessiona)


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Noah Gift]

  drawbacks of TG is it's documentation (or lack thereof) when it comes to
  using SA together with it.


I actually asked Michael Bayer a couple of dumb questions as result of not
knowing what the controller did in a SQLAlchemy turbogears project.  I am
glad I wasn't the only one who was confused.  The automatic transactions in
the controller were unexpected as I went through the SQLAlchemy tutorial
thinking I had to control my transactions...which I liked!

That statement is very important though.  I think that Django's
documentation is something to aspire to and that if Turbogears doesn't reach
that level then it will turn into Twisted.  Every time I see someone give a
demo on Twisted I thinkwow...that is crazy, but then I start getting
into the documentation and I think...huh?

I am pretty sure Turbogears riding on Pylons will be on a vengeance to
document things correctly and not just the API, but usage scenarios for
everything.  Something can be the most innovative solution in the world, but
if no one knows about it, it is a complete waste


I guess that will change once TG uses SA by
  default.


I





http://www.blog.noahgift.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[[EMAIL PROTECTED]]

For me TG on Pylons would be a set of  similar packages-components
that make thinks easier and quicker
- wrappers for various backends (tg-user for authkit and others)
- set of helpers (toscawidgets, webhelpers)


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Uwe C. Schroeder]

Funny you mention Twisted. May app is actually Twisted based - and yes, I 
agree. I had some good (and some bad) experiences with Glyph emailing back 
and forth to get things resolved :-)


On Thursday 28 June 2007, Noah Gift wrote:
   drawbacks of TG is it's documentation (or lack thereof) when it comes to
 
   using SA together with it.

 I actually asked Michael Bayer a couple of dumb questions as result of not
 knowing what the controller did in a SQLAlchemy turbogears project.  I am
 glad I wasn't the only one who was confused.  The automatic transactions in
 the controller were unexpected as I went through the SQLAlchemy tutorial
 thinking I had to control my transactions...which I liked!

 That statement is very important though.  I think that Django's
 documentation is something to aspire to and that if Turbogears doesn't
 reach that level then it will turn into Twisted.  Every time I see someone
 give a demo on Twisted I thinkwow...that is crazy, but then I start
 getting into the documentation and I think...huh?

 I am pretty sure Turbogears riding on Pylons will be on a vengeance to
 document things correctly and not just the API, but usage scenarios for
 everything.  Something can be the most innovative solution in the world,
 but if no one knows about it, it is a complete waste


 I guess that will change once TG uses SA by

   default.

 I





 http://www.blog.noahgift.com

 


-- 
UC

--
Open Source Solutions 4U, LLC   1618 Kelly St
Phone:  +1 707 568 3056 Santa Rosa, CA 95401
Cell:   +1 650 302 2405 United States
Fax:+1 707 568 6416

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---

Re: Turbogears now based on Pylons!

[Mike Orr]

On 6/28/07, Noah Gift [EMAIL PROTECTED] wrote:
 I am pretty sure Turbogears riding on Pylons will be on a vengeance to
 document things correctly and not just the API, but usage scenarios for
 everything.

Joining forces will make a big difference.  TG and Pylons combined
probably account for half of the non-Zope Python web applications
written in the past two years, give or take ten percent, and roughly
similar for the amount of developer activity.  Having twice as many
people working on the same codebase and docs rather than two separate
ones should accellerate developement.  There may still be two wiki
demos but they'll be able to borrow half their code from each other
rather than just the look  feel.  Our skills are also complementary.
TG developers do a better job of documentation and newbie handholding,
while Pylons has attracted developers who are obsessed with modularity
and reuse to a greater degree.  (TG developers may doubt this, but
many people came to Pylons because they found TG more monolithic than
they liked.)  Nine months till the next PyCon: it'll be exciting to
see where we're at then, and what sprints we'll be having.

-- 
Mike Orr [EMAIL PROTECTED]

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---