Re: [Zope-dev] zope.sqlalchemy locks up transaction

2008-09-19 Thread Hermann Himmelbauer
Am Donnerstag 18 September 2008 07:08:56 schrieb Dieter Maurer:
 Brian Sutherland wrote at 2008-9-17 12:33 +0200:
 On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote:
  Brian Sutherland wrote:
   Hi,
  
   I've recently seen a situation where zope.sqlalchemy locked up the
   transaction machinery. I'm not sure exactly what happened, but have
   attached a failing test for at least one bug which may have caused it.
   Hopefully it's self explanatory;)
  
   If someone could help me solve this, that would be great!
 
  Could you try this with latest trunk. I checked in a fix the other day
  that may help.
 
 I just checked in a fix, please feel free to comment on/revert it if
 it's not up to standard:)

 It looks not yet right to clear the state in tpc_vote when
 a two phase commit is used (which is now supported by SQLAlchemy).

 In addition, there may be a problem in case session.close() raises
 an exception. Then, _finish would not be called.

This may not be connected, but I just wanted to point out a problem with 
configurations with autoflush=False. If I change some database object 
attributes in the session but do not call session.flush(), the data is not 
stored into the database, as no commit() is called, which would then flush 
out/commit the data.

So perhaps this is something to think about, or, at least something to be 
denoted in the documentation.

I circumvented the problem by setting autoflush=True and modifying my code so 
that it works (although it's quite complicated).

Best Regards,
Hermann

-- 
[EMAIL PROTECTED]
GPG key ID: 299893C7 (on keyservers)
FP: 0124 2584 8809 EF2A DBF9  4902 64B4 D16B 2998 93C7
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] zope.sqlalchemy locks up transaction

2008-09-19 Thread Laurence Rowe
2008/9/19 Hermann Himmelbauer [EMAIL PROTECTED]:
 Am Donnerstag 18 September 2008 07:08:56 schrieb Dieter Maurer:
 Brian Sutherland wrote at 2008-9-17 12:33 +0200:
 On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote:
  Brian Sutherland wrote:
   Hi,
  
   I've recently seen a situation where zope.sqlalchemy locked up the
   transaction machinery. I'm not sure exactly what happened, but have
   attached a failing test for at least one bug which may have caused it.
   Hopefully it's self explanatory;)
  
   If someone could help me solve this, that would be great!
 
  Could you try this with latest trunk. I checked in a fix the other day
  that may help.
 
 I just checked in a fix, please feel free to comment on/revert it if
 it's not up to standard:)

 It looks not yet right to clear the state in tpc_vote when
 a two phase commit is used (which is now supported by SQLAlchemy).

 In addition, there may be a problem in case session.close() raises
 an exception. Then, _finish would not be called.

 This may not be connected, but I just wanted to point out a problem with
 configurations with autoflush=False. If I change some database object
 attributes in the session but do not call session.flush(), the data is not
 stored into the database, as no commit() is called, which would then flush
 out/commit the data.

 So perhaps this is something to think about, or, at least something to be
 denoted in the documentation.

 I circumvented the problem by setting autoflush=True and modifying my code so
 that it works (although it's quite complicated).

I don't think this is related to the above problem, but it may be a
bug in SQLAlchemy 0.4 whose behaviour I have replicated.

In 0.4 orm.session during a commit flush is called:

245 if self.autoflush:
246 self.session.flush()

whereas in 0.5 flush is called unconditionally so long as there is not
a flush in progress.

358 if not self.session._flushing:
359 self.session.flush()

The documentation for 0.4 appears to be misleading:

commit() serves two purposes; it issues a flush() unconditionally to
persist any remaining pending changes.


I think I should change zope.sqlalchemy to flush during it's tpc_begin
phase. This should only be done if a flush is not already in progress,
as otherwise sqlalchemy would raise an error. self.session._flushing
is only in 0.5 though, so maybe it is time to up the dependency. Or
maybe it is impossible for it to be in the flushing state at that
point anyway, as all user code will have completed.

Laurence
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] zope.sqlalchemy locks up transaction

2008-09-19 Thread Hermann Himmelbauer
Am Freitag 19 September 2008 11:59:55 schrieb Laurence Rowe:
 2008/9/19 Hermann Himmelbauer [EMAIL PROTECTED]:
  Am Donnerstag 18 September 2008 07:08:56 schrieb Dieter Maurer:
  Brian Sutherland wrote at 2008-9-17 12:33 +0200:
  On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote:
   Brian Sutherland wrote:
Hi,
   
I've recently seen a situation where zope.sqlalchemy locked up the
transaction machinery. I'm not sure exactly what happened, but have
attached a failing test for at least one bug which may have caused
it. Hopefully it's self explanatory;)
   
If someone could help me solve this, that would be great!
  
   Could you try this with latest trunk. I checked in a fix the other
   day that may help.
  
  I just checked in a fix, please feel free to comment on/revert it if
  it's not up to standard:)
 
  It looks not yet right to clear the state in tpc_vote when
  a two phase commit is used (which is now supported by SQLAlchemy).
 
  In addition, there may be a problem in case session.close() raises
  an exception. Then, _finish would not be called.
 
  This may not be connected, but I just wanted to point out a problem with
  configurations with autoflush=False. If I change some database object
  attributes in the session but do not call session.flush(), the data is
  not stored into the database, as no commit() is called, which would then
  flush out/commit the data.
 
  So perhaps this is something to think about, or, at least something to be
  denoted in the documentation.
 
  I circumvented the problem by setting autoflush=True and modifying my
  code so that it works (although it's quite complicated).

 I don't think this is related to the above problem, but it may be a
 bug in SQLAlchemy 0.4 whose behaviour I have replicated.

 In 0.4 orm.session during a commit flush is called:

 245   if self.autoflush:
 246   self.session.flush()

 whereas in 0.5 flush is called unconditionally so long as there is not
 a flush in progress.

 358   if not self.session._flushing:
 359   self.session.flush()

 The documentation for 0.4 appears to be misleading:

 commit() serves two purposes; it issues a flush() unconditionally to
 persist any remaining pending changes.

Ah, I see, that's really misleading.

 I think I should change zope.sqlalchemy to flush during it's tpc_begin
 phase. This should only be done if a flush is not already in progress,
 as otherwise sqlalchemy would raise an error. 

That's maybe a good idea.

 self.session._flushing 
 is only in 0.5 though, so maybe it is time to up the dependency. 

What do you mean with up the dependency? If you mean dropping support for SA 
0.4, please, *please* don't do that (0.5's autoflushing breaks things here 
and Database support for MaxDB/SAPDB is somehow flawed.). Due to many 
incompatibilites, I cannot switch to SA 0.5, so I'd really appreciate support 
for SA 0.4!

 Or 
 maybe it is impossible for it to be in the flushing state at that
 point anyway, as all user code will have completed.

If I understand it right, the problem does not occur in SA 0.5, right? If 
that's the case, maybe it is enough to outline the behaviour of SA 0.4 in the 
documentation and leave it as it is.

Best Regards,
Hermann

-- 
[EMAIL PROTECTED]
GPG key ID: 299893C7 (on keyservers)
FP: 0124 2584 8809 EF2A DBF9  4902 64B4 D16B 2998 93C7
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] zope.sqlalchemy locks up transaction

2008-09-19 Thread Laurence Rowe
2008/9/19 Laurence Rowe [EMAIL PROTECTED]:
 2008/9/19 Hermann Himmelbauer [EMAIL PROTECTED]:
 Am Donnerstag 18 September 2008 07:08:56 schrieb Dieter Maurer:
 Brian Sutherland wrote at 2008-9-17 12:33 +0200:
 On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote:
  Brian Sutherland wrote:
   Hi,
  
   I've recently seen a situation where zope.sqlalchemy locked up the
   transaction machinery. I'm not sure exactly what happened, but have
   attached a failing test for at least one bug which may have caused it.
   Hopefully it's self explanatory;)
  
   If someone could help me solve this, that would be great!
 
  Could you try this with latest trunk. I checked in a fix the other day
  that may help.
 
 I just checked in a fix, please feel free to comment on/revert it if
 it's not up to standard:)

 It looks not yet right to clear the state in tpc_vote when
 a two phase commit is used (which is now supported by SQLAlchemy).

 In addition, there may be a problem in case session.close() raises
 an exception. Then, _finish would not be called.

 This may not be connected, but I just wanted to point out a problem with
 configurations with autoflush=False. If I change some database object
 attributes in the session but do not call session.flush(), the data is not
 stored into the database, as no commit() is called, which would then flush
 out/commit the data.

 So perhaps this is something to think about, or, at least something to be
 denoted in the documentation.

 I circumvented the problem by setting autoflush=True and modifying my code so
 that it works (although it's quite complicated).

 I don't think this is related to the above problem, but it may be a
 bug in SQLAlchemy 0.4 whose behaviour I have replicated.

 In 0.4 orm.session during a commit flush is called:

 245 if self.autoflush:
 246 self.session.flush()

 whereas in 0.5 flush is called unconditionally so long as there is not
 a flush in progress.

 358 if not self.session._flushing:
 359 self.session.flush()

 The documentation for 0.4 appears to be misleading:

 commit() serves two purposes; it issues a flush() unconditionally to
 persist any remaining pending changes.


 I think I should change zope.sqlalchemy to flush during it's tpc_begin
 phase. This should only be done if a flush is not already in progress,
 as otherwise sqlalchemy would raise an error. self.session._flushing
 is only in 0.5 though, so maybe it is time to up the dependency. Or
 maybe it is impossible for it to be in the flushing state at that
 point anyway, as all user code will have completed.

Correction, this is my misunderstanding of sqlalchemy, as explained on irc:
stepz elro: that's when you do session.begin(autoflush=False)

So I should just make it an unconditional flush()

Laurence
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] zope.sqlalchemy locks up transaction

2008-09-18 Thread Brian Sutherland
On Thu, Sep 18, 2008 at 07:08:56AM +0200, Dieter Maurer wrote:
 Brian Sutherland wrote at 2008-9-17 12:33 +0200:
 On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote:
  Brian Sutherland wrote:
   Hi,
   
   I've recently seen a situation where zope.sqlalchemy locked up the
   transaction machinery. I'm not sure exactly what happened, but have
   attached a failing test for at least one bug which may have caused it.
   Hopefully it's self explanatory;)
   
   If someone could help me solve this, that would be great!
  
  Could you try this with latest trunk. I checked in a fix the other day 
  that may help.
 
 I just checked in a fix, please feel free to comment on/revert it if
 it's not up to standard:)
 
 It looks not yet right to clear the state in tpc_vote when
 a two phase commit is used (which is now supported by SQLAlchemy).

It looks to me that in a two phase commit, the tpc_vote which is called
is:

def tpc_vote(self, trans):
if self.tx is not None: # there may have been no work to do
self.tx.prepare()
self.state = 'voted'

is that what you mean by clearing state?

 In addition, there may be a problem in case session.close() raises
 an exception. Then, _finish would not be called.

I modified my patch, moving session.close() into _finish and actually
calling close() after the state is cleared. That way, the errors can
propagate, but the transaction will not get wedged.

-- 
Brian Sutherland
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] zope.sqlalchemy locks up transaction

2008-09-17 Thread Brian Sutherland
On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote:
 Brian Sutherland wrote:
  Hi,
  
  I've recently seen a situation where zope.sqlalchemy locked up the
  transaction machinery. I'm not sure exactly what happened, but have
  attached a failing test for at least one bug which may have caused it.
  Hopefully it's self explanatory;)
  
  If someone could help me solve this, that would be great!
 
 Could you try this with latest trunk. I checked in a fix the other day 
 that may help.

I just checked in a fix, please feel free to comment on/revert it if
it's not up to standard:)

 
 Laurence
 
 ___
 Zope-Dev maillist  -  Zope-Dev@zope.org
 http://mail.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists - 
  http://mail.zope.org/mailman/listinfo/zope-announce
  http://mail.zope.org/mailman/listinfo/zope )

-- 
Brian Sutherland
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] zope.sqlalchemy locks up transaction

2008-09-17 Thread Laurence Rowe
Brian Sutherland wrote:
 On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote:
 Brian Sutherland wrote:
 Hi,

 I've recently seen a situation where zope.sqlalchemy locked up the
 transaction machinery. I'm not sure exactly what happened, but have
 attached a failing test for at least one bug which may have caused it.
 Hopefully it's self explanatory;)

 If someone could help me solve this, that would be great!
 Could you try this with latest trunk. I checked in a fix the other day 
 that may help.
 
 I just checked in a fix, please feel free to comment on/revert it if
 it's not up to standard:)

Great, I'll take a look at it and do a release some time this week.

Laurence

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] zope.sqlalchemy locks up transaction

2008-09-17 Thread Dieter Maurer
Brian Sutherland wrote at 2008-9-16 09:56 +0200:
I've recently seen a situation where zope.sqlalchemy locked up the
transaction machinery. I'm not sure exactly what happened, but have
attached a failing test for at least one bug which may have caused it.
Hopefully it's self explanatory;)

If someone could help me solve this, that would be great!

We have seen similar problems and hope to understand/work around
them soon.



-- 
Dieter
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] zope.sqlalchemy locks up transaction

2008-09-17 Thread Dieter Maurer
Brian Sutherland wrote at 2008-9-17 12:33 +0200:
On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote:
 Brian Sutherland wrote:
  Hi,
  
  I've recently seen a situation where zope.sqlalchemy locked up the
  transaction machinery. I'm not sure exactly what happened, but have
  attached a failing test for at least one bug which may have caused it.
  Hopefully it's self explanatory;)
  
  If someone could help me solve this, that would be great!
 
 Could you try this with latest trunk. I checked in a fix the other day 
 that may help.

I just checked in a fix, please feel free to comment on/revert it if
it's not up to standard:)

It looks not yet right to clear the state in tpc_vote when
a two phase commit is used (which is now supported by SQLAlchemy).

In addition, there may be a problem in case session.close() raises
an exception. Then, _finish would not be called.



-- 
Dieter
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )