Re: [Zope-dev] zope.sqlalchemy locks up transaction
Am Donnerstag 18 September 2008 07:08:56 schrieb Dieter Maurer: Brian Sutherland wrote at 2008-9-17 12:33 +0200: On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote: Brian Sutherland wrote: Hi, I've recently seen a situation where zope.sqlalchemy locked up the transaction machinery. I'm not sure exactly what happened, but have attached a failing test for at least one bug which may have caused it. Hopefully it's self explanatory;) If someone could help me solve this, that would be great! Could you try this with latest trunk. I checked in a fix the other day that may help. I just checked in a fix, please feel free to comment on/revert it if it's not up to standard:) It looks not yet right to clear the state in tpc_vote when a two phase commit is used (which is now supported by SQLAlchemy). In addition, there may be a problem in case session.close() raises an exception. Then, _finish would not be called. This may not be connected, but I just wanted to point out a problem with configurations with autoflush=False. If I change some database object attributes in the session but do not call session.flush(), the data is not stored into the database, as no commit() is called, which would then flush out/commit the data. So perhaps this is something to think about, or, at least something to be denoted in the documentation. I circumvented the problem by setting autoflush=True and modifying my code so that it works (although it's quite complicated). Best Regards, Hermann -- [EMAIL PROTECTED] GPG key ID: 299893C7 (on keyservers) FP: 0124 2584 8809 EF2A DBF9 4902 64B4 D16B 2998 93C7 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.sqlalchemy locks up transaction
2008/9/19 Hermann Himmelbauer [EMAIL PROTECTED]: Am Donnerstag 18 September 2008 07:08:56 schrieb Dieter Maurer: Brian Sutherland wrote at 2008-9-17 12:33 +0200: On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote: Brian Sutherland wrote: Hi, I've recently seen a situation where zope.sqlalchemy locked up the transaction machinery. I'm not sure exactly what happened, but have attached a failing test for at least one bug which may have caused it. Hopefully it's self explanatory;) If someone could help me solve this, that would be great! Could you try this with latest trunk. I checked in a fix the other day that may help. I just checked in a fix, please feel free to comment on/revert it if it's not up to standard:) It looks not yet right to clear the state in tpc_vote when a two phase commit is used (which is now supported by SQLAlchemy). In addition, there may be a problem in case session.close() raises an exception. Then, _finish would not be called. This may not be connected, but I just wanted to point out a problem with configurations with autoflush=False. If I change some database object attributes in the session but do not call session.flush(), the data is not stored into the database, as no commit() is called, which would then flush out/commit the data. So perhaps this is something to think about, or, at least something to be denoted in the documentation. I circumvented the problem by setting autoflush=True and modifying my code so that it works (although it's quite complicated). I don't think this is related to the above problem, but it may be a bug in SQLAlchemy 0.4 whose behaviour I have replicated. In 0.4 orm.session during a commit flush is called: 245 if self.autoflush: 246 self.session.flush() whereas in 0.5 flush is called unconditionally so long as there is not a flush in progress. 358 if not self.session._flushing: 359 self.session.flush() The documentation for 0.4 appears to be misleading: commit() serves two purposes; it issues a flush() unconditionally to persist any remaining pending changes. I think I should change zope.sqlalchemy to flush during it's tpc_begin phase. This should only be done if a flush is not already in progress, as otherwise sqlalchemy would raise an error. self.session._flushing is only in 0.5 though, so maybe it is time to up the dependency. Or maybe it is impossible for it to be in the flushing state at that point anyway, as all user code will have completed. Laurence ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.sqlalchemy locks up transaction
Am Freitag 19 September 2008 11:59:55 schrieb Laurence Rowe: 2008/9/19 Hermann Himmelbauer [EMAIL PROTECTED]: Am Donnerstag 18 September 2008 07:08:56 schrieb Dieter Maurer: Brian Sutherland wrote at 2008-9-17 12:33 +0200: On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote: Brian Sutherland wrote: Hi, I've recently seen a situation where zope.sqlalchemy locked up the transaction machinery. I'm not sure exactly what happened, but have attached a failing test for at least one bug which may have caused it. Hopefully it's self explanatory;) If someone could help me solve this, that would be great! Could you try this with latest trunk. I checked in a fix the other day that may help. I just checked in a fix, please feel free to comment on/revert it if it's not up to standard:) It looks not yet right to clear the state in tpc_vote when a two phase commit is used (which is now supported by SQLAlchemy). In addition, there may be a problem in case session.close() raises an exception. Then, _finish would not be called. This may not be connected, but I just wanted to point out a problem with configurations with autoflush=False. If I change some database object attributes in the session but do not call session.flush(), the data is not stored into the database, as no commit() is called, which would then flush out/commit the data. So perhaps this is something to think about, or, at least something to be denoted in the documentation. I circumvented the problem by setting autoflush=True and modifying my code so that it works (although it's quite complicated). I don't think this is related to the above problem, but it may be a bug in SQLAlchemy 0.4 whose behaviour I have replicated. In 0.4 orm.session during a commit flush is called: 245 if self.autoflush: 246 self.session.flush() whereas in 0.5 flush is called unconditionally so long as there is not a flush in progress. 358 if not self.session._flushing: 359 self.session.flush() The documentation for 0.4 appears to be misleading: commit() serves two purposes; it issues a flush() unconditionally to persist any remaining pending changes. Ah, I see, that's really misleading. I think I should change zope.sqlalchemy to flush during it's tpc_begin phase. This should only be done if a flush is not already in progress, as otherwise sqlalchemy would raise an error. That's maybe a good idea. self.session._flushing is only in 0.5 though, so maybe it is time to up the dependency. What do you mean with up the dependency? If you mean dropping support for SA 0.4, please, *please* don't do that (0.5's autoflushing breaks things here and Database support for MaxDB/SAPDB is somehow flawed.). Due to many incompatibilites, I cannot switch to SA 0.5, so I'd really appreciate support for SA 0.4! Or maybe it is impossible for it to be in the flushing state at that point anyway, as all user code will have completed. If I understand it right, the problem does not occur in SA 0.5, right? If that's the case, maybe it is enough to outline the behaviour of SA 0.4 in the documentation and leave it as it is. Best Regards, Hermann -- [EMAIL PROTECTED] GPG key ID: 299893C7 (on keyservers) FP: 0124 2584 8809 EF2A DBF9 4902 64B4 D16B 2998 93C7 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.sqlalchemy locks up transaction
2008/9/19 Laurence Rowe [EMAIL PROTECTED]: 2008/9/19 Hermann Himmelbauer [EMAIL PROTECTED]: Am Donnerstag 18 September 2008 07:08:56 schrieb Dieter Maurer: Brian Sutherland wrote at 2008-9-17 12:33 +0200: On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote: Brian Sutherland wrote: Hi, I've recently seen a situation where zope.sqlalchemy locked up the transaction machinery. I'm not sure exactly what happened, but have attached a failing test for at least one bug which may have caused it. Hopefully it's self explanatory;) If someone could help me solve this, that would be great! Could you try this with latest trunk. I checked in a fix the other day that may help. I just checked in a fix, please feel free to comment on/revert it if it's not up to standard:) It looks not yet right to clear the state in tpc_vote when a two phase commit is used (which is now supported by SQLAlchemy). In addition, there may be a problem in case session.close() raises an exception. Then, _finish would not be called. This may not be connected, but I just wanted to point out a problem with configurations with autoflush=False. If I change some database object attributes in the session but do not call session.flush(), the data is not stored into the database, as no commit() is called, which would then flush out/commit the data. So perhaps this is something to think about, or, at least something to be denoted in the documentation. I circumvented the problem by setting autoflush=True and modifying my code so that it works (although it's quite complicated). I don't think this is related to the above problem, but it may be a bug in SQLAlchemy 0.4 whose behaviour I have replicated. In 0.4 orm.session during a commit flush is called: 245 if self.autoflush: 246 self.session.flush() whereas in 0.5 flush is called unconditionally so long as there is not a flush in progress. 358 if not self.session._flushing: 359 self.session.flush() The documentation for 0.4 appears to be misleading: commit() serves two purposes; it issues a flush() unconditionally to persist any remaining pending changes. I think I should change zope.sqlalchemy to flush during it's tpc_begin phase. This should only be done if a flush is not already in progress, as otherwise sqlalchemy would raise an error. self.session._flushing is only in 0.5 though, so maybe it is time to up the dependency. Or maybe it is impossible for it to be in the flushing state at that point anyway, as all user code will have completed. Correction, this is my misunderstanding of sqlalchemy, as explained on irc: stepz elro: that's when you do session.begin(autoflush=False) So I should just make it an unconditional flush() Laurence ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.sqlalchemy locks up transaction
On Thu, Sep 18, 2008 at 07:08:56AM +0200, Dieter Maurer wrote: Brian Sutherland wrote at 2008-9-17 12:33 +0200: On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote: Brian Sutherland wrote: Hi, I've recently seen a situation where zope.sqlalchemy locked up the transaction machinery. I'm not sure exactly what happened, but have attached a failing test for at least one bug which may have caused it. Hopefully it's self explanatory;) If someone could help me solve this, that would be great! Could you try this with latest trunk. I checked in a fix the other day that may help. I just checked in a fix, please feel free to comment on/revert it if it's not up to standard:) It looks not yet right to clear the state in tpc_vote when a two phase commit is used (which is now supported by SQLAlchemy). It looks to me that in a two phase commit, the tpc_vote which is called is: def tpc_vote(self, trans): if self.tx is not None: # there may have been no work to do self.tx.prepare() self.state = 'voted' is that what you mean by clearing state? In addition, there may be a problem in case session.close() raises an exception. Then, _finish would not be called. I modified my patch, moving session.close() into _finish and actually calling close() after the state is cleared. That way, the errors can propagate, but the transaction will not get wedged. -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.sqlalchemy locks up transaction
On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote: Brian Sutherland wrote: Hi, I've recently seen a situation where zope.sqlalchemy locked up the transaction machinery. I'm not sure exactly what happened, but have attached a failing test for at least one bug which may have caused it. Hopefully it's self explanatory;) If someone could help me solve this, that would be great! Could you try this with latest trunk. I checked in a fix the other day that may help. I just checked in a fix, please feel free to comment on/revert it if it's not up to standard:) Laurence ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.sqlalchemy locks up transaction
Brian Sutherland wrote: On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote: Brian Sutherland wrote: Hi, I've recently seen a situation where zope.sqlalchemy locked up the transaction machinery. I'm not sure exactly what happened, but have attached a failing test for at least one bug which may have caused it. Hopefully it's self explanatory;) If someone could help me solve this, that would be great! Could you try this with latest trunk. I checked in a fix the other day that may help. I just checked in a fix, please feel free to comment on/revert it if it's not up to standard:) Great, I'll take a look at it and do a release some time this week. Laurence ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.sqlalchemy locks up transaction
Brian Sutherland wrote at 2008-9-16 09:56 +0200: I've recently seen a situation where zope.sqlalchemy locked up the transaction machinery. I'm not sure exactly what happened, but have attached a failing test for at least one bug which may have caused it. Hopefully it's self explanatory;) If someone could help me solve this, that would be great! We have seen similar problems and hope to understand/work around them soon. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.sqlalchemy locks up transaction
Brian Sutherland wrote at 2008-9-17 12:33 +0200: On Tue, Sep 16, 2008 at 02:01:07PM +0100, Laurence Rowe wrote: Brian Sutherland wrote: Hi, I've recently seen a situation where zope.sqlalchemy locked up the transaction machinery. I'm not sure exactly what happened, but have attached a failing test for at least one bug which may have caused it. Hopefully it's self explanatory;) If someone could help me solve this, that would be great! Could you try this with latest trunk. I checked in a fix the other day that may help. I just checked in a fix, please feel free to comment on/revert it if it's not up to standard:) It looks not yet right to clear the state in tpc_vote when a two phase commit is used (which is now supported by SQLAlchemy). In addition, there may be a problem in case session.close() raises an exception. Then, _finish would not be called. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )