On Fri, Sep 19, 2008 at 3:33 PM, Tres Seaver <[EMAIL PROTECTED]> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> Marco Bizzarri wrote:
>> On Fri, Sep 19, 2008 at 9:23 AM, Marco Bizzarri
>> <[EMAIL PROTECTED]> wrote:
>>> Hi all.
>>> I'm working on an application which uses Zope (2.8, at the moment) and
>>> ZPsycopgDA (toghter with a number of other products).
>>> While writing an acceptance test, I encountered a strange problem: the
>>> test locks up.
>>> A further investigation shown that there were two connections at the
>>> database; one of them was not committed, the other one was blocked
>>> waiting for the other to commit.
>>> I therefore used the pdb in order to stop the execution of the test
>>> inside the connect method of the ZPsycopgDA.DA. Once I had that
>>> breakpoint, I was able to get the logs of the two transactions on the
>>> database, and I had the confirmation that indeed there were two
>>> different transactions.
>>> So, I wondered what could possibily happen, I mean why during a test
>>> there could be a second connect to the database.
>>> I issued a "bt" to see the stack of calls leading to the connect, and
>>> what I could see was that the coonect was called inside the
>>> __setstate__ method of Shared/DC/ZRDB/Connection.py.
>>> I assume therefore that the ZPsycopgDA object has been "ghostified",
>>> during the transaction. But this "assumption" is not supported by any
>>> evidence. In particular, it is not supported by my knowledge of the
>>> internal behaviour of ZODB on objects during a single transaction.
>>> Can anyone provide suggestion on this topic?
>>> Marco Bizzarri
>> I did further investigation on the topic, and I think I've pinned the
>> problem. I don't know the solution, but I can reproduce the problem
>> with a small sample. Here is the sample:
>> import os
>> import sys
>> import unittest
>> if __name__ == '__main__':
>> execfile(os.path.join(sys.path, '../framework.py'))
>> from Testing import ZopeTestCase
>> from OFS import Image
>> from Products.ZPsycopgDA.DA import manage_addZPsycopgConnection
>> from Products.ZSQLMethods import SQL
>> class DoubleTransactionTest(ZopeTestCase.ZopeTestCase):
>> def _add_big_image(self, value, data):
>> Image.manage_addFile(self.app, "f%06s" % value, data , "a title")
>> def test_showdouble(self):
>> manage_addZPsycopgConnection(self.app, "db_connection", "",
>> "host=localhost user=postgres dbname=template1")
>> self.app._setObject('sql', SQL.SQL("sql", "", "db_connection",
>> "", "select * from pg_tables"))
>> data = "*" * (1 << 20)
>> for x in range(1000):
>> self._add_big_image(x, data)
>> print "Added %s " % x
>> if __name__ == '__main__':
>> I'm doing three things here:
>> - creating a db connection
>> - making a query to the db (this causes a transaction to begin)
>> - creating a lot of "big" files (expecially, larger than 2 * 2 ^ 16 *)
>> - making another query to the db;
>> Once I create a big file I fall into the following branch inside the
>> if size <= 2*n:
>> if size < n: return read(size), size
>> return Pdata(read(size)), size
>> # Make sure we have an _p_jar, even if we are a new object, by
>> # doing a sub-transaction commit.
>> This causes, at the end, to call the ZODB.Connection.savepoint which,
>> just before returning, calls a cacheGC to be called, which, I'm
>> afraid, causes the db_connection to be "sent" out of the cache itself,
>> thus leaving it without the _v_ attributes.
>> Hope this can help in giving suggestions.
> Thanks for digging further into it; I couldn't imagine how that was
> occurring. In this case, the large number of created Pdata objects (one
> per 64k chunk of each of your images) are causing your connection object
> to be evicted from the cache at one of the savepoints, and thus
> ghostified (which is where it loses its volatiles).
> There is a special 'STICKY' state which prevents ghostifying, but it
> can't be set from Python code. You could, however, set '_p_changed' on
> the connection at the beginning of the method, and then delete it at the
> end: changed objects can't be ghostified. E.g.:
> def my_method(self):
> self.connection._p_changed = 1
> # now do the stuff which used to ghostify the connection
> del self.connection._p_changed
> It is a nasty workaround, but should help prevent the lockup.
> - --
> Tres Seaver +1 540-429-0999 [EMAIL PROTECTED]
> Palladion Software "Excellence by Design" http://palladion.com
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> -----END PGP SIGNATURE-----
Thanks for the suggestion, Tres, I'm trying it right now.
I think this could be responsible for the problem I had a few months
ago, under the name: "Asking advice on a Zope "stuck" (or: what did I
Do you think there will be some sort of "general" solution to the
problem? I mean, the problem is actually that there are some objects
which should not be ghostified, or am I wrong?
Zope maillist - Zope@zope.org
** No cross posts or HTML encoding! **
(Related lists -