Re: [Zope] [Further investigations] Re: A question about __setstate__ in Shared/DC/ZRDB/Connection.py
Sorry to bother you guys: is there any suitable workaround for this? I tried the suggested one, but it does not work. Regards Marco -- Marco Bizzarri http://notenotturne.blogspot.com/ http://iliveinpisa.blogspot.com/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] [Further investigations] Re: A question about __setstate__ in Shared/DC/ZRDB/Connection.py
On Sat, Sep 20, 2008 at 8:24 AM, Dieter Maurer <[EMAIL PROTECTED]> wrote: > Tres Seaver wrote at 2008-9-19 09:33 -0400: >> ... >>There is a special 'STICKY' state which prevents ghostifying, but it >>can't be set from Python code. You could, however, set '_p_changed' on >>the connection at the beginning of the method, and then delete it at the >>end: changed objects can't be ghostified. E.g.: >> >> def my_method(self): >> self.connection._p_changed = 1 >> try: >> self.sql() >> # now do the stuff which used to ghostify the connection >> finally: >> del self.connection._p_changed > > Are you sure that this works? > > According to my (not very clear) memory, "_p_changed" in a C level attribute > (that is definite) which could be set to "1" from application level > but not reset (that is not sure). > > > > -- > Dieter > ___ As I said in my previous post, I modified my test case to check if this works, but I'm afraid it does not (i.e. I can still see two connections at the database). Regards Marco -- Marco Bizzarri http://notenotturne.blogspot.com/ http://iliveinpisa.blogspot.com/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] [Further investigations] Re: A question about __setstate__ in Shared/DC/ZRDB/Connection.py
Tres Seaver wrote at 2008-9-19 10:15 -0400: > ... >There are two problems here: > > - We need a way to keep the Pdata objects from evicting "precious" > objects; ideally, Pdata instances would never be added to the cache > at all. I worked a bit on a spike in which the Pdata iterator part > would use a one-off connection with a zero-sized cache, but got > stuck somewhere; maybe somebody else can make it work. But other large operations, too, can flush objects from the cache -- e.g. large scale "catalog_object". Thus, special treatment of "Pdata" can only reduce the risk but not remove it. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] [Further investigations] Re: A question about __setstate__ in Shared/DC/ZRDB/Connection.py
Tres Seaver wrote at 2008-9-19 09:33 -0400: > ... >There is a special 'STICKY' state which prevents ghostifying, but it >can't be set from Python code. You could, however, set '_p_changed' on >the connection at the beginning of the method, and then delete it at the >end: changed objects can't be ghostified. E.g.: > > def my_method(self): > self.connection._p_changed = 1 > try: > self.sql() > # now do the stuff which used to ghostify the connection > finally: > del self.connection._p_changed Are you sure that this works? According to my (not very clear) memory, "_p_changed" in a C level attribute (that is definite) which could be set to "1" from application level but not reset (that is not sure). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] [Further investigations] Re: A question about __setstate__ in Shared/DC/ZRDB/Connection.py
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Marco Bizzarri wrote: > Thanks for the suggestion, Tres, I'm trying it right now. > I think this could be responsible for the problem I had a few months > ago, under the name: "Asking advice on a Zope "stuck" (or: what did I > do wrong?)" > Do you think there will be some sort of "general" solution to the > problem? I mean, the problem is actually that there are some objects > which should not be ghostified, or am I wrong? There are two problems here: - Some objects need to be able to mark themselves as "sticky" for at least the duration of a transaction; my workaround is hackish, because if you omit the 'del conn._p_changed' it causes the object to be written needlessly; likewise, if the conn object *is* actually written to during the transaction, those changes will be discarded. - We need a way to keep the Pdata objects from evicting "precious" objects; ideally, Pdata instances would never be added to the cache at all. I worked a bit on a spike in which the Pdata iterator part would use a one-off connection with a zero-sized cache, but got stuck somewhere; maybe somebody else can make it work. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI07Po+gerLs4ltQ4RAvz1AJ9kZ+XFucS9Eq4rFkGQ7NI12F9ItACeLE9/ YkOPDbgH5UbO+uHQ4IDIyHU= =y5iN -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] [Further investigations] Re: A question about __setstate__ in Shared/DC/ZRDB/Connection.py
On Fri, Sep 19, 2008 at 3:33 PM, Tres Seaver <[EMAIL PROTECTED]> wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Marco Bizzarri wrote:
>> On Fri, Sep 19, 2008 at 9:23 AM, Marco Bizzarri
>> <[EMAIL PROTECTED]> wrote:
>>> Hi all.
>>>
>>> I'm working on an application which uses Zope (2.8, at the moment) and
>>> ZPsycopgDA (toghter with a number of other products).
>>>
>>> While writing an acceptance test, I encountered a strange problem: the
>>> test locks up.
>>>
>>> A further investigation shown that there were two connections at the
>>> database; one of them was not committed, the other one was blocked
>>> waiting for the other to commit.
>>>
>>> I therefore used the pdb in order to stop the execution of the test
>>> inside the connect method of the ZPsycopgDA.DA. Once I had that
>>> breakpoint, I was able to get the logs of the two transactions on the
>>> database, and I had the confirmation that indeed there were two
>>> different transactions.
>>>
>>> So, I wondered what could possibily happen, I mean why during a test
>>> there could be a second connect to the database.
>>>
>>> I issued a "bt" to see the stack of calls leading to the connect, and
>>> what I could see was that the coonect was called inside the
>>> __setstate__ method of Shared/DC/ZRDB/Connection.py.
>>>
>>> I assume therefore that the ZPsycopgDA object has been "ghostified",
>>> during the transaction. But this "assumption" is not supported by any
>>> evidence. In particular, it is not supported by my knowledge of the
>>> internal behaviour of ZODB on objects during a single transaction.
>>>
>>> Can anyone provide suggestion on this topic?
>>>
>>> Regards
>>> Marco
>>> --
>>> Marco Bizzarri
>>> http://notenotturne.blogspot.com/
>>> http://iliveinpisa.blogspot.com/
>>>
>>
>> I did further investigation on the topic, and I think I've pinned the
>> problem. I don't know the solution, but I can reproduce the problem
>> with a small sample. Here is the sample:
>>
>>
>> import os
>> import sys
>> import unittest
>>
>> if __name__ == '__main__':
>> execfile(os.path.join(sys.path[0], '../framework.py'))
>>
>> from Testing import ZopeTestCase
>>
>> from OFS import Image
>>
>> from Products.ZPsycopgDA.DA import manage_addZPsycopgConnection
>> from Products.ZSQLMethods import SQL
>>
>>
>> class DoubleTransactionTest(ZopeTestCase.ZopeTestCase):
>>
>> def _add_big_image(self, value, data):
>> Image.manage_addFile(self.app, "f%06s" % value, data , "a title")
>>
>> def test_showdouble(self):
>> manage_addZPsycopgConnection(self.app, "db_connection", "",
>> "host=localhost user=postgres dbname=template1")
>> self.app._setObject('sql', SQL.SQL("sql", "", "db_connection",
>> "", "select * from pg_tables"))
>> self.app.sql()
>> data = "*" * (1 << 20)
>> for x in range(1000):
>> self._add_big_image(x, data)
>> print "Added %s " % x
>> self.app.sql()
>>
>> if __name__ == '__main__':
>> unittest.main()
>>
>>
>> I'm doing three things here:
>>
>> - creating a db connection
>> - making a query to the db (this causes a transaction to begin)
>> - creating a lot of "big" files (expecially, larger than 2 * 2 ^ 16 *)
>> - making another query to the db;
>>
>> Once I create a big file I fall into the following branch inside the
>> OFS.Image._read_data
>>
>>
>> if size <= 2*n:
>> seek(0)
>> if size < n: return read(size), size
>> return Pdata(read(size)), size
>>
>> # Make sure we have an _p_jar, even if we are a new object, by
>> # doing a sub-transaction commit.
>> transaction.savepoint(optimistic=True)
>>
>> This causes, at the end, to call the ZODB.Connection.savepoint which,
>> just before returning, calls a cacheGC to be called, which, I'm
>> afraid, causes the db_connection to be "sent" out of the cache itself,
>> thus leaving it without the _v_ attributes.
>>
>> Hope this can help in giving suggestions.
>
> Thanks for digging further into it; I couldn't imagine how that was
> occurring. In this case, the large number of created Pdata objects (one
> per 64k chunk of each of your images) are causing your connection object
> to be evicted from the cache at one of the savepoints, and thus
> ghostified (which is where it loses its volatiles).
>
> There is a special 'STICKY' state which prevents ghostifying, but it
> can't be set from Python code. You could, however, set '_p_changed' on
> the connection at the beginning of the method, and then delete it at the
> end: changed objects can't be ghostified. E.g.:
>
> def my_method(self):
> self.connection._p_changed = 1
> try:
> self.sql()
> # now do the stuff which used to ghostify the connection
> finally:
> del self.connection._p_changed
>
> It is a nasty workaround, but should help prevent the lockup.
>
> Tres.
> - --
> ==
Re: [Zope] [Further investigations] Re: A question about __setstate__ in Shared/DC/ZRDB/Connection.py
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Marco Bizzarri wrote:
> On Fri, Sep 19, 2008 at 9:23 AM, Marco Bizzarri
> <[EMAIL PROTECTED]> wrote:
>> Hi all.
>>
>> I'm working on an application which uses Zope (2.8, at the moment) and
>> ZPsycopgDA (toghter with a number of other products).
>>
>> While writing an acceptance test, I encountered a strange problem: the
>> test locks up.
>>
>> A further investigation shown that there were two connections at the
>> database; one of them was not committed, the other one was blocked
>> waiting for the other to commit.
>>
>> I therefore used the pdb in order to stop the execution of the test
>> inside the connect method of the ZPsycopgDA.DA. Once I had that
>> breakpoint, I was able to get the logs of the two transactions on the
>> database, and I had the confirmation that indeed there were two
>> different transactions.
>>
>> So, I wondered what could possibily happen, I mean why during a test
>> there could be a second connect to the database.
>>
>> I issued a "bt" to see the stack of calls leading to the connect, and
>> what I could see was that the coonect was called inside the
>> __setstate__ method of Shared/DC/ZRDB/Connection.py.
>>
>> I assume therefore that the ZPsycopgDA object has been "ghostified",
>> during the transaction. But this "assumption" is not supported by any
>> evidence. In particular, it is not supported by my knowledge of the
>> internal behaviour of ZODB on objects during a single transaction.
>>
>> Can anyone provide suggestion on this topic?
>>
>> Regards
>> Marco
>> --
>> Marco Bizzarri
>> http://notenotturne.blogspot.com/
>> http://iliveinpisa.blogspot.com/
>>
>
> I did further investigation on the topic, and I think I've pinned the
> problem. I don't know the solution, but I can reproduce the problem
> with a small sample. Here is the sample:
>
>
> import os
> import sys
> import unittest
>
> if __name__ == '__main__':
> execfile(os.path.join(sys.path[0], '../framework.py'))
>
> from Testing import ZopeTestCase
>
> from OFS import Image
>
> from Products.ZPsycopgDA.DA import manage_addZPsycopgConnection
> from Products.ZSQLMethods import SQL
>
>
> class DoubleTransactionTest(ZopeTestCase.ZopeTestCase):
>
> def _add_big_image(self, value, data):
> Image.manage_addFile(self.app, "f%06s" % value, data , "a title")
>
> def test_showdouble(self):
> manage_addZPsycopgConnection(self.app, "db_connection", "",
> "host=localhost user=postgres dbname=template1")
> self.app._setObject('sql', SQL.SQL("sql", "", "db_connection",
> "", "select * from pg_tables"))
> self.app.sql()
> data = "*" * (1 << 20)
> for x in range(1000):
> self._add_big_image(x, data)
> print "Added %s " % x
> self.app.sql()
>
> if __name__ == '__main__':
> unittest.main()
>
>
> I'm doing three things here:
>
> - creating a db connection
> - making a query to the db (this causes a transaction to begin)
> - creating a lot of "big" files (expecially, larger than 2 * 2 ^ 16 *)
> - making another query to the db;
>
> Once I create a big file I fall into the following branch inside the
> OFS.Image._read_data
>
>
> if size <= 2*n:
> seek(0)
> if size < n: return read(size), size
> return Pdata(read(size)), size
>
> # Make sure we have an _p_jar, even if we are a new object, by
> # doing a sub-transaction commit.
> transaction.savepoint(optimistic=True)
>
> This causes, at the end, to call the ZODB.Connection.savepoint which,
> just before returning, calls a cacheGC to be called, which, I'm
> afraid, causes the db_connection to be "sent" out of the cache itself,
> thus leaving it without the _v_ attributes.
>
> Hope this can help in giving suggestions.
Thanks for digging further into it; I couldn't imagine how that was
occurring. In this case, the large number of created Pdata objects (one
per 64k chunk of each of your images) are causing your connection object
to be evicted from the cache at one of the savepoints, and thus
ghostified (which is where it loses its volatiles).
There is a special 'STICKY' state which prevents ghostifying, but it
can't be set from Python code. You could, however, set '_p_changed' on
the connection at the beginning of the method, and then delete it at the
end: changed objects can't be ghostified. E.g.:
def my_method(self):
self.connection._p_changed = 1
try:
self.sql()
# now do the stuff which used to ghostify the connection
finally:
del self.connection._p_changed
It is a nasty workaround, but should help prevent the lockup.
Tres.
- --
===
Tres Seaver +1 540-429-0999 [EMAIL PROTECTED]
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment
[Zope] [Further investigations] Re: A question about __setstate__ in Shared/DC/ZRDB/Connection.py
On Fri, Sep 19, 2008 at 9:23 AM, Marco Bizzarri
<[EMAIL PROTECTED]> wrote:
> Hi all.
>
> I'm working on an application which uses Zope (2.8, at the moment) and
> ZPsycopgDA (toghter with a number of other products).
>
> While writing an acceptance test, I encountered a strange problem: the
> test locks up.
>
> A further investigation shown that there were two connections at the
> database; one of them was not committed, the other one was blocked
> waiting for the other to commit.
>
> I therefore used the pdb in order to stop the execution of the test
> inside the connect method of the ZPsycopgDA.DA. Once I had that
> breakpoint, I was able to get the logs of the two transactions on the
> database, and I had the confirmation that indeed there were two
> different transactions.
>
> So, I wondered what could possibily happen, I mean why during a test
> there could be a second connect to the database.
>
> I issued a "bt" to see the stack of calls leading to the connect, and
> what I could see was that the coonect was called inside the
> __setstate__ method of Shared/DC/ZRDB/Connection.py.
>
> I assume therefore that the ZPsycopgDA object has been "ghostified",
> during the transaction. But this "assumption" is not supported by any
> evidence. In particular, it is not supported by my knowledge of the
> internal behaviour of ZODB on objects during a single transaction.
>
> Can anyone provide suggestion on this topic?
>
> Regards
> Marco
> --
> Marco Bizzarri
> http://notenotturne.blogspot.com/
> http://iliveinpisa.blogspot.com/
>
I did further investigation on the topic, and I think I've pinned the
problem. I don't know the solution, but I can reproduce the problem
with a small sample. Here is the sample:
import os
import sys
import unittest
if __name__ == '__main__':
execfile(os.path.join(sys.path[0], '../framework.py'))
from Testing import ZopeTestCase
from OFS import Image
from Products.ZPsycopgDA.DA import manage_addZPsycopgConnection
from Products.ZSQLMethods import SQL
class DoubleTransactionTest(ZopeTestCase.ZopeTestCase):
def _add_big_image(self, value, data):
Image.manage_addFile(self.app, "f%06s" % value, data , "a title")
def test_showdouble(self):
manage_addZPsycopgConnection(self.app, "db_connection", "",
"host=localhost user=postgres dbname=template1")
self.app._setObject('sql', SQL.SQL("sql", "", "db_connection",
"", "select * from pg_tables"))
self.app.sql()
data = "*" * (1 << 20)
for x in range(1000):
self._add_big_image(x, data)
print "Added %s " % x
self.app.sql()
if __name__ == '__main__':
unittest.main()
I'm doing three things here:
- creating a db connection
- making a query to the db (this causes a transaction to begin)
- creating a lot of "big" files (expecially, larger than 2 * 2 ^ 16 *)
- making another query to the db;
Once I create a big file I fall into the following branch inside the
OFS.Image._read_data
if size <= 2*n:
seek(0)
if size < n: return read(size), size
return Pdata(read(size)), size
# Make sure we have an _p_jar, even if we are a new object, by
# doing a sub-transaction commit.
transaction.savepoint(optimistic=True)
This causes, at the end, to call the ZODB.Connection.savepoint which,
just before returning, calls a cacheGC to be called, which, I'm
afraid, causes the db_connection to be "sent" out of the cache itself,
thus leaving it without the _v_ attributes.
Hope this can help in giving suggestions.
Regards
Marco
--
Marco Bizzarri
http://notenotturne.blogspot.com/
http://iliveinpisa.blogspot.com/
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
