AW: AW: AW: AW: [firebird-python] FDB v1.4.2 [1 Attachment]
Hi Pavel,
I believe that I have found a solution by shuffling the __wait_for_events()
invoke. This is the patch (also available as attachment):
--- C:/Python27/Lib/site-packages/fdb-1.4.3-py2.7.egg/fdb/fbcore.py Mo Nov
17 08:27:57 2014
+++ C:/Python27/Lib/site-packages/fdb-1.4.3-py2.7.egg/fdb/fbcore.py Mo Nov
17 08:29:08 2014
@@ -1729,9 +1729,10 @@ class EventBlock(object):
self.buf_length = api.isc_event_block(ctypes.pointer(self.event_buf),
ctypes.pointer(self.result_buf),
*[b(x) for x in event_names])
-self.__wait_for_events()
def __lt__(self,other):
return self.event_id.value other.event_id.value
+def begin(self):
+self.__wait_for_events()
def __wait_for_events(self):
api.isc_que_events(self._isc_status,self._db_handle,self.event_id,
self.buf_length,self.event_buf,
@@ -1841,7 +1842,9 @@ class EventConduit(object):
for block_events in self.__blocks:
-self.__event_blocks.append(EventBlock(self.__queue,
self._db_handle, block_events))
+event_block = EventBlock(self.__queue, self._db_handle,
block_events)
+self.__event_blocks.append(event_block)
+event_block.begin()
With this, even after 5+ events raised and processed and 2400+ event
conduits created and disposed, gc.garbage doesn't contain EventBlock items
anymore. *HOORAY!* :-)
While inspecting gc.garbage I have further noticed that it does contain several
fdb.fbcore.PreparedStatement items. According to
https://docs.python.org/2/library/gc.html#gc.garbage :
By default, this list contains only objects with __del__() methods. [...] Note
that these objects are kept alive even so by virtue of being in the garbage
list, so they should be removed from garbage too. For example, after breaking
cycles, do del gc.garbage[:] to empty the list. It’s generally better to avoid
the issue by not creating cycles containing objects with __del__() methods, and
garbage can be examined in that case to verify that no such cycles are being
created.
To me some obvious questions arise:
1] Is this behavior expected?
2] If [1], does fdb clean up gc.garbage from time to time?
3] If not [1], should fdb.fbcore.PreparedStatement better not override __del__
such that gc can do its job?
4] If not [2] and not [3], should fdb clean up gc.garbage?
Please note also that the memory foodprint of the fdb.fbcore.PreparedStatement
items in question is minimal because all containing data in
fdb.fbcore.PreparedStatement is freed. Due to this this issue is not
problematic in normal environments, but might become critical in an
environment that runs for a long time under heavy load.
Cheers,
Dominik
Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com]
Gesendet: Freitag, 14. November 2014 15:03
An: firebird-python@yahoogroups.com
Betreff: Re: AW: AW: AW: [firebird-python] FDB v1.4.2
Hi,
Can you try to add a call to self.close() before exception is raised in
__wait_for_events ? Does it helps?
if db_api_error(self._isc_status):
self.close()
raise exception_from_status(DatabaseError, self._isc_status, Error
while waiting for events:)
best regards
Pavel
Dne 14.11.2014 v 14:08 Dominik Psenner dominik.psen...@topcontrol.it
[firebird-python] napsal(a):
I refreshed the package and it looks like the memory leak is almost gone. I
can observe that the number of EventBlock’s left in memory are equal to the
count of errors raised with the invoke of begin() (namely one of: unknown ISC
error 0, error reading from connection, error writing to connection).
So there’s still a troublesome execution path left and it took some time
until I had wrapped my head around it. According to the stack trace the
exception is thrown in the constructor of EventBlock:
File dbEvents.py, line 103, in run
event_listener.begin()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1844, in begin
self.__event_blocks.append(EventBlock(self.__queue, self._db_handle,
block_events))
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1732, in __init__
self.__wait_for_events()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1741, in __wait_for_events
Error while waiting for events:)
DatabaseError: ('Error while waiting for events:\n- SQLCODE: -902\n- Error
reading data from the connection.', -902, 335544726)
That would be right here where the EventBlock should be appended to the list
of event blocks:
for block_events in self.__blocks:
self.__event_blocks.append(EventBlock(self.__queue, self._db_handle,
block_events))
But like this an initialized and broken EventBlock is never appended to the
__event_blocks list and therefore EventBlock.close() is never invoked. ☹
Von: firebird-python@yahoogroups.com
AW: AW: AW: AW: [firebird-python] FDB v1.4.2
*hm* .. not really.. then an AttributeError pops up:
2014-11-14 15:15:08.498000 50/51 (1|50:51|3|0)
event_conduit(['TC_IMPORT_QUEUE_UPDATED'])Exception in thread Thread-6:
Traceback (most recent call last):
File C:\Python27\lib\threading.py, line 810, in __bootstrap_inner
self.run()
File C:\Python27\lib\threading.py, line 763, in run
self.__target(*self.__args, **self.__kwargs)
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1831, in event_process
events = data.count_and_reregister()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1751, in count_and_reregister
self.__wait_for_events()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1742, in __wait_for_events
Error while waiting for events:)
DatabaseError: ('Error while waiting for events:\n- SQLCODE: 0\n- unknown ISC
error 0', 0, 0)
2014-11-14 15:15:08.608000 50/52 (2|25:26|4|1) Traceback (most recent call
last):
File dbEvents.py, line 103, in run
event_listener.begin()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1845, in begin
self.__event_blocks.append(EventBlock(self.__queue, self._db_handle,
block_events))
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1732, in __init__
self.__wait_for_events()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1740, in __wait_for_events
self.close()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1763, in close
del self.__callback
AttributeError: _EventBlock__callback
Von: firebird-python@yahoogroups.com [mailto:firebird-python@yahoogroups.com]
Gesendet: Freitag, 14. November 2014 15:03
An: firebird-python@yahoogroups.com
Betreff: Re: AW: AW: AW: [firebird-python] FDB v1.4.2
Hi,
Can you try to add a call to self.close() before exception is raised in
__wait_for_events ? Does it helps?
if db_api_error(self._isc_status):
self.close()
raise exception_from_status(DatabaseError, self._isc_status, Error
while waiting for events:)
best regards
Pavel
Dne 14.11.2014 v 14:08 Dominik Psenner
dominik.psen...@topcontrol.itmailto:dominik.psen...@topcontrol.it
[firebird-python] napsal(a):
I refreshed the package and it looks like the memory leak is almost gone. I
can observe that the number of EventBlock’s left in memory are equal to the
count of errors raised with the invoke of begin() (namely one of: unknown ISC
error 0, error reading from connection, error writing to connection).
So there’s still a troublesome execution path left and it took some time
until I had wrapped my head around it. According to the stack trace the
exception is thrown in the constructor of EventBlock:
File dbEvents.py, line 103, in run
event_listener.begin()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1844, in begin
self.__event_blocks.append(EventBlock(self.__queue, self._db_handle,
block_events))
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1732, in __init__
self.__wait_for_events()
File C:\Python27\lib\site-packages\fdb-1.4.3-py2.7.egg\fdb\fbcore.py, line
1741, in __wait_for_events
Error while waiting for events:)
DatabaseError: ('Error while waiting for events:\n- SQLCODE: -902\n- Error
reading data from the connection.', -902, 335544726)
That would be right here where the EventBlock should be appended to the list
of event blocks:
for block_events in self.__blocks:
self.__event_blocks.append(EventBlock(self.__queue, self._db_handle,
block_events))
But like this an initialized and broken EventBlock is never appended to the
__event_blocks list and therefore EventBlock.close() is never invoked. ☹
Von: firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.com
[mailto:firebird-python@yahoogroups.com]
Gesendet: Freitag, 14. November 2014 13:38
An: firebird-python@yahoogroups.commailto:firebird-python@yahoogroups.com
Betreff: Re: AW: AW: [firebird-python] FDB v1.4.2
Hi,
Fixed. This time I've just refreshed the 1.4.3 package, as I don't want
to make another point release in single day. If anyone already got
unpatched 1.4.3 (unlikely that anyone beside you get it so quickly),
then it will replaced by next version 1.5 (full Firebird 3.0 support)
scheduled for release before end of year.
best regards
Pavel Cisar
IBPhoenix
Dne 14.11.2014 v 12:24 Dominik Psenner
dominik.psen...@topcontrol.itmailto:dominik.psen...@topcontrol.itmailto:dominik.psen...@topcontrol.it%3cmailto:dominik.psen...@topcontrol.it
[firebird-python] napsal(a):
Ah, you were faster than I could write. Sorry for me being so persistent.
Unfortunately I am unable to post a file in the bug tracker now that the
issue is resolved, therefore I write you here.
The good thing is, 1.4.3 improved the situation. One cyclic reference has
gone away but there’s still one cyclic reference left to
